Sourcehut will blacklist the Go module mirror

https://sourcehut.org/blog/2023-01-09-gomodulemirror/?

by Tomte • 4 years ago

550 321 4 years ago

Mhmmm. Yeah I wish the default were to not use a proxy. Though to be fair I'm not sure exactly what the performance implications would be.

xena3• 4 years ago

From what I understand, the proxy also helps people make sure that an upstream deleting their GitHub repos doesn't result in builds breaking on new machines that don't have it cached locally. Imagine the problems that could happen if someone new joins your team, runs `go build` and then one of the vital dependencies 404s.

The other problem is that it's Google so their perception of "not much traffic" is "biblical floods" to other people.

_wolfie_• 4 years ago

But that is what should happen. The build should break. So that someone can fix it.

cesarb• 4 years ago

the proxy also helps people make sure that an upstream deleting their GitHub repos doesn't result in builds breaking

This is a consequence of what IMO is another bad decision by the Go team: having packages be obtained from (and identified by) random github repos, instead of one or more central repositories like Maven Central for Java or crates.io for Rust. The proxy ends up being nothing more than an ad-hoc, informally-specified, bug-ridden re-implementation of half of a central repository, to paraphrase Greenspun's tenth rule.

Macha• 4 years ago

If they change the case on their username on the other hand, the Go ecosystem explodes: https://github.com/sirupsen/logrus/issues/570#issuecomment-3...

xena3• 4 years ago

They should probably use GOPRIVATE[1] instead. GOPRIVATE doesn't disable the module cache globally, it just disables it for individual domains or paths on domains. This is mainly used with private repo dependencies on GitHub.

[1] https://goproxy.io/docs/GOPRIVATE-env.html

ddevault13• 4 years ago

Thanks for the tip, will update the post.

pantojax45• 4 years ago

Not a good look for Google here.

galleywest200• 4 years ago

It should not be necessary to fetch the same git module up to 2,000 times per day.

Holy cow Google! Wouldn't it behoove us to check if any changes occurred before downloading an entire repo?

pdonis• 4 years ago

I don't understand why they have to do a fresh clone every time.

winterqt• 4 years ago

Or, as mentioned in the post, why they don't do a shallow clone if they have to fetch it every time for whatever reason. Seems like a weird decision either way.

blibble• 4 years ago

presumably the proxy backend is stateless

yamtaddle9• 4 years ago

It sure reads like someone wrote this the dumbest way that could possibly work, without a thought for what the effects would be.

interactivecode• 4 years ago

could it be a CI service that builds the go-proxy server for testing and that build process does an initial clone of all sorts of go modules?

teeray2• 4 years ago

and implies a trust relationship with Google to return authentic packages.

The entire point of the sumdb (go.sum), is to prevent the need for such a relationship. If Google (or any proxy you use) tries to return questionable packages, it will be detected by that system.

ignaloidas2• 4 years ago

If the proxy changes a new version of a package, when you update it, there's no way to detect it since it fetches through the cache anyways, so a poisoned sum will be added to sumdb, and anyone who isn't fetching their packages through Google's proxy will get told that whatever they're using is trying to trick them.

morelisp4• 4 years ago

anyone who isn't fetching their packages through Google's proxy will get told that whatever they're using is trying to trick them.

No, the error message you get is neutral about which side might be wrong - it says "verifying module: checksum mismatch" and "This download does NOT match the one reported by the checksum server." (I've seen it a lot because it also appears when module authors rebase, which a small but surprisingly high number do...)

teeray2• 4 years ago

anyone who isn't fetching their packages through Google's proxy will get told that whatever they're using is trying to trick them.

That is exactly the detection of a poisoned module in the ecosystem. It would break builds, issues would get filed, and a new version would be released (and the malicious party may not be so lucky this time since it’s trust on anyone’s first use).

kevmo314• 4 years ago

In practice, I've never validated that any of the checksums in go.sum are either correct or consistent.

leetrout2• 4 years ago

I feel like Drew has been in a pissing match with the Go team for a while so this outcome doesnt surprise me.

I love Go but not Google's stewardship of it. The tracking proxy, Russ' takeover / squash of the package management work, the weird silence / stonewalling on other community issues...

Drew has a valid complaint. I hate to hear he was banned from the issue tracker but that sounds about right.

As a sibling said - GOPRIVATE is probably a good solution without throwing the baby out with the bathwater.

wwarner2• 4 years ago

Have to agree in as much banning people with legitimate issues is unhelpful. I was also taken aback when Peter Bourgon, a great programmer and contributor to the go ecosystem was banned from all go channels.

Beltalowda13• 4 years ago

I was also taken aback when Peter Bourgon, a great programmer and contributor to the go ecosystem was banned from all go channels.

Bourgon was frequently helpful and great, but also frequently rude, condensing, dismissive, and generally just unpleasant. I've seen this countless of times first-hand on Slack, Reddit, and Lobsters. I specifically stopped interacting with him long before he was banned. Whether he's a great programmer/contributor not isn't really important here.

I kind of hate how it's brought up here because I think he's not a bad bloke at all and I know the entire situation caused great personal hurt to him. But that doesn't change that he was the kind of "brilliant jerk" that would chase people out of the community with his behaviour, and that he was unreceptive to criticism of it (often getting pretty defensive/aggressive). No one liked how all of this turned out, but it did make the Go community a better place. Being helpful yesterday doesn't cancel out being a jerk today.

Same with Drew: he posted legitimate helpful issues. And he also ranted about how people were all a bunch of morons. I don't blame anyone for getting tired of that.

Zamicol2• 4 years ago

I've blocked many of the people that called for Peter's ban on Twitter. I don't want to be on their radar or a target of some sort of witch hunt. I consider myself a nice person and not inflammatory/offensive, but I'm a belt and suspenders type of person. It's more important that I can submit an issue on the tracker than interacting "socially" with people who have a higher chance of ostracizing targeted individuals. The risks in my mind now far outweigh the rewards.

counttheforks6• 4 years ago

Why does the Go team ban so many people? Doesn't sound like a healthy community.

no_wizard• 4 years ago

What’s the deal on package mangement? They did quite a few iterations here so I don’t follow. Don’t know about the Russ takeover either

pjmlp5• 4 years ago

TL;DR version.

Go package management was quite poor for many real use cases outside Google, so the community railed around many competing proposals, and when it appeared there was a clear winner, the Go team came out with their own solution instead.

This after kind of supporting the ongoing community efforts.

pjmlp5• 4 years ago

Many issues with Go design steem from "how Google does it", this is just yet another one.

throwaway89434518• 4 years ago

I doubt this is true in any objective sense. "issues" above is almost certainly subjective, and I don't think even in this particular case it's so much "how Google does it" as much as "Google isn't prioritizing limited Go-team resources to this issue right now".

In general, I've been very happy with Go's design for the last decade with relatively few, relatively minor exceptions. I specifically have enjoyed that the Go team has resisted demands from the wider community to add every feature that every other programming language has support for. Biasing toward minimalism has made the language very easy to learn and consequently any Go programmer can jump into any Go project and start being productive in a matter of minutes, and any non-Go programmer can be productive in a matter of hours. Similarly, the tooling is novel in that it has sane defaults (static, native compilation; testing out of the box; profiling out of the box; build tooling out of the box; reproducible package management and hosting out of the box; documentation generation and hosting out of the box; etc).

adql2• 4 years ago

For me it seems that it's more that the few people at helm didn't really bother to learn any modern language or update themselves on theory so it feels like slighty nicer 90's language, not something developed recently.

And their weird entrenchment in "doing things the way plan9 did"

Hell, how you design a statically typed language and go "nah, we don't need sum types! That's too complex!"...

brnt3• 4 years ago

Seems like they could also let one request per hour through, say, and then serve up the rest a 429.

Users trying to clone their project would hit an almost certainly up to date Google cache and thus be happy and sr.ht save on pretty much all that traffic and thus be also happy.

Pet_Ant• 4 years ago

I like this because it feels like malicious compliance and it would allow Google to seemlessly continue working should Google figure out their caching situation. Plus it uses a rarely used HTTP status. Love it.

nouveaux2• 4 years ago

I think the actions here are meant to generate attention. Google is acting in bad faith and needs to be called out.

m4lvin2• 4 years ago

I thought that too, but then sr.ht needs to keep state which IP made how many requests already, and as the post mentions google makes these requests from many different places. So they would have to count requests from the specific user agent and by that point it is again special purpose work and load on sr.ht (whereas general rate limiting per IP might be a good idea anyway).

wut42• 4 years ago

Why the burden of making this google-specific rate-limiter should fall on Sourcehut ? They are a small team and a user-agent-per-repo specific limiter have rare uses.

Google should definitely fix their shit, but when we see that a googler said "it would be a fair bit of extra work for us to read robots.txt", I understand the frustration on Sourcehut side, and banning an user agent is usually just a single line of configuration to add.

alphazard2• 4 years ago

Drew strikes me as very ideological, so it's refreshing to see a move this pragmatic. This has a better chance at bringing awareness to these issues, and resolving the crawler bullying than his usual antics.

calvinmorrison• 4 years ago

Doesn't sound like a pissing match:

"Due to the relatively large traffic requirements of git clones, this represents about 70% of all outgoing network traffic from git.sr.ht. A single module can produce as much as 4 GiB of daily traffic from Google."

znpy• 4 years ago

The go language should probably be spun off to an independent non-profit foundation.

Of course I’m well aware this is not going to happen.

xenophonf• 4 years ago

This:

I was banned from the Go issue tracker without explanation, and was unable to continue discussing the problem with Google.

is completely asinine. But it's also par for the course when it comes to interacting with Google. When is anyone going to hold them to account for their terrible customer service and community interaction?

Thaxll2• 4 years ago

It's one side of the story. I don't think he was banned for no reasons.

yxhuvud• 4 years ago

Perhaps, but those reasons should still be communicated.

ComputerGuru3• 4 years ago

Knowing him, it probably wasn’t nothing. But that also does Dr matter - he runs a hosting service that sees a fair amount of traffic and the onus is on Google as stewards of the golang community and the brilliant minds behind invisibly proxing requests through Google’s servers to continue to figure this issue out. They have his email address.

ddevault13• 4 years ago

I would share their side as well, but I never heard it. This is a violation of their own code of conduct, which requires them to notify the affected person, explain why, and offer the opportunity to mediate the situation. This is not the first time I was banned from the Go community without notice or explanation, and the first time turned out to be frivolous -- the ban was overturned months later with an admission that it was never justified in the first place. Community management in Go strikes me as very insular and unprofessional.

Regardless, I don't really want to re-litigate it here. The main issue is that Google has been DoS'ing SourceHut's servers for two years, and I think we can all agree that there is no conduct violation for which DoS'ing your servers is a valid recourse.

Semaphor3• 4 years ago

The other side: https://github.com/golang/go/issues?q=is%3Aissue+sort%3Aupda...

Spoiler: Nothing.

edit: Possibly not as much nothing, see the replies to [0]. But the GH search kinda sucks for this.

[0] https://news.ycombinator.com/item?id=34311799

mekoka• 4 years ago

https://drewdevault.com/2022/05/25/Google-has-been-DDoSing-s...

I was banned from the Go issue tracker for mysterious reasons [ In violation of Go’s own Code of Conduct, by the way, which requires that participants are notified moderator actions against them and given the opportunity to appeal. I happen to be well versed in Go’s CoC given that I was banned once before without notice — a ban which was later overturned on the grounds that the moderator was wrong in the first place. Great community, guys. ]

When a story has two sides and one party chooses to keep silent when accused, I tend to favor the accuser.

jchw• 4 years ago

I find it annoying that blocking the Go module proxy (on the server side) doesn't cause a graceful fallback. Am I unreasonable in thinking that it should? Doesn't the sum file prevent you from getting a maliciously modified copy?

ysleepy• 4 years ago

Are google's requests done on-demand, or part of some scheduled refresh?

I am wondering, because even if google's traffic is unreasonable, it might still be less than without the proxy.

CI configurations are notoriously inefficient with dependency fetching, so I would not be surprised if the actual client traffic is massive and might overwhelm sourcehut if all migrate to direct fetches.

ComputerGuru3• 4 years ago

Per the issue, both.

lanstin8• 4 years ago

I recently moved to docker build pipeline for a project, and it’s redownloading all deps on each source file change, unlike the efficient on disk incremental compilation, because of how docker layer caching works, so my usage skyrocketed (and my build times went from seconds to minutes).

ComputerGuru3• 4 years ago

From the GitHub issue, by a Googler:

For boring technical reasons, it would be a fair bit of extra work for us to read robots.txt […]

This is coming from one of the biggest, richest, most well-staffed companies on the planet. It’s too much work for them to read a robots.txt file like the rest of the world (and plenty of one-man teams) do before hammering a server with terabytes of requests.

If this is too much for them then no wonder they won’t implement smarter logic like differential data downloads or traffic synchronization among peer nodes.

psanford6• 4 years ago

Why did sourcehut not take the offer to be added to the refresh exclusion list like the other two small hosting providers did? It seems like that would have resolved this issue last year.

fluidcruft• 4 years ago

I wondered that too, but then I wondered if that's what sourcehut have actually done. I didn't notice any details about how go module mirror will be blocked.

Wouldn't the effect on sourcehut users be identical?

rakoo6• 4 years ago

Why should you opt-in to a No-DDOS list ? Why is it not the default ?

xigoi• 4 years ago

IIRC, because that would only help the official SourceHut instance, not other instances.

ddevault13• 4 years ago

For a number of reasons. For a start, what does disabling the cache refresh imply? Does it come with a degradation of service for Go users? If not, then why is it there at all? And if so, why should we accept a service degradation when the problem is clearly in the proxy's poor engineering and design?

Furthermore, we try to look past the tip of our own nose when it comes to these kinds of problems. We often reject solutions which are offered to SourceHut and SourceHut alone. This isn't the first time this principle has run into problems with the Go team; to this day pkg.go.dev does not work properly with SourceHut instances hosted elsewhere than git.sr.ht, or even GitLab instances like salsa.debian.org, because they hard-code the list of domains rather than looking for better solutions -- even though they were advised of several.

The proxy has caused problems for many service providers, and agreeing to have SourceHut removed from the refresh would not solve the problem for anyone else, and thus would not solve the problem. Some of these providers have been able to get in touch with the Go team and received this offer, but the process is not easily discovered and is poorly defined, and, again, comes with these implied service considerations. In the spirit of the Debian free software guidelines, we don't accept these kinds of solutions:

The rights attached to the program must not depend on the program's being part of a Debian system. If the program is extracted from Debian and used or distributed without Debian but otherwise within the terms of the program's license, all parties to whom the program is redistributed should have the same rights as those that are granted in conjunction with the Debian system.

Yes, being excluded from the refresh would reduce the traffic to our servers, likely with less impact for users. But it is clearly the wrong solution and we don't like wrong solutions. You would not be wrong to characterize this as somewhat ideologically motivated, but I think we've been very reasonable and the Go team has not -- at this point our move is, in my view, justified.

jlarocco• 4 years ago

Supposedly the operator of sourcehut has been banned from posting to the Go issue tracker: https://github.com/golang/go/issues/44577#issuecomment-11378...

So, obviously he's supposed to know the even more obscure and annoying method of opting out.

pas2• 4 years ago

Hm. https://sourcehut.org/robots.txt HTTP 404

brnt3• 4 years ago

https://sr.ht/robots.txt

cp93• 4 years ago

presumably the robots.txt entry they are talking about is https://git.sr.ht/robots.txt

TehCorwiz• 4 years ago

They're referring to the actual service domain not the public static domain.

https://git.sr.ht/robots.txt

throwaway89434518• 4 years ago

I don't have any particular affinity for Google, but they're still a business and they're already developing the Go language (and relevant infrastructure) at their own expense. It's not like the Go team at Google has access to the entire Alphabet war chest like your "biggest, richest, well-staffed companies on the planet" suggests.

nouveaux2• 4 years ago

Go since inception has always been well funded. It is authored by some of the biggest names in programming and they are on staff at Google. This is not a side hobby. Not sure why you're suggesting that Go is lacking in resources.

treis2• 4 years ago

But this isn't one of those "we developed fast and did dumb stuff". They put significant effort into doing something dumb.

Analemma_• 4 years ago

Surely Google of all places has the most tested, battle-hardened robots.txt library in existence, and they have a company-wide public monorepo to boot. There's no excuse for this.

Dylan16807• 4 years ago

It seems like they're getting tons of bandwidth paid by the war chest if they don't care about this waste at all.

wmf• 4 years ago

This problem isn't really specific to Google. If some hobby project was DoSing sites it would get banned. "We don't have the resources to not DoS" is not a valid excuse. The Go team needs to scope their ambitions properly; if they can't make their proxy work safely they should not have bothered to develop it.

Trumpi• 4 years ago

Well, for boring technical reasons, I guess they will remain blocked.

tex0• 4 years ago

Is the source code of the service behind proxy.golang.org actually open-source?

comex• 4 years ago

According to some comments in the linked GitHub issue, including [1] from last May, Drew could have simply asked to be excluded from automatic refresh traffic from the mirror. If I understand correctly, that would still leave traffic from the mirror when it’s acting as a direct proxy for someone’s request, but that is traffic that would be going to sr.ht regardless.

For some reason he did not do this and instead chose an option that causes breakage.

[1] https://github.com/golang/go/issues/44577#issuecomment-11378...

FiloSottile2• 4 years ago

I can confirm asking to be excluded from refreshes (which AFAIK is still a standing offer, but I obviously can't speak for the mirror team because I am not at Google anymore) would stop all automated traffic from the mirror, and that sr.ht could send a simple HTTP GET to proxy.golang.org for new tags if it wished not to wait for users to request new versions.

throwaway_620222• 4 years ago

I think it is problematic that we are using github issues as "support forum" for asking a git host provider to be excluded from the refresh list. This should not have come to that. Whatever happened to "reasonable defaults", so as a random person hosting a single Go module doesn't get DOSed - https://github.com/golang/go/issues/44577#issuecomment-86087... ?

FiloSottile2• 4 years ago

Everyone can make their own assessment of what is a reasonable default and what counts as a DoS (and they are welcome to opt-out of any traffic), but note that 4GB per day is 0.3704 Mbps.

stonogo• 4 years ago

Seems like a weird default policy, until you remember Google's wifi mapping opt-out process: https://support.google.com/maps/answer/1725632?hl=en#zippy=%...

"All this would go away if other people would just do what they're told" is a pretty dystopian policy, but it seems to be a popular choice in Google.

kramerger• 4 years ago

That right there is why i stopped using Google.

Anyway, things get a bit kafka-esque when you realise that there is another company doing this WiFi thing and to opt out from that one you need a different SSID suffix. Since you can't have both, you end up with at least one company data mining you.

Why GDPR has not put a stop to this is baffling

tristan957• 4 years ago

Should every person that hosts an instance of SourceHut, Gitea, or Forgejo have to opt-in to this? That just doesn't scale at all. Drew is standing up for all independent hosters as much as he is standing up for his own business interests.

jdiff10• 4 years ago

No other independent hosters are having issues. I'm on one with no more than a few dozen users. Multiple of us write in Go, including the owner of the service, and yet Gitea and the VPS hosting it have never even blinked. If there were more of an issue, there would be more of a fuss than just two individuals. And one of those individuals was completely satisfied with the temporarily hackjob while a more permanent solution has been in the works. The other just denied it with no reason ever given cause he wants to keep litigating the issue.

zozbot234• 4 years ago

Sr.ht should set up their own graceful rate-limiting anyway, or they're going to have problems with badly coded CI setups as the service becomes more popular.

necron992• 4 years ago

Wouldn't this just make the failures more mysterious and harder to track down for users? Failing sometimes for no reason that's obvious or apparent to end users is worse than always failing, IMO.

E.g.: source-based packages on distributions where users may not be Go programmers, or even non-programmers, will compile and install Go software where some nested library dependency is on sr.ht. These packages will now fail and, sadly, this is going to cause widespread disruption. I think it'd be worse if those failures only happened occasionally, and not reliably repeatably.

cratermoon2• 4 years ago

The situation remained so for over a year. In that time, I was banned from the Go issue tracker without explanation, and was unable to continue discussing the problem with Google

Something tells me there's another side to this story that we aren't hearing.

josephcsible4• 4 years ago

Why? Google has quite a reputation for wrongfully banning people from things. This doesn't seem unusual for them at all.

cratermoon2• 4 years ago

Google has quite a reputation for wrongfully banning people from things.

Extraordinary claims require extraordinary evidence.

HideousKojima• 4 years ago

Well Go's own CoC requires that Google explain the reason for the ban, and as of yet they've failed to do so.

SuperSandro2000• 4 years ago

So either you disable the proxy completely for your site or get overwhelmed by traffic from a not to well written service, similar to a small DDOS attack, which is run by google and they are not planning to fix this? Did I get anything wrong here?

tssva• 4 years ago

There is a refresh exclusion list which you can request your site to be added to. The proxy will continue to process requests for modules from your site but will not perform the automatic refresh which caused issues for Sourcehut. The Go team extended an offer to add Sourcehut to the list if a request to do so was made. The request never came and instead Sourcehut blocked the proxy.

0cf8612b2e1e3• 4 years ago

Does every site need to personally request they not be DoS attacked?

cbrozefsky3• 4 years ago

"Opt out of me DoSing" you is not legit behavior -- especially when the victim has raised it with you multiple times, suggested fixes, and you have then blocked them from communicating in your issue tracker.

That's some really entitled thinking on the part of the Go team at Google, and it's sad to see people stanning for them.

simonw• 4 years ago

Perform a shallow git clone rather than a full git clone; or, ideally, store the last seen commit hash for each reference and only fetch if it has been updated.

I'd be interested to understand why that solution hasn't been implemented yet.

alpaca1282• 4 years ago

Because from Google's perspective everything works just fine.

callmeal• 4 years ago

From the github issue,

"it would be a fair bit of extra work for us to read robots.txt", so clearly tracking commit hashes would be even more work.

paulcarroty• 4 years ago

https://git.sr.ht/robots.txt

YandexBot is a dickhead, too aggressive User-agent: Yandex Disallow: /

Hahaha.

n8ta• 4 years ago

# Used for Alexa, I guess, who cares

User-agent: Amazonbot

Disallow: /

rakoo6• 4 years ago

or, ideally, store the last seen commit hash for each reference and only fetch if it has been updated.

Isn't that exactly what "git fetch" is doing already ?

WhyNotHugo• 4 years ago

Yes, but they do a complete clone on each run.

necron992• 4 years ago

Whether Drew's right or wrong, this is disappointing. At the very least, it's going to make recommending Sourcehut as an alternative to Github harder to justify.

This is really unfortunate, especially as sr.ht isn't free (any longer). As far as I know, it's the only remaining managed source hosting/CI option for Mercurial users.

What a depressing way to start out 2023.

MaxBarraclough• 4 years ago

especially as sr.ht isn't free (any longer)

This isn't a bad thing, it was always meant to be financially supported by its users. The alternatives are ads or for Drew and co to run it at a loss indefinitely.

counttheforks6• 4 years ago

I'll just rehost my git repos on a git host who doesn't mind actually doing their job as a git host by letting git clients clone git repos.

Arnavion6• 4 years ago

What does your ideal git host do when one of its (non-paying) clients uses bots to clones repos so hard that its (paying) human clients are unable to clone their repos?

counttheforks6• 4 years ago

Improve infrastructure. Apparently they are trivially DDOSable by anyone renting a few servers and running git clone in a loop? That's a problem they should solve. And no, blocking by user agent is not adequate protection.

0cf8612b2e1e3• 4 years ago

Some napkin math:

- peak queries are 2500 requests per hour

- some repos are 4gb in size

- let’s round and say 2000 requests at 1gb = 2000 gb/hour = 48000 gb/day

- AWS bandwidth at $0.02 / gb = $960 / day = $28,800 / month

So, one of the richest companies in the world is charging you nearly $30k monthly because they cannot be bothered to be polite. Would you be ok with that situation?

jdiff10• 4 years ago

From what I can tell, per Drew's own characterization of the requests[0], this is wildly off in all aspects. Most repos are hundreds of KB. Large repos are a dozen MB. Per the other guy's characterization[1], his repo was less than 8MB, right in line with what I see in other repos. He saw 4GB of traffic split up across "more than 500" requests. And as Drew said, these bouts of activity aren't sustained even across an entire hour. To top it off, the cost of AWS bandwidth is about the least representative example you could have picked. You've wandered off the path at nearly every available opportunity.

[0] https://github.com/golang/go/issues/44577#issuecomment-78531...

[1] https://github.com/golang/go/issues/44577#issuecomment-85107...

adenozine2• 4 years ago

What's the value proposition of not using Github? Github is such an incredibly useful project. I actually go out of my way to avoid projects on Gitlab and co, just because I don't want to have to worry that it's going to just disappear one day because they thought they could out-build Microsoft.

vladharbuz2• 4 years ago

Your argument seems to be based on the idea that Microsoft would do a better job than its competition technically. The fact that you're happy to rely on them also seems to imply you think they're unlikely to act abusively. However, you're writing your comment in a thread about a similar megacorporation, Google, who is acting abusively, because their engineers are saying parsing a robots.txt file would be too difficult.

adenozine2• 4 years ago

I actually wasn't making an argument but posing a question. Your reply seems defensive and emotional.

I'm not sure what you mean by "act abusively" since the core of the product is storing and viewing source code. I mean, I could verify checksums if I needed to, and such a thing is common in workplaces I frequent.

What I asked before, and I will reiterate for an additional attempt to your comprehension, is how can Sourcehut store and view source code BETTER than Github can?

rsc4• 4 years ago

The Go team has been making progress toward a complete fix to this problem.

Go 1.19 added "go mod download -reuse", which lets it be told about the previous download result including the Git commit refs involved and their hashes. If the relevant parts of the server's advertised ref list is unchanged since the previous download, then the refresh will do nothing more than the ref list, which is very cheap.

The proxy.golang.org service has not yet been updated to use -reuse, but it is on our list of planned work for this year.

On the one hand Sourcehut claims this is a big problem for them, but on the other hand Sourcehut also has told us they don't want us to put in a special case to disable background refreshes (see the comment thread elsewhere on this page [1]).

The offer to disable background refreshes until a more complete fix can be deployed still stands, both to Sourcehut and to anyone else who is bothered by the current load. Feel free to post an issue at https://go.dev/issue/new or email me at rsc@golang.org if you would like to opt your server out of background refreshes.

[1] https://news.ycombinator.com/item?id=34311621

cstejerean• 4 years ago

I suspect they have a problem with this DDoS by default unless you ask to opt out behavior. Why is anyone getting hit with these expensive background refreshes until you have a chance to do it right? Why is it still not done right 2 years after this was first reported?

Maybe it should be an opt-in list where the big providers (such as github) can be hit by an army of bots and everyone else is safe by default.

dpkirchner2• 4 years ago

And why isn't this a priority? These folks are offering a service that benefits the Go ecosystem.

It smells like Go is on its way out.

maximilianburke• 4 years ago

I realize in the real world most modules are probably hosted by large providers that can absorb the bandwidth, like Github, but it seems incredibly discourteous to not prioritize the hammering of small providers, especially two years on when the response is still "maybe later this year".

I think Drew is right in that he shouldn't take a personalized Sourcehut-only exception because this doesn't address the core issue for any new small providers that pop up.

Between this and the response in the original thread that said, "For boring technical reasons, it would be a fair bit of extra work for us to read robots.txt," it gives the impression that the Go team doesn't care. Sometimes what we _need_ to do to be good netizens is a fair bit of boring technical work but it's essential.

yamtaddle9• 4 years ago

It's super-weird that the Google-side Go folks' responses to this have basically been "we don't have the resources to run this service that we decided to run and that's now misbehaving, responsibly". Like... don't, then? Why take on that kind of thing in the first place if urgent fixes to its generating abusive traffic for no good reason take three years?

simon_o2• 4 years ago

he shouldn't take a personalized Sourcehut-only exception because this doesn't address the core issue for any new small providers that pop up

Exactly. We already saw how this ended with Google vs. people running mail servers.

cowl• 4 years ago

The Exception "exclusion solution" definitely it's not personalized and Sourcehut-only but available to anyone that requests it and in the issue track you can see several people that are using this exclusion already. True that an opt-out (what this solution boils down to) is not ideal but it's way better than using your users quality of service to try strong-arming your side. And anyway from the go team it's been made clear that they are working on improving the situation of the refreshes and this opt-out is just the temporary solution until they fix the real issue.

ddevault13• 4 years ago

Hi Russ! Thank you for sharing. I am pleased to hear that there is finally some progress towards a solution for this problem. If you or someone working on the issue can reach out via email (sir@cmpwn.com), I would be happy to discuss the issue further. What you described seems like an incomplete solution, and I would like to discuss some additional details with your team, but it is a good start. I'm also happy to postpone or cancel the planned ban on the Go proxy if there's active motion towards a fix from Google's end. I am, however, a bit uneasy that you mentioned that it's only prioritized for "this year" -- another year of enduring a DoS from Google does not sound great.

I cannot file an issue; as the article explains I was banned from the Go community without explanation or recourse; and the workaround is not satisfying for reasons I outlined in other HN comments and on GitHub. However, I would appreciate receiving a follow-up via email from someone knowledgeable on the matter, and so long as there is an open line of communication I can be much more patient. These things are easily solved when they're treated with mutual respect and collaboration between engineering teams, which has not been my experience so far. That said, I am looking forward to finally putting this issue behind us.

rsc4• 4 years ago

Sounds good. I will reach out over email. Thanks.

eadmund• 4 years ago

This reminds me a bit of a disfunctional relationship: clearly Sourcehut wants Google to stop DDoS their servers; clearly Google don’t actually want to DDoS Sourcehut, but Sourcehut also doesn’t want to ask Google to stop, and Google also want to be asked to stop. And so nothing gets done.

The question is who will swallow their pride first: Sourcehut or Google.

rsc4• 4 years ago

This isn't true. Sourcehut reported a bug, and since the bug is somewhat involved to fix entirely, we asked what the impact of the bug is to them and offered to make a custom change for the site in the interim. The impact matters: the appropriate response is different for "I saw this in my logs and it looks weird but it's not bothering me" versus "this is causing serious problems for my site". We have been getting mixed signals about which it is, as I noted, but since Sourcehut told us explicitly not to put in a special case, we haven't.

vladharbuz2• 4 years ago

Thanks for the insight, Russ. Would you comment on what the potential consequences of opting out of background refreshes would be? Could there be any adverse effects for users?

rsc4• 4 years ago

Opting out of background refreshes would mean that fetching a module version that (1) no one else had fetched in a few days and (2) does not use a recognized open-source license might not be in the cache, which would make 'go get' take a little extra time while the proxy fetched it on demand. The amount of time would depend on the size of the repo, of course.

The background refresh is meant to prefetch for that situation, to avoid putting that time on an actual user request. It's not perfect but it's far less disruptive than having to set GOPRIVATE.

cmatthias6• 4 years ago

Why does the Go team and/or Google think that it's acceptable to not respect robots.txt and instead DDoS git repositories by default, unless they get put on a list of "special case[s] to disable background refreshes"?

Why was the author of the post banned without notice from the Go issue tracker, removing what is apparently the only way to get on this list aside from emailing you directly?

Do you, personally, find any of this remotely acceptable?

jslql• 4 years ago

Why should a git client respect an http standard such as robots.txt?

kevincox3• 4 years ago

FWIW I don't think this really fits into robots.txt. That file is mostly aimed at crawlers. Not for services loading specific URLs due to (sometimes indirect) user requests.

...but as a place that could hold a rate limit recommendation it would be nice since it appears that the Git protocol doesn't really have the equivalent of a Cache-Control header.

michaelcampbell• 4 years ago

Going up the stack a bit this feels to me like the same sort of "we know better" mentality that said no one really needs generics.

codesniperjoe• 4 years ago

If google is on the way to fix this, a look at the resources on (EVERYONES LOCAL!) not de-duplicated $GOMODCACHE would be very welcome too!

(At least from an enviromental perspective.)

Easy to verify, get a report [1]: go install paepcke.de/fsdd/cmd/fsdd@latest && cd $GOMODCACHE && fsdd .

[1] Warning: Apple user with fixed restricted & expensive nvme space maybe very upset. Easy fixable via fsdd . --hard-link.

ddevault13• 4 years ago

Hey Russ, I got your messages that my emails aren't coming through but I'm not sure why. As an alternative, you can reach me on IRC at ddevault on Libera Chat. I'm in CEST, but my bouncer is always online. Cheers!

n4jm4• 4 years ago

Vendoring in go mod should be mandatory.

kwhitefoot2• 4 years ago

"it would be a fair bit of extra work for us to read robots.txt",

Can someone explain what this means? After all reading a small text file is plainly easy to do so its meaning must be something other than obvious.

yamtaddle9• 4 years ago

All I can figure is they're trying really, really hard to keep this very parallel on their side and also avoiding having to coordinate between nodes. It can't possibly be the reading of the robots.txt that's hard, so I think that statement has more to do with applying those policies to all the nodes—they must regard coordination between nodes to ensure the system as a whole isn't exceeding e.g. request rate maximums, to be "a fair bit of extra work".

Judging just from the linked post, the issue on which this was discussed, and this thread, it's feeling a lot like this proxy was some kind of proof-of-concept that escaped its cage and got elevated to production.

daenney• 4 years ago

I can see this happening on my own git hosting too. I've started moving my Go code off GitHub and the module mirror shows up every 25min for each repo it's aware of doing a full clone. Thankfully the few modules I've moved are very small ones with very little history. This won't come anywhere near my egress allocation for that box.

But the whole thing is frankly a little rude.

lmas2• 4 years ago

Yeah I wanted to do the same thing and self host my repos but that will have to wait now until this issue is fixed, it looks like...

Could you tell me more about what tool you used to host the repos? And I assume you noticed the traffic in your web logs?

davidbanham• 4 years ago

The proposed resolutions from Google have been along the lines of changes to the proxy application code. The resistance has been that this is hard because of the way responsibility is divided between the thin layer that provides the proxy web service and the ‘go’ command itself, which actually fetches the packages.

Would it be simple to solve this with an additional layer of the same proxy? Currently, end users request a package from proxy.golang.org as per the default value of the GO_MOD_PROXY env var. Google runs many of these to handle the traffic, let’s say 1000. They all maintain a mirror of all packages that have been recently requested (note that I expect there’s more nuance here around shared lists of required packages, etc, but the structure should hold true)

The result is that every one of the 1000 proxy instances requests the source data from git.sr.ht every day.

Google could set GO_MOD_PROXY on the existing instances to internalproxy.golang.org. They could then run 100, or maybe 10 of these internal instances. This would drop the traffic to hit.sr.ht by one or two orders of magnitude.

I suspect it would require minimal if any change to application code. This might be accomplished entirely within the remit of a sysadmin (SRE?).

Any holes in my reasoning?

Sourcehut will blacklist the Go module mirror

Related Stories