PyPI Was Subpoenaed

https://blog.pypi.org/posts/2023-05-24-pypi-was-subpoenaed/

by quercusa • 3 years ago

1153 594 3 years ago

blog.pypi.org

snapcaster2• 3 years ago

Really weird, anyone have some inside gossip on what this is about?

paulpauper2• 3 years ago

maybe to do with web scrapping, auto-posting spam, etc.

https://www.developer-tech.com/news/2023/may/22/pypi-suspend...

yuvadam• 3 years ago

Subpoenas are from March and April, predating the spamming of the past few weeks.

Retr0id2• 3 years ago

The most optimistic reason would be that they were investigating a supply-chain attack, or something of that nature.

commandlinefan• 3 years ago

This makes me wonder... it's entirely possible that the PyPI people would be enthusiastic about helping to track down offenders, and their users might agree, _if they knew what the offense was_. Instead, they're presented with a typically antagonistic demand for details, so they understandably get defensive on behalf of their users. I wonder if there's not a better, less heavy-handed way to get cooperation with law enforcement when the request is reasonable.

bhouston• 3 years ago

I wish it was that but those people would be smart enough to not use their real name when signing up - those doing supply chain attacks are often at least somewhat professional and take precautions.

I suspect it was more about going after software that was enabling piracy, those are often created by naive students who are not expecting the power of government to be unleashed on them.

jamesmurdza• 3 years ago

It could be related to the large number of malicious or booby-trapped packages that have been uploaded recently to the index.

foota• 3 years ago

My guess? A hacking case against someone for typosquatting or malicious packages or something.

guhcampos• 3 years ago

Could be anything I guess, even legitimate reasons. T Think of the supply-chain attacks going on in the past few years. I'd say investigating these would be a legitimate reason for a subpoena.

misterpigs• 3 years ago

I love this level of transparency.

voynich• 3 years ago

Yeah, whether necessary or not, it's still nice to have such a level of detail in a transparency report.

tomjen32• 3 years ago

We will not be releasing the usernames involved publicly or to the users themselves.

Which is the most important part.

tptacek11• 3 years ago

They're not allowed to release that.

Edit

I read 'chaps as saying there was an NDA on the subpoena, but apparently there wasn't, so this might just be flatly wrong.

dev_tty01• 3 years ago

Releasing the user names would not be respecting the privacy of the users.

SV_BubbleTime• 3 years ago

Sure. But I would love if they had considered this from the start:

As a result we are currently developing new data retention and disclosure policies.

“I guess we don’t actually need that” should have been the idea from the start.

thih92• 3 years ago

After a quick glance at the information listed in the report I didn't notice excessive data collection on pypi's part.

I'd say they followed "I guess we don't actually need that" approach reasonably well so far and good for them if they want to improve that even more.

donaldstufft7• 3 years ago

One important thing to remember here is that PyPI was originally started in 2002 as a weekend hack project that grew overtime to become the piece of critical infrastructure it is today. There's a lot of stuff in PyPI that exists as historical baggage and cruft and reviewing them just never bubbled up to be a priority. Likewise a lot of the policies it has have been added and grown overtime as something happened that caused us to need one.

On top of all of that, it's volunteer run and has been understaffed for basically it's entire life, so sitting down and figuring out a proper data retention policy that takes a holistic view of everything we have just never bubbled up.

In general I think we already do a pretty good job of collecting a minimal amount of data, and hopefully with proper policies we can do an even better job.

throwaway_131403• 3 years ago

Do you still love it if it enables a terrorist or otherwise very bad person to evade capture?

evandale• 3 years ago

Not OP but yeah. I don't buy into the whole "to protect you from bad people I need to erode your rights" argument.

Never made sense to me. Terrorists and other very bad people usually aren't in the business of following laws so I don't know what crimes you'd prevent by weakening the rights of everyone else.

Aachen• 3 years ago

Climate activism is also being considered an act of terrorism by some now (particularly some Christian party in Germany), dunno if those people label themselves as 'very bad persons'. Probably goes for all terrorists, but this might be easier to relate to as it's grounded in reality and we'd likely agree with the change they seek

Child porn and terrorism are the favorite subjects of politicians looking to enact a new law but idk if it's good to follow that thinking and use it as an example as opposed to a serial killer or something

Danjoe4• 3 years ago

Yes. Truth itself stands at the top of the moral hierarchy. It can stand alone without any justification. "You told the truth" will never be immoral, consequences be damned.

itake• 3 years ago

I can't tell if this is sarcastic.

While they are transparent the events happened, they are not transparent about which packages and what authors are being flagged, which is unfortunate.

thih92• 3 years ago

Is it possible that they can't publish that? Perhaps even not allowed to say that they can't publish that?

einpoklum2• 3 years ago

While they are transparent the events happened

Considering they are admitting they will always obey government commands, including regarding non-disclosure of actions to affected users, it is prudent to assume they are, in fact, not transparent about events; only about those events which the government has let them tell you about. Other events (e.g. National Security Letters) may or may not have occurred.

jacquesm4• 3 years ago

We will not be releasing the usernames involved publicly or to the users themselves.

Why not to the users themselves? Have they been prohibited from doing so? (TFA does not say afaict)

shadowgovt3• 3 years ago

That, or they have reason to believe the investigation is legitimate and they would prefer not to hinder it.

jacquesm4• 3 years ago

They say very explicitly that they do not know what it was about.

ruffrey• 3 years ago

Often subpoenas are part of an ongoing investigation, and they require not releasing information to those who’s data was subpoenaed.

stonepresto3• 3 years ago

The why: notifying a suspect in a criminal investigation could allow them to flee or take measures to avoid prosecution, destroy evidence, etc.

tptacek11• 3 years ago

The subpoena probably includes a nondisclosure clause; a court order certainly would. The mechanics of nondisclosures on subpoenas is interesting and I don't totally understand it (by definition, a subpoena is a document authorized by someone other than a judge).

jacquesm4• 3 years ago

So is this message a way to obliquely signal to those users (whoever they are) that they may be under investigation without actual disclosure?

chaps2• 3 years ago

There was no NDA:

  "We have waited for the string of subpoenas to subside, though we were committed from the beginning to write and publish this post as a matter of transparency, and as allowed by the lack of a non-disclosure order associated with the subpoenas received in March and April 2023."

jsjohnst3• 3 years ago

by definition, a subpoena is a document authorized by someone other than a judge

Uhm, am I misunderstanding what you wrote, because that is definitely not true. Subpoenas require an officer of the court by definition (in the US anyway), which can be a judge, a court clerk, or even lawyers in some jurisdictions.

toast0• 3 years ago

The users themselves already know their own usernames, presumably. They could let the users know they were subpoenaed without letting them know their username. :P

rolph• 3 years ago

[In March and April 2023, the Python Software Foundation (PSF) received three (3) subpoenas for PyPI user data. All three subpoenas were issued by the United States Department of Justice. The PSF was not provided with context on the legal circumstances surrounding these subpoenas. In total, user data related to five (5) PyPI usernames were requested.]

either a small group of users, or one user with multi aliases wrote a nastyapp ?

cubefox4• 3 years ago

Apparently no plans to set up a canary.

skullone• 3 years ago

Why would they? It's a public repository, nothing confidential or private

JohnFen3• 3 years ago

Account details are confidential and private.

jrockway• 3 years ago

Is there any precedent for people not facing legal consequences for failing to update the canary? The subpoena probably says "and also update your warrant canary to say there were no legal requests." Now you're in contempt of court and in jail for 5 years while you wait for your "compelled speech" case to go to the Supreme Court.

In general, I think it usually goes poorly when programmers invent clever legal workarounds. The legal system isn't a computer program. It's guys with guns.

buildbot3• 3 years ago

Isn’t the idea that the (US) government can’t (technically) compel you to lie?

bitxbitxbitcoin• 3 years ago

Exactly. Warrant canaries are security theatre.

burnished3• 3 years ago

Can a subpoena stipulate that?

redox99• 3 years ago

Can you provide any evidence of the US forcing someone to update their canary?

JohnFen3• 3 years ago

The subpoena probably says "and also update your warrant canary to say there were no legal requests."

I think that would be outside what can be done with a subpoena. It would require a court order.

Zetice2• 3 years ago

If you can just say, "We got subpoenaed" in a blog post, isn't that even more effective than a canary would be?

cubefox4• 3 years ago

There was a delay.

caturopath2• 3 years ago

Canaries would be for times when they couldn't legally say that.

tptacek11• 3 years ago

Canaries probably don't work, which makes them worse than theater.

waselighis2• 3 years ago

Long ago, Apple included a warrant canary in their transparency report. One day, it disappeared. Nothing came of it.

https://www.theverge.com/2014/9/18/6409575/apple-warrant-can...

The problem with a warrant canary is there's too much doubt about why it disappeared. Did they actually receive a warrant, or is it just a decision from corporate to discontinue the practice?

cubefox4• 3 years ago

There can be some doubt, but too much?

actionfromafar2• 3 years ago

A decision from corporate to discontinue is also a signal.

DANmode• 3 years ago

why it disappeared

The result is the same.

woodruffw9• 3 years ago

I don't understand (genuinely, I'd like to!) what a warrant canary would have done here: this was a subpoena, not a warrant, and PyPI is a public package index.

cubefox4• 3 years ago

I'm obviously talking about a subpoena canary.

LordShredda• 3 years ago

I'm guessing some poor typosquatter managed to hit a gov agency and is about to get alphabet soup all over him.

eur0pa• 3 years ago

That or fairly unlucky bug bounty hunters

nonrepeating• 3 years ago

“Get alphabet soup all over him”

This is my new favorite alternative to “vanned” (or “v&”)

flyinghamster• 3 years ago

I think I'm gonna snarf that one too. It's just too good.

tenpies• 3 years ago

“vanned” (or “v&”)

Also note that the noun associated with being "vanned" would be a "party van", not just a "van".

To be vanned/V& is to have the glowies inside the party van take the vanned party away.

https://knowyourmeme.com/memes/4chan-party-van

paulddraper• 3 years ago

poor typosquatter

fmajid• 3 years ago

More likely it is DRM-cracking packages.

stonepresto3• 3 years ago

I'd like to put $5 on "ransomeware threat actors"

quickthrower2• 3 years ago

Putting my conspiracy hat back in the closet, I guess there is plausible reason to believe that someone used PyPI for fraud, CP distribution or some other crime.

mistrial92• 3 years ago

accusations and planting evidence are everyday things in some security circles.. plus idiots abuse open systems.. not proof of malfeasance, maybe just a "setup"

kjkjadksj• 3 years ago

I don’t understand how the information requested is relevant at all for any purpose. Most users of pypi merely download through pip; they arent registering anything. Furthermore, I would think a bad actor who would register would spoof their ip and use burner accounts anyhow.

buildbot3• 3 years ago

Correlating IP address use to something else happening at the same time? Like a malware author being incredibly dumb and using their home IP to upload PyPy packages, while IDK, using that same IP as a C&C server endpoint.

shadowgovt3• 3 years ago

They may not even need to have slipped up and direct-connected via their home IP. The FBI has sufficiently compromised subsets of Tor in the past to do correlative attacks on specific targets.

caturopath2• 3 years ago

Most users of pypi

Presumably the 5 users in question were interesting in some way, not just random.

I would think a bad actor who would register would spoof their ip and use burner accounts anyhow

Maybe, but they could find that out with the information. If there's a 10% chance each was sloppy or un-paranoid, there's a 40% chance they get at least one piece of real info.

The person might not have thought they were doing anything wrong. Some judge might have greenlit this for a piracy case against the five maintainaers of youtube_dl{c} or something silly.

tomjen32• 3 years ago

We will not be releasing the usernames involved publicly or to the users themselves.

They point out that they are not subject to a gag order.

buildbot3• 3 years ago

Yeah this is interesting, because they could in absence of the gag order but choose not too. Unless it’s not a gag order but a specific don’t tell these users anything?

stjohnswarts2• 3 years ago

I doubt any investigators worth their salt would let the people they're investigating know that they are investigating them before they're ready to charge them.

firstlink2• 3 years ago

associated with the subpoenas received in March and April 2023.

Oddly specific wording there. It would seem they have received additional subpoenas outside that timeframe which do have gag orders, and someone slipped up and failed to put the gag orders in these particular subpoenas.

Seems like the DOJ may be doing some long-term fishing for, what, software developers? First the DOJ came for the conservatives, and I said, "go get 'em!" because I wasn't a conservative; next the DOJ came for ____?

bratbag• 3 years ago

It's interesting how you manage to leap from what is probably a supply chain attack investigation straight into a pErSeCuTeD CoNsErVaTeS conspiracy.

metiscus2• 3 years ago

Across the ages, government has applied a disproportionate level of scrutiny to groups and people perceived as dissidents, minorities, and anyone else who could potentially be conceived as a threat to institutional power regardless of the magnitude of the threat or if that threat is true or false. Historically this is a bipartisan issue, for decades the FBI vigorously attacked anti-war groups, black civil rights groups, and various left wing groups via COINTELPRO. I guess to summarize, the way I think of it is that the government is coming for anyone they see as a potential threat and it doesn't matter what the politics of that group are.

burnished3• 3 years ago

Step one: have position and power as part of a dominant group

Step two: style yourself as an oppressed minority

Step three: defend any action, decision, or position as a persecuted martyr

Zetice2• 3 years ago

Dumb legal question; what's the difference, if any, between "We've been subpoenaed" and "Someone had a warrant for data"?

paxys• 3 years ago

Warrant = we (police or other authority) have the right to come and search your property for evidence.

Subpoena = the court compels you to hand over the evidence we need.

woodruffw9• 3 years ago

Subpoenas are orders, but they're not necessarily court-issued. Warrants, on the other hand, are court-issued -- the police can't issue warrants on their own in the US.

woodruffw9• 3 years ago

Not a dumb question: a subpoena is an order to provide information or access, while a warrant is a court-issued document authorizing the government (or an agent of the government) to perform an act (e.g., an arrest, or seizure of an item).

Subpoenas can be issued by attorneys (including prosecuting attorneys) as part of the investigative and discovery processes.

indymike• 3 years ago

Subpoena = "Ask firmly, but nicely"

Warrant = "Back up the van and haul it away"

schoen2• 3 years ago

The subpoena is a command to the possessor of the data, which tells the possessor of the data to produce it, with a particular deadline. Since this deadline is in the future, the subpoena can be challenged legally (normally by requesting a court to "quash" it; more riskily, sometimes by complying imperfectly or not at all, and then arguing in response to an attempt to punish the noncompliance that this was reasonable). A subpoena can be issued by many entities, for example including some law enforcement entities themselves, or a lawyer actively involved in litigation. (Yes, lawyers can personally write and issue subpoenas.) The subpoena is, however, enforced by a court, in the sense that the court is asked to punish people who fail to obey it.

The warrant is a command to a law enforcement officer, which allows the law enforcement officer to personally go and search and seize things (or people), while overriding some rights that would normally prevent this. Normally it is issued by a court. Generally there is no way to challenge a warrant to prevent its execution, because it is not disclosed to the target before it's executed (i.e., a law enforcement officer shows up with the warrant and begins executing it immediately, by force if necessary).

(Edit: I wrote above that it's risky to comply imperfectly with a subpoena and then argue in court that this was reasonable, but usually if a lawyer gives a professional opinion that the subpoena is invalid or overbroad for some reason, then the recipient of the subpoena won't be punished for following that advice. The lawyer may also attempt to negotiate directly with the issuer of the subpoena, for example by sending a letter explaining why the the subpoena appears to be invalid. The legal standards for issuance of subpoenas are also pretty broad. For civil litigation, which is not what DoJ is doing here, they are set out in https://www.law.cornell.edu/rules/frcp/rule_26; notably, they can be issued to third parties.)

therein• 3 years ago

Really nice response, I'm not the one who asked the question but I learned something from your response.

Zuiii• 3 years ago

https://www.law.cornell.edu/rules/frcp/rule_26;

This url does not exist. Was this response generated by an LLM?

ur-whale• 3 years ago

We will not be releasing the usernames involved publicly or to the users themselves.

Emphasis on the last part: or to the users themselves.

In other words: unless they actually let the users involved in spite of claiming the opposite, the whole article is complete posturing.

burnished3• 3 years ago

What a weird take

ralmidani• 3 years ago

Does a "subpoena" mean a judge was involved? The post says the subpoena was issued by the DOJ.

tptacek11• 3 years ago

It means a court is involved, but not a judge.

Edit

Even that is technically wrong; some DOJ subpoenas are apparently preauthorized by statute.

idlewords• 3 years ago

There's a pretty extensive list of administrative subpoena authority here:

https://www.justice.gov/archive/olp/rpt_to_congress.htm

tl;dr: Everyone from the Appalachian Regional Commission on down can subpoena you without a court being involved. And of course Congress has inherent subpoena powers.

avgcorrection5• 3 years ago

The privacy of PyPI users is of utmost concern to PSF and the PyPI Administrators, and we are committed to protecting user data from disclosure whenever possible.

Don’t lead with this.

In this case, however, PSF determined with the advice of counsel that our only course of action was to provide the requested data.

If you’re going to say this.

I’m not judging their decision. Maybe not going to prison is a greater concern to them. It’s fine to just say that you thought it was best to comply because [lawyer reasons that you don’t have to disclose to anyone]/counsel.

EDIT: Or say “there are bad people out there and we trust the DOJ”. Whatever.

tptacek11• 3 years ago

Lighten up. Nobody's going to federal court to stop the DOJ from investigating botnets, carding rings, and ransomware scams, which is what these things are usually about. Nobody's mental model of PyPI was that they had Signal's priorities.

avgcorrection5• 3 years ago

Then all the less reason to roll out the “of utmost importance” boilerplate. So what’s your point?

Also I don’t see how being light-hearted has anything to do with this submission, Thomas.

mrguyorama2• 3 years ago

Then they shouldn't say protecting their users are their top priority, because they have shown it is not. That's called lying. A correct statement would have been "we will comply with lawful LEO requests"

paulgb• 3 years ago

It's a completely reasonable reading of their message to assume that the “possible” in “whenever possible” roughly means “legal”. I don't think any reasonable reading of it means to imply that they are willing to violate federal law.

avgcorrection5• 3 years ago

Fair point.

HelloNurse• 3 years ago

sudo give us user data

duxup• 3 years ago

I don’t see anything conflicting in what they said.

They can feel that way, and comply.

avgcorrection5• 3 years ago

Yeah. I was probably being a little too boilerplate (what looked like) -intolerant. ;)

aa_is_op• 3 years ago

By the number of malicious packages that site has hosted over the past few months, this was only a matter of time.

I've lost track of the number of "white hats" that contact us with extortion requests after they used some dependency confusion attack.

Mystery-Machine2• 3 years ago

Why is it an extortion request and "white hats" if they have successfully found a security issue in your project and reported it to you, without actually exploiting it? Would you rather them not report it to you or even worse, exploit it?

cjsawyer• 3 years ago

Sounds like they’re the ones who implemented the bad code in the first place, as a honeypot. That just extortion with extra steps.

JadeNB• 3 years ago

Why is it an extortion request and "white hats" if they have successfully found a security issue in your project and reported it to you, without actually exploiting it?

Presumably because there is some demand for compensation before disclosure?

dahwolf2• 3 years ago

Pay me or I will harm you is extortion, as simple as that.

There's an entire industry now of people that check known vulnerabilities (so they don't invent anything themselves) in software/packages and cross check this against outdated websites, at a very large scale.

They have no morals or security ethics, they barely even have knowledge, they just want to make money with the least amount of effort possible.

Don't ever pay them a cent. They're just as ruthless as spammers.

accrual• 3 years ago

Is it still "white hat" if money or a transaction is involved? My understanding is it's either black hat, the exploit is sold for money. Red team, you paid to be exploited for your own benefit. Or white hat, an exploit was found and it's communicated to limit black hat and red team. White hat + money would just be gray hat or blackmail.

rocqua5• 3 years ago

White hats can still get bug-bounties. Though if a company hasn't published such a bounty and a hat 'extorts' the bounty by demanding payment or else they will publish, that hat has a tint of grey.

passion__desire• 3 years ago

Why can't PyPI safeguard popular packages by making sure that new packages are few (4 or 5) edit-distance away to make sure popular ones don't get intermixed with malicious packages. Is that difficult to implement?

rocqua5• 3 years ago

pip-env, pipe, pipes, sip, siv, lipo, etc Are all within an edit distance of 4 from pip, and would all be blocked.

Besides 'dependecy confusion' is not typo-squatting at all. It is about having a public package that masks the name of a private package repo. The default behavior of pip is to then use the public repo, which can let outsiders who know internal package names totally take over those internal packages.

heavyset_go4• 3 years ago

That's certainly a way to look a gift horse in the mouth.

wongarsu2• 3 years ago

We have waited for the string of subpoenas to subside, though we were committed from the beginning to write and publish this post as a matter of transparency, and as allowed by the lack of a non-disclosure order associated with the subpoenas received in March and April 2023.

That's suspiciously specific. Sounds to me like they also received some other subpoenas they aren't allowed to talk about.

VWWHFSfQ• 3 years ago

I think it just sounds like the three subpoenas they received

wongarsu2• 3 years ago

I'm not sure I'd call three subpoenas "a string of subpoenas" even if it's technically correct. But I'm more talking about specifically mentioning that the subpoenas from March and April 2023 don't have a gag order. Why mention those months specifically if in the other months they didn't receive any? The natural thing would have been to end the sentence six words earlier.

samanator• 3 years ago

Yep, I was thinking the same thing. What a beautiful way of communicating that.

Loquebantur3• 3 years ago

What a weird way to think about such events.

Such subpoenas are clandestine surveillance of citizens by their state. The problem with such types of surveillance in particular is the lack of accountability.

How does the ethical use of this prolematic tool get ascertained? Where and how is the democratic oversight implemented? How is misuse treated and prevented?

tapoxi• 3 years ago

Sounds like they got a National Security Letter.

Breza• 3 years ago

Agreed. It's like the librarians who responded to the Patriot Act by putting up "The FBI has not been here" signs.

dragonwriter10• 3 years ago

That's suspiciously specific. Sounds to me like they also received some other subpoenas they aren't allowed to talk about.

It could be, it could also be that they were trying to communicate both the timing of the subpoena string and why they are able to talk about it, and there aren’t any others.

OJFord• 3 years ago

It already says at the beginning when they were received (not mentioning potential others).

It's definitely unnatural to say again 'as allowed by the ones received in those months we already mentioned'.

smeagull• 3 years ago

I always wondered why you couldn't get all your subpoenas passed through an intermediary who is instructed to post about them before reading them.

jchook• 3 years ago

“The lack of a non-disclosure order”

dkga• 3 years ago

Seems like the non-disclosure order did not say anything preventing them from disclosing said non-disclosure order itself

aaroninsf• 3 years ago

https://www.aclu.org/other/national-security-letters

throw_a_grenade• 3 years ago

So just yesterday PyPI announced they're retiring cryptographic signatures: https://news.ycombinator.com/item?id=36044543.

It's hard to keep those things separated. I would very much like the code submitted to PyPI be protected end-to-end by cryptographic signatures, when PyPI has either no resources, or no spine to stand up to a government. Any signatures, even PGP, which should be in place until someone provides better mechanism.

stjohnswarts2• 3 years ago

I don't have a problem with this as it was 5 particular users and not "give us all the data for for all your users". They didn't really have much of a choice. I don't think they would have had a choice in any of the 5 eyes countries or their allies

etaioinshrdlu2• 3 years ago

Amazing how upset users here get over the very reasonable response to very normal police work.

throwaway_131403• 3 years ago

Agreed - how else was the DOJ supposed to do their job? They clearly need the data for an investigation. No need for PyPI to give information about how current users can alter their accounts to thwart future requests.

sneak3• 3 years ago

Normal police work doesn't go fishing for the IP addresses (potentially millions of users) of everyone who downloaded a package.

"IP download logs of any Python Package Index (PyPI) packages uploaded by..." given usernames

Do you feel the same way if the cops are receiving the IPs of everyone who downloaded yt-dlp? IP addresses and timestamps resolve to physical locations and oftentimes street addresses.

buzzscale• 3 years ago

That doesn't make any sense though. What benefit would DOJ get from getting the IP address of everyone who downloaded ytp-dlp? They aren't the enforcement arm of google's terms of service, which is a civil matter.

Even if they were, and the DOJ was going for a dragnet operation to go after tools that could potentially infringe terms of service of big corporations, they would go after every tool and every fork. Not just 1 package. But again, what court would allow such action and why?

If I was in the DOJ and was investigating a malicious package uploaded to PyPI, I would ask for the IP's of the downloaders to see if the uploaders dun goofed and downloaded their package shortly after uploading off VPN. Or to find out if any major corporations were impacted by downloading the malicious package and to inform them.

ranger_danger• 3 years ago

In the US at least, it has been ruled that an IP address is not sufficient evidence to link activity to any particular person. You could have been hacked for example.

etaioinshrdlu2• 3 years ago

(Deleted comment as it was wrongly assuming bias)

account426• 3 years ago

Exactly. This is like the police going to a store with a list of suppliers and demanding personal data of everyone who bought any of those suppliers' products. That's well beyond "normal" but somehow for digital data its ok?

Vervious2• 3 years ago

Yeah, I feel like this crowd sometimes forgets that the department of justice exists first and foremost to keep us safe.

With PyPi hosting a ton of malicious packages and malware, certainly I am not morally opposed.

winrid2• 3 years ago

Same with the dozen street cameras at every intersection in China, right? Right? :)

smolder• 3 years ago

You could reasonably argue it exists foremost to keep wealthy, well-connected people and organizations safe, and to punish their adversaries.

NelsonMinar• 3 years ago

Total speculation on my part but PyPI hosts yt-dlp, the unauthorized video downloader. https://pypi.org/project/yt-dlp/

vore3• 3 years ago

I would think the government has bigger fish to fry than to spend time subpoenaing yt-dlp.

dual_dingo• 3 years ago

Not a US cititzen, but "The government" is a wide term and any law enforcement agency would fit this, including the ones that are responsible to deal with things like copyright enforcement - that's exactly the type of fish they exist to fry ...

sp332• 3 years ago

While copyright infringement is usually a civil matter, there are times the DoJ gets involved. They even got a guy sentenced to jail for it in 2018. https://www.justice.gov/usao-ndga/pr/owner-sharebeastcom-sen...

throwaway09223• 3 years ago

Google is a pretty big fish themselves.

What usually happens is the large corporation lays out a case like "yt-dlp is responsible for billions in damages" and they press the DOJ to investigate and prosecute.

sam0x17• 3 years ago

One would think that yes.... but this is the U.S. :/

Sparkyte• 3 years ago

I wouldn't be surprised if it was more of AI based impersonation stuff. AI in the government is big because people can use it impersonate people as a form of identity fraud.

lazide6• 3 years ago

The FBI has it's own 'copyright enforcement' division who has as their sole job enforcing copyright, and has it's own dedicated funding

[https://archives.fbi.gov/archives/news/testimony/intellectua...]

heavyset_go4• 3 years ago

The FBI does sting operations to arrest homeless people who sell bootleg movies and bags on the street.

astrea• 3 years ago

Amazingly hilarious comment given the history of copyright prosecution.

WhyNotHugo• 3 years ago

Unlikely, due to:

"Records of all Python Package Index (PyPI) packages uploaded by..." given usernames
"IP download logs of any Python Package Index (PyPI) packages uploaded by..." given usernames

I don't think they'd want a list of packages uploaded by a given user if they were after yt-dlp devs. They'd be asking for a list of maintainers of a given package.

ed25519FUUU• 3 years ago

If yt-dlp was illegal the first thing they’d do is a takedown request, not a subpoena but leave it online.

heavyset_go4• 3 years ago

That's if they want to go down the DMCA route and aren't using a subpoena in a different case.

NoZebra120vClip2• 3 years ago

But what if they want records on everyone who downloaded a tool?

phkahler• 3 years ago

Thanks, I was wondering what it might be about. That makes some sense.

schleck8• 3 years ago

No it doesn't. Noone at the government level gives a shit about a youtube downloader package, typosquatting would be way more likely. Pypi is riddled with malware AFAIK, they don't really moderate it.

slenk2• 3 years ago

yt-dlp is everywhere - why would they go after pypi and not the source at https://github.com/yt-dlp/yt-dlp

heavyset_go4• 3 years ago

Not that I necessarily believe this is happening, but this can be a case of making an example out of anyone that wants to distribute the software they wanted censored via chilling effects.

not2b• 3 years ago

It seems much more likely that some typosquatter managed to compromise the security of government sites by uploading malware, and Uncle Sam wants to catch the culprit.

stonepresto3• 3 years ago

Or they're chasing threat actors, ransom groups, etc as part of a larger investigation.

CarbonCycles• 3 years ago

What an odd article and release statement. It’s almost as if they’re signaling w-out literally signaling the parties of interest.

Surprised the doj didn’t issue any gag orders.

throwaway_131403• 3 years ago

Exactly. I guess the transparency is nice but at what point are you potentially helping someone cover their tracks who may or may not actually deserve that help?

rossdavidh2• 3 years ago

One gets the impression that this was an artfully crafted way around the specifics of the gag order, to disclose whatever wasn't specifically prohibited by it. IANAL.

whimsicalism• 3 years ago

as allowed by the lack of a non-disclosure order associated with the subpoenas received in March and April 2023.

Yeah no way they haven't had other subpoenas then.

sneak3• 3 years ago

"IP download logs of any Python Package Index (PyPI) packages uploaded by..." given usernames

This is way overbroad. The fact that a judge granted this is very bad.

kjs32• 3 years ago

We get "please provide the logged IP addresses of user X" subpoenas on a weekly if not daily basis. Which law school did you go to so I can tell our corp counsel they've been doing it wrong and stop asking?

robryk• 3 years ago

Note that GP complains not about the request for IP addresses of user X, but the request for IP addresses of anyone who downloaded content uploaded by user X.

tw-0981230981• 3 years ago

You should re-read the quote. This was not a request for the IP addresses of the users in question, but for the everyone that downloaded any packages uploaded by those users.

duskwuff• 3 years ago

It's hard to say that it's "overbroad" without knowing the details of the situation.

It's not hard at all, on the other hand, to imagine situations where this would be a reasonable request. Probably the most obvious would be if the packages contained material which was illegal to possess or distribute (like CSAM). Another would be if the packages were being used as part of a malware C&C operation -- knowing what IP addresses downloaded the packages would aid in determining the scope of the campaign.

hajimuz• 3 years ago

Dude I thought it’s Chinese Gov. Hey, America!!!

junon5• 3 years ago

Good on the PyPi folks. This is an incredibly well done disclosure, an example to be sure.

jupp0r• 3 years ago

"9. IP download logs of any Python Package Index (PyPI) packages uploaded by the given usernames"

This was the point where I was wondering if this is really about some malicious packages or something more along the lines of copyright infringement software.

femto113• 3 years ago

This definitely seems like a significant element of the ask, but for any popular package a list of all the downloaders would be pretty overwhelming in size (and I think of very limited utility). I'm guessing that some versions of some more obscure package(s) were identified as being used in an attack and they're either trying to identify potential attackers or other victims (or both) of that attack.

From a 2021 article[1] about packages used to deliver malware "we have alerted PyPI about the existence of the malicious packages which promptly removed them. Based on data from pepy.tech, we estimate the malicious packages were downloaded about 30,000 times."

For comparison yt-dlp has tens of millions of total downloads and gets downloaded over 70,000 times every day [2]

[1] https://jfrog.com/blog/malicious-pypi-packages-stealing-cred...

[2] https://pepy.tech/project/yt-dlp

rektide• 3 years ago

It'd be lovely to see better patterns emerge to aggregate and/or anonymize data.

Great respect for the response. Reevaluating data retention is a great move.

rendx• 3 years ago

Way too much unnecessary data collected and stored as usual. But one of the best transparency reports ever. Thanks PyPI!

dvt• 3 years ago

Most likely caused by phishing, ransomware, or (unlikely) crypto mining. I'd bet someone from some agency had credentials leaked due to a malicious package. Honestly, PyPI is stuck between a rock and a hard place, but having something like a "verified" badge (where someone's real identity is tied to it) for certain packages would go a long way to ensure some level of security.

The problem gets a bit hairier when dealing with dependency chains, however.

tgbugs• 3 years ago

One theory that I don't see mentioned yet is that someone used an upload to pypi to exfiltrate data or simply as a way to upload arbitrary data somewhere. In a sense pypi is just a file hosting service, so it could have nothing to do with any actual python projects at all.

rocqua5• 3 years ago

Interesting approach to data exfil. Though it seems predictable that exactly this kind of subpoena would be issued. If you can predict it, you can probably mitigate it.

Which means the subpoena would only be useful if the criminals made an opsec mistake. That is generally how most sophisticated criminals get caught, but here it feels like anyone inventive enough to try will probably also be prudent enough not to leave a trail.

scrum-treats• 3 years ago

It would be great to see this for VS Code extensions as well.

BrandoElFollito3• 3 years ago

I wonder why such organizations that hold critical data for the community at large do not use an international canary system.

Should one of the countries issue an order, the ones outside of its jurisdiction can openly disclose the information. Say if the US forces the US entity to not do something, the French one sees it and can warn all users.

detaro2• 3 years ago

"I've been ordered not to tell the details, but I know you will publish them, so I'm going to tell you the details" is not going to be taken as "obeyed the order" by law enforcement or courts.

BrandoElFollito3• 3 years ago

Sorry but I think I do not understand (English is not my first language). Who would be in trouble?

In case anything happens with the content of the service, the detail of the changes would be made clear by someone outside the jurisdiction.

A typical example is TrueCrypt that, one day, changed their page to say to use something else instead of their product.

If the code was shared between several countries, the others could simply publish that this and that was changes out of band, and that it means that the code is now positively unsafe.

zerealshadowban• 3 years ago

They log too much data about their users.

So they should promptly update their policies to a) stop logging so much, b) delete all past logs, and c) sharply limit the span of time until deletion of whatever logs they decide they really need to track for internal needs.

They should avoid logging, and rapidly rotate logs, to thwart future subpoenas from the total surveillance state.

takeda2• 3 years ago

For the kind of service they are providing I think the logging is appropriate.

I mean if DOJ is interested in PyPI logs the only reason I could think of, is if it was used as a supply chain vector into breaking in into other organizations.

alwayslikethis2• 3 years ago

or locating the people responsible for DRM breakers like youtube-dl? Keeping all the data makes you susceptible to subpoenas like this, which costs money to comply with. There is no reason to keep any data that isn't necessary to the service.

dpifke4• 3 years ago

PyPI used to be somewhat decentralized, with multiple independent organizations (in multiple legal jurisdictions) running mirrors.

For $REASONS they've decided Fastly should be the only official mirror. This makes it a one-stop shop for malicious governments.

donaldstufft7• 3 years ago

This post is pretty misleading.

PyPI still fully supports mirrors (though it is becoming increasingly hard to run a full mirror of PyPI, last I looked a full copy of PyPI is about 30TB).

The only thing we ever removed was designating any particular mirror as official and an auto discovery protocol that was quite frankly extremely insecure and slow. That worked by giving every single mirror that wanted to be an "official" mirror for auto discovery a subdomain of `pypi.python.org`, labeled {a-z}.pypi.python.org. A client would determine what mirrors were available by querying last.pypi.python.org, which was a CNAME pointing to the last letter that we had assigned, that would tell it how many mirrors there were, then they could work backwards from that letter. So if the CNAME pointed to c.pypi.python.org, the client would know that a, b, and c existed.

Immediately you should be able to see a few problems with this:

- It is grossly insecure. Subdomains of a domain can set cookies on the parent domain, depending on ~things~ they can also read cookies.

- It does not scale past having 26 mirrors.

- It does not support removing a mirror, there can be no gaps in the letters.

So we needed to remove that auto discovery mechanism, which raised the question of what, if anything, we should replace it with?

Well at the time we had only ever made it up to g.pypi.python.org. So there was only 7 total mirrors that ever asked to become an official mirror. To my knowledge we never reused a letter, if a mirror went away we would just point the mirror back at the main PyPI instance. I don't remember exactly, but my email references there being only 4 mirrors left.

From my memory at the time, most of those 4 mirrors were regularly hours or days behind PyPI, would regularly go offline, etc.

But again, we never stopped anyone from running a mirror, we just removed the auto discovery mechanism and required them get their own domain name. We even linked to a third party site that would index all of the servers and keep track of how "fresh" they were, and other stats (at least until that site went away).

Running a mirror of PyPI is a non trivial undertaking, and most people simply don't want to do that. We never had many mirrors of PyPI running, and as it turns out once we improved PyPI most people decided they simply didn't care to use a mirror and preferred to just use PyPI, but still to this day we support anyone to mirror us.

chatmasta2• 3 years ago

PyPi is in a tough spot because they're also getting hit with an onslaught of malicious packages, which got to such a bad point they had to disable signups. How do they mitigate that kind of activity without logging basic metadata like the IP address that published a package? Also, as a user of PyPi, wouldn't you prefer that a malicious package is at least _somewhat_ traceable to an attacker? Of course most would be behind a VPN but it's better than nothing (or maybe it's not, depending on the tradeoff).

Note that the blog post doesn't say they handed the entire database over to the feds. They received three warrants scoped to specific packages and returned only the data they had available that was associated with those packages.

ginto3• 3 years ago

How do they mitigate that kind of activity without logging basic metadata like the IP address that published a package

An effective mitigation for abuse/spam is to increase its cost. One example is to require payment.

pluto_modadic• 3 years ago

This is for package management. I want the supply chain to be secure and would rather know when something unusual happens. Not logging that data would be irresponsible on PyPi's part.

oxguy3• 3 years ago

Which data are excessive? Everything described in the post seemed totally reasonable to me.

einpoklum2• 3 years ago

Here is what I consider the key section:

The privacy of PyPI users is of utmost concern to PSF and the PyPI Administrators, and we are committed to protecting user data from disclosure whenever possible. In this case, however, PSF determined with the advice of counsel that our only course of action was to provide the requested data. I, as Director of Infrastructure of the Python Software Foundation, fulfilled the requests in consultation with PSF's counsel.

The first part of this section contradicts all of the rest. If user data privacy is of "utmost concern", then it is a concern above fulfilling legal obligations under US law. Plus, such supposed obligations must be staunchly fought before even considering whether or not to observe them. So, in fact, user data privacy is a minor concern for the Python Software Foundation, while swift prostration towards the US federal state is what's of utmost concern.

Of course, they almost admit it themselves. If we carefully read the second clause, they don't say "we're committed to protecting user data from disclosure", but - the "we're committed... when possible". So, they're saying that if they believe it isn't possible to protect, they have _no_ commitment to try their utmost to protect. i.e. when they see fit, user data protection is _not_ a concern at all. ... ok, ok, it is a public relations concern.

ChrisMarshallNY• 3 years ago

That's an excellent transparency report.

jehb2• 3 years ago

Suggestion: Start slipping unique URLs into the "hidden" backend fields of systems where you'd like to know if your data was breached, improperly used, or handed over to a three letter agency.

Suddenly getting hits at mydomain.com/[uuid]? At least you know somebody has looked at the data, or at the very least fed it through some processing tool that is extracting and visiting the URLs.

mmsc• 3 years ago

This is called a canary and can be used in so many places: https://blog.thinkst.com/2022/09/sensitive-command-token-so-...

austinjp2• 3 years ago

I'm pretty sure I've seen a SaaS that does this, but I can't remember the name.

bl4ckneon• 3 years ago

"Thinkst canary" I think you're thinking about. https://canary.tools/

tailspin2019• 3 years ago

https://canarytokens.org

krick2• 3 years ago

How do you suggest it should've been used in this case? As a PyPI username (obviously pointless), or what?

jehb2• 3 years ago

Good question. I'm not familiar with what fields might be be collected on the PyPi backend. But the email address field alone could be enough, if you set up a wildcard DNS and made your account email something@[uuid].mydomain.com, and looked for any http traffic to that subdomain.

__MatrixMan__• 3 years ago

Kudos to PyPI for handling this professionally.

That said, I think we should be working towards a world where they're unnecessary. As a middle party to what ought to be a developer/developer trust relationship, they're attack surface that threatens depender sometimes and dependee other times.

Going peer-to-peer will be less convenient, but worth the investment in the long run.

tru3_power• 3 years ago

Is this related to that Microsoft disclosure?

svaha1728• 3 years ago

If it takes subpoenas to get package management fixed in Python so be it. Can the Rust Crates.io team take over Python management as well?

b33j0r• 3 years ago

Why don’t nerds get the same rights?

According to US news over the past 3-4 years, you can just ignore subpoenas, then get a contributor job on a cable news network. Bonus points, the more you flout the law as arrogantly as possible ;p

gjmacd• 3 years ago

I would point to Jim Jordan and all the other Republicans after January 6th who didn't honor a subpoena and toss them in the trash. Nobody in our government honors them, why should we in the private sector? What's going to happen, they going to raid offices and get a bunch of PC's and books?

dpifke4• 3 years ago

Being reminded that PyPI is a target for law enforcement makes me even more irked that they've removed end-to-end package signing without providing a replacement[0].

PGP signatures—even though rarely used—would allow someone to verify that a signed package was not modified by PyPI after being uploaded by its original author.

Without any sort of signing mechanism, we have to trust the U.S. Government to never demand that PyPI insert a backdoor, via a National Security Letter, FISA court order, or other kangaroo court process. Good luck with that.

The existing PGP signing mechanism had usability issues and security footguns[1], but was better than nothing. It's a shame they didn't roll out a more usable and secure alternative before removing the existing functionality.

[0] https://news.ycombinator.com/item?id=36044543

[1] https://blog.yossarian.net/2023/05/21/PGP-signatures-on-PyPI...

wokwokwok• 3 years ago

If you want to start with tinfoil hat theories, think about this:

The PGP signatures were removed, nominally because few people used them. ...but the timing of the removals is coincidental, no?

"You need to have a backdoor that lets us see who's downloading what packages and let us inject custom code to particular targets"

"That's technically impossible because of..."

"Here is a court order. Implementation is your problem. You're not allowed to tell anyone you even received a court order."

"...well, I guess signed packages have to go then..."

I don't actually believe that, since PGP signing was frankly, barely used and really there's hardly any meaningful difference between a PGP you can't verify (which was most of them) and not having it; in fact the illusion of security is probably worse than not having it at all.

...but still. As you say. It sucks there's no meaningful replacement for it.

donaldstufft7• 3 years ago

I would resign from PyPI before I ever allowed a backdoor to be installed.

I haven't explicitly asked, but I would be very surprised if any of the other PyPI admins felt differently.

woodruffw9• 3 years ago

I'm the author of that post. There is absolutely no meaningful sense in which PyPI's previous PGP support was (or ever did) provide end-to-end package signing. At the absolute most, when used correctly (which, overwhelmingly, it was not), it provided one half of package signing.

The other half (key retrieval and identity binding) was never provided, because PGP as an ecosystem made doing so intractable. It was not better than nothing, because it was nothing; anything you could have done with it can be done with your own sidecar signatures.

gardnr• 3 years ago

One half is better than nothing. Even if it just made users wonder what it was, it was better than nothing.

brabel3• 3 years ago

PGP didn't make it intractable, the problem is itself intractable... you're referring to the public key infrastructure (PKI)[1] problem, which many have tried to solve and failed.

PGP can use the only known solution to the problem, which is letting several key servers be configured by the user to import keys (which can then be verified by checking the key fingerprint on another source which is "trusted", like the publisher's own website).

You can still import keys by physically exchanging trusted keys with others (so called Key Signing Party[2]) but that obviously cannot scale... or using any innovative method you come up with, but no one has found a bullet proof way to do this that's usable.

But saying PGP only solves half the problem is wrong. It solves one problem: that of how to verify a publisher's artifacts were not modified, which is valuable.

The next problem to solve is how to obtain and vet public keys from publishers. The solution could work somewhat like TLS certificates (with certificate authorities playing the role of trusted key servers) or using blockchain (perhaps a rare problem for which blockchain could actually be helpful) but both of these bring their own issues with them. If you know of a better solution, though, do bring it up instead of throwing the bathwater out with the "baby"!

[1] https://en.wikipedia.org/wiki/Public_key_infrastructure

[2] https://en.wikipedia.org/wiki/Key_signing_party

Hackbraten• 3 years ago

If those agencies can order PyPI to insert a backdoor, wouldn’t they be able to coerce keyservers into updating the public key at the same time?

froh• 3 years ago

the original uploaders key? without anybody noticing? I don't think so.

upofadown2• 3 years ago

Trust in PGP land is end to end. The keyservers don't matter. They are only a place to pick up keys. Your software verifies that the key is unchanged in that the fingerprint is unchanged. Otherwise it is treated as a separate key. Dead simple.

The confusion here comes from the confusion in the PyPI article about PGP. The article complained that many keys could not be found on keyservers as if that mattered.

The Debian web of trust is a good example of how this stuff actually works. Before you can submit packages to Debian you have to get an existing Debian developer to sign your PGP key. In Debian the trust flows downward from older developers to newer developers.

dhx• 3 years ago

Perhaps look at Gentoo's model of a single monolithic Git repository. It is possibly the largest and most distributed Merkle tree of software distribution signatures in existence. It is updated a few times every hour by a diverse community and each commit has to be GPG signed so you have the opportunity to verify signatures by looking up developer websites, slides from FOSS conferences, etc to confirm whether the keys have been widely published.

There are some caveats:

* Avoid -9999 packages as you won't get any guarantee of authenticity of whatever will be obtained from the upstream repository, other than whatever trust you place in a X.509 certificate that in all likelihood is controlled by either Microsoft (GitHub) or otherwise accessible to Amazon, Google, etc by nature of common open source project hosting arrangements.

* When syncing your local repository, verify all changes since your last sync. This could be as simple as syncing to a point n-days ago, after which numerous developers you know have signed more recent commits on top (you at least know those developers have been impacted too if the whole repository was compromised and the compromise is now on the public record).

* You don't really know how many people are using the packages you care about, and thus how many other people across the world are also exposed to (and possibly reporting problems with) signatures that Gentoo developers have committed.

In addition to relying on existing sources such as the Gentoo Git repository, an additional way to build trust is setting up software "looking glass" tools in different jurisdictions to check that software downloaded from different carriers in different jurisdictions are all the same.

At least with these measures the attacker has to compromise everyone and make this compromise a public record, rather than just silently compromise one target.

WiSaGaN• 3 years ago

How difficult is it for the US government to supply chain attack other countries that are using popular Python packages such as PyTorch given their power over these kind of global software infrastructure under US internal law? Are we able to assess it given there are gag orders as well?

asne113• 3 years ago

I keep seeing people trying to assure other readers that the recipients of these subpoenas have some recourse to appeal.

This is not the case if the subpoena is issued by the FISA court, otherwise known as "the court of no rejection."

dragonwriter10• 3 years ago

I keep seeing people trying to assure other readers that the recipients of these subpoenas have some recourse to appeal.

Because they do, in regular federal courts.

This is not the case if the subpoena is issued by the FISA court

The Foreign Intelligence Surveillance Court doesn’t (and courts in general don’t) issue subpoenas.

asne113• 3 years ago

The Foreign Intelligence Surveillance Court doesn’t (and courts in general don’t) issue subpoenas.

This sounds like a matter of technicality, no? Whatever you call it, what they issue has the effect of a subpoena that you can neither appeal nor speak about.

My point is not that this is what's going on here, but that the right to appeal is not generally a given.

casey2• 3 years ago

How come when PyPI hosts unwanted malware they get subpoenaed but when Apple or Microsoft or anyone else with a big team of lawyers distributes auto-installing "updates" designed to harm/scam users the DOJ is silent?

phendrenad2• 3 years ago

An important aspect here is the fact that subpoenaes only share information one way. You are asked to provide information, and you get zero information about why. There is a simpler way to get information: Contact the person you want information from and make the case that they should give you the information. Using a subpoenae seems like something a government would do when they're doing something shady and don't want you to know about it.

paradox242• 3 years ago

When your platform is the source of malware used in supply chain attacks you gotta expect the Feds are going to come knocking eventually.

HeavyStorm• 3 years ago

There were lots of specifics asks from the court. What if Pypi didn't had those data, like IP addresses? I don't remember ever recording those in a longstanding storage.

morkalork• 3 years ago

Is there a way to be only contactable in public? Like say a blind man standing in a town square, where any subpoena would have to be read out loud to them in earshot of anyone walking by? Is such a thing possible, digitally?

seventytwo• 3 years ago

Any bets on this being part of a broader investigation into software supply-chain attacks?

krick2• 3 years ago

I'd say nothing but nickname and the list of packages uploaded (which should be public in the first place) should've been stored anyway.

It immediately reminded me, that PyPI content is really trash as it is because of all the squatting and pointless unfinished toy-projects, and whatever they are logging clearly doesn't help, but I think that big problem for PyPI is one seemingly minor detail: lack of namespaces (as in packagist.org of pkg.go.dev). It is not a solution for all sorts of malicious behaviour, of course, but it really makes things much easier. It doesn't solve typosquatting and such, but, honestly, neither does the current system, obviously. And at least it allows to keep the actual package names semantic. And which one of countless "*/time" libraries you wanna get you just kinda have to decide separately, using the number of starts on the github as a reference and carefully copy-pastying the id to your requirements.txt

The same issue I have with Cargo. I mean, really, isn't it obvious that making users compete for better project names just makes everything shit?

fijiaarone5• 3 years ago

It’s ok, this government is perfect.

Liberonostrud• 3 years ago

How can you pronounce this word?

Aeolun2• 3 years ago

Kind makes me feel like a lot of these services should just _not_ be hosted in the US? If we'd have this hosted in Germany or Sweden, would the government be so casually requesting data from these registries?

mschuster912• 3 years ago

PyPI retains records of all changes to projects on the index, and has since 2002-11-01 17:11:36 UTC.

Wow, I had zero idea how old Python is.

dragonwriter10• 3 years ago

Python was released in 1991 with 1.0 in 1994, PyPI is newer (the date listed there appears to be when it went online.)

jwilk2• 3 years ago

https://www.pypa.io/en/latest/history.html#before-2013 says PyPI was launched in 2003.

dark-star• 3 years ago

Strange that they didn't mention in 3b for how long they record IP addresses and logins. One has to assume that they keep this data indefinitely, even though they don't need it except for such subpoenas

account426• 3 years ago

PyPI retains records of all changes to projects on the index, and has since 2002-11-01 17:11:36 UTC.
submitted_from | IP Address
PyPI retains records of critical user events including account creation, emails sent, email address changes, logins, and login failures. See this list for the comprehensive set of events recorded.
ip_address_string | IP Address

Retaining IP addresses for over a decade doesn't sound GDPR-compliant.

bogwog• 3 years ago

It’s nice that they’re committed to user privacy, and this post really gives me confidence that my privacy will be reasonably protected.

…but why is that a goal for PyPi? As a publisher of packages, it’s a nice-to-have, but as an end user it’s kind of scary. I don’t want to use software packages published by anonymous and potentially unaccountable people. That’s probably why they have so many malicious packages.

Maybe you live in an oppressive regime who will imprison/murder you for publishing some code; ok, but that’s an outlier, and there are a lot of ways to get around that situation.

I just don’t see the benefit of privacy in this situation? Is it just to reduce the administrative overhead of collecting/verifying identity info? I’m genuinely curious to learn about a realistic use case that justifies the risks to all users.

I know you can self host your own package index, but very few users have the resources to do that.

kortex• 3 years ago

I think largely because the prerogative is on the code author to reveal as little or as much about themselves, and the prerogative of library users is to sufficiently vet a package. If folks want to publish code pseudonymously, and folks want to use that code, as long as it's not abusive, what's to stop them? You can achieve basically the same effect with github, gitlab, or even plain self-hosted HTTP packages (pip just uses a convention for listing packages in a dir, any HTTP file host can be a package server), without PyPI.

I actually think the larger problem is Python's reliance on imperative code that executes at install time. Yeah you can use pip --download and extract it yourself, but folks rarely do that.

miga• 3 years ago

Excellent example of how difficult it is to design privacy-friendly service.

Any information that can be subpoenae-ed can also be hacked away.

berkes• 3 years ago

Any information that can be subpoenae-ed

What if you operate from e.g. Iceland, Switzerland or even many EU states? Can a US Gov. agency still subpoena you?

kgbcia• 3 years ago

Probably someone made a YouTube or Tor package so the state is after them.

stuaxo• 3 years ago

The list of things they asked for seems to be basically "everything" - seems like a dragnet approach.

PyPI Was Subpoenaed

Related Stories