PyPI Migrated to New Infrastructure

https://mail.python.org/pipermail/distutils-sig/2014-January/023522.html

by donaldstufft • 12 years ago

52 18 12 years ago

mail.python.org

SEJeff• 12 years ago

They are using salt to configure their infrastructure now instead of chef, interesting!

https://github.com/saltstack/salt

jonesetc• 12 years ago

They felt dirty using a ruby project.

eberfreitas• 12 years ago

Ansible would be another awesome python solution to use in this case.

jbeja• 12 years ago

Why you have the notion that they despite ruby?

steder• 12 years ago

Being a Pythonista and working with Chef for the last year I think that this is makes some sense. We've been recently contemplating the same migration.

Of course, this case the number of servers being managed is low enough that there isn't really a going to be much of a performance difference between Chef or Salt so this is likely just the preference of the current maintainers.

rdtsc• 12 years ago

+1 for Salt. I really like it. I came to it not as a replacement of puppet or chef or other mature solutions but as a better "log in with ssh, run commands and dump some some config scripts" system. Ansible was another contender but Windows support in Salt won over (and Salt now also has ssh-only mode I hear).

stonesixone• 12 years ago

Their reason for using Salt (from the Distutils mailing list [1]):

"> Can you say a little about the choice to use Salt instead of Chef? I don't really care either way, but am just curious. Is it because Salt is written in Python, or were there other reasons (functionality, etc)?

"I’d need to ask Ernest to be sure, but I believe it was mostly that he was more familiar with it. The fact that it was written in Python was a bonus as well ;) I don’t think that there was anything that Chef was missing or that Salt had over Chef, just familiarity of the person who did most of the work. I’ll double check with Ernest to make sure there wasn’t another reason :)"

[1] https://mail.python.org/pipermail/distutils-sig/2014-January...

donaldstufft• 12 years ago

I have actually now asked Ernest, and that was basically the reason. Familiarity. He did most of the work and was more familiar so it enabled him to complete the work quicker. We still do use Chef on the python.org infra, just not for PyPI itself.

e12e4• 12 years ago

Well, great news for pypi. As for using salt -- I thought: "Well, I guess it's time I had another look, they probably have their security story together now, after all.".

So I have a look at the "Masterless Quickstart" and "Installation" pages [1,2] and find:

   wget -O - http://bootstrap.saltstack.org | sudo sh

And the server answering on 443 on that host doesn't have a certificate for that domain, or the bootstrap script (clearly it's not configured for a ssl vhost by that that name/id -- through SNI or otherwise).

So, just run some code downloaded over plain http as root on all my servers? Great plan!

edit: I think I'll stick with Ansible for now.

edit2: For those wanting a look at the salt states, from the thread[3]:

Where are the states stored?

https://github.com/python/pypi-salt

1: http://docs.saltstack.com/topics/tutorials/quickstart.html

2: http://docs.saltstack.com/topics/installation/index.html

3: https://mail.python.org/pipermail/distutils-sig/2014-January...

jyap• 12 years ago

On the Installation page you linked to it clearly mentions:

PLATFORM-SPECIFIC INSTALLATION INSTRUCTIONS

These guides go into detail how to install salt on a given platform.

A QuickStart is just that. Something which serves its purpose as a QuickStart. If you have security concerns then read the source code or test it out in a VM.

e12e4• 12 years ago

It doesn't change the fact that running code that's downloaded over http,unverified by any kind of signature, as root -- is a very bad idea, and just having it there in official documentation for package whose purpose is managing systems gives me a very poor impression of the projects security goals.

andreasvc2• 12 years ago

The alternative is:

1. get tarball

2. do full code audit

3. run code

However, almost no one will do step 2, so running code piped from wget is effectively equivalent with doing step 1 and 3. https provides some level of security, but a determined attacker can make sure you don't get warnings from their MITM attack, which gets us back to having to do step 2.

e12e4• 12 years ago

No. The alternative is to provide a gpg signature of the shell script, and instruct new users to verify the signature, before running the script. The script can then embed signature checks for any downloads.

The point isn't to make sure "everything is secure", the point is to avoid terrible advice on the fourth line of the documentation.

Another (IMHO not as good) alternative is to simply distribute the script over ssl.

A third alternative is for the instructions to show how to first get the file, then check that it is the right file -- but that implies that the reader is qualified to audit the shell script. That's a lot easier said than done. A signature would go a long way to allow the person installing to simply trust the salt project -- without a signature or an ssl distribution point, it doesn't help to trust the salt project: you have no idea what is being piped into the root shell.

And it doesn't fill me with confidence when the project advocates doing this -- that leads me to not trust the rest of the code as much, either.

As other has pointed out -- distribution via apt etc is also good -- as indicated in the platform specific instructions -- but the documentation lists the apt-platforms among those where the script has been tested.

Shish2k• 12 years ago

How to install ansible: sudo rm -rf /

Oh no! Now you can't use ansible either, because insecure installation instructions exist, and thus you are compelled to use them instead of the sensible alternatives :P

mrweasel• 12 years ago

I'm impressed how little infrastructure they where make due with previously. PyPI hasn't been too bad considering the backend.

PyPI Migrated to New Infrastructure

Related Stories