On Python security amidst recent Rails/YAML vulnerabilities

http://nedbatchelder.com/blog/201302/war_is_peace.html?

by whalesalad • 13 years ago

161 44 13 years ago

zobzu2• 13 years ago

and don't forget to sign your packages, because vulnerabilities will always happen anyway, and if those compromise a distribution point, its hard to authenticate those said packages.

__alexs• 13 years ago

If PyPi was compromised like RubyGems I'm not sure they'd be more able to reliably recover from it quickly either.

PyPi does support package signing (with GPG) but pip doesn't support signature verification and hardly any packages are actually signed anyway. It's actually probably more secure to load your python packages off of specific commits on a public git repo over HTTPS right now. (Except that pip also doesn't validate HTTPS certificates either...) And if you are lucky enough to be using a package that is signed, establishing a WOT with the author to validate their cert might not be easy.

It's not like people aren't working on this stuff though. https://www.updateframework.com/ have a 'secure' (the upstream ins't obviously) PyPi mirror and the PEP427 Wheel http://wheel.readthedocs.org/en/latest/ format seems to be giving security more consideration than previous attempts at Python packaging have.

wyuenho• 13 years ago

Please +1 this pip ticket if you feel that supporting TLS cert and GPG verification should be given the highest priority.

https://github.com/pypa/pip/issues/425

I think it's paramount that pip gets this done right now. Installing code directly from PyPI is extremely scary now and has been for years...

zobzu2• 13 years ago

That's exactly why I posted this comment. I'm fully aware of the state of the things :)

I'm hoping this starts to raise enough attention for people to actually fix these things up. The patches for supports are around, and btw, pip now also supports HTTPS properly. (yeah, the experimental branch this is actually merged in)

But, people in general have to understand the benefit AND figure out it actually serves a purpose (because as always with security, nobody gives a crap 'til someone gets compromised in a terrible way). That's also because all the library makers (which is virtually everyone and their dog nowadays) have to actually get a proper gpg key, understand how it works, and actually sign their stuff. That's a major effort.

grey-area• 13 years ago

The comments on this post are really interesting ( save the first, which unfortunately stoops to much the sort of tribal content-free attacks we've seen on rails vuln news on HN recently). I particularly liked the long one from Nick Coghlan, and though it does seem there are still some worrying vulnerabilities in python, they are ahead of ruby in their packaging system at least in relying on signatures. keen to see ruby gems step up and take security equally seriously.

PommeDeTerre2• 13 years ago

Yes, Ruby and Ruby on Rails have received a lot of flak lately, but it is well-deserved and I don't think it is "tribal" in nature.

To many of us, these are just yet another set of tools in our very large toolbox. It's obvious that some tools are inherently better than others, however.

When I point out that Ruby on Rails or JavaScript have some serious inherent problems, it's not because I think that I belong to some Python "tribe" or the Perl "camp", for instance. It's because I'm doing rational, emotionally-detached analysis of certain pieces of software, and this analysis shows there to be serious problems with said software.

I think the same goes for the other people out there who have the courage to point out flaws with JavaScript, Ruby and related technologies. If anyone is acting "tribal", it's those who are so emotionally tied to a particular language or web development framework that they can't stand to hear legitimate concerns regarding important factors like security, performance, maintainability and reliability.

JshWright• 13 years ago

In my opinion, the first comment on the post isn't 'tribal' simply because it blindly attacks Ruby, but because it also blindly disregards the vulnerabilities in Python (which the author carefully and reasonably outlined).

byroot• 13 years ago

> emotionally-detached analysis of certain pieces of software

Really ? Ok so please expose us your analysis if you have one instead of repeating again an again your supposed conclusion without giving us any argument.

kisielk• 13 years ago

Yes the packaging system has signatures but unfortunately Python's SSL implementation doesn't verify certificates, nor do its packaging tools. The signatures come from the same server as the packages, so if someone MITMs the server you can still get hosed. There was a talk at last year's PyCon demonstrating this and these kinds of problems still have not been fixed.

reidrac• 13 years ago

I'm not sure if we're talking about a language problem here. The tools are there for you to use them in the proper way and there's always a chance that someone misunderstands how things work.

Perl Data::Dumper can be used with eval for serialization, but it is a bad idea. Just like using pickle in Python (most of the time, at least).

No matter what you do, there's always room for someone doing something stupid. So I rather have the tools.

Argorak• 13 years ago

Saying that it is a language problem might be true for that specific case. YAML#load and #dump do Object marshaling, which is always dangerous when the marshaled objects come from untrusted sources and parsed without a template. Similar techniques exist in almost all other languages and object instantiation attacks are nothing unheard of. So most of the bile is unwarranted, unless you are asking for bile back if something like this happens in your language. Everyone who is ranting about the security bugs of other projects clearly lacks the humility that Nick Coghlan is asking for in the comments.

But the reason why this specific one is very widespread is actually a cultural one: YAML was propagated as a very convenient serialization format despite the described property, especially as it was in stdlib very early. It turned into one of Rubys beloved conventions. E.g. some static website generators like Jekyll use YAML for meta data, called "front matter" (they use safe_yaml now, don't try), Rubygems used it to dump their specs, etc. Combine that with the fact that Rails activated certain parameter parsers without the users knowledge (by convention, again) and made everyone vulnerable and you have a recipe for desaster.

This is hard to fix, but thats the pain of suddenly being under attack. But instead of all the hate, members of other communities should take away these learnings and educate everyone they know that uses Ruby about these topics. In clear words, but without hate.

patio112• 13 years ago

I think load and dangerous_load is a wonderful paradigm for API design. It bakes security into the core use of your framework for non-security devs. It also makes the framework more productive for security professionals, since for an assessment you can grep for dangerous_load and focus your efforts on making sure those (rare) calls are hermetically sealed from user input rather than having to audit every possible use of the 100x more common load command. (That is where the Rails community is right now, and it sucks.)

This is similar to requiring a whitelist for mass assignment (rather than making a blacklist optional): it calls out in the code "These are our weak points! Check them carefully!"

thibaut_barrere3• 13 years ago

I suggest Rails people have a look at safe_yaml which (now) replaces YAML.load by a safe (safer?) version:

https://github.com/dtao/safe_yaml

(nb: I've enable it on apps without issues)

patio112• 13 years ago

I like that project but have been trying to avoid saying "Use this!" because I haven't personally audited it and am scared that Rails devs might adopt it and think it means it definitively resolves all of their issues for February, which it might not.

PommeDeTerre2• 13 years ago

This reminds me a lot of how the PHP crowd handles these kinds of problems.

So there's some piece of functionality, but security or some other obviously-important factor wasn't considered at all when it was initially "designed" and implemented. It's soon found to exhibit numerous problems, often including major security flaws.

Then a "safe" version of said functionality is offered. Yet for some reason (incompetence?) it has its own set of problems. And so the developers keep trying again and again, never seeming to get anywhere close to even a suitable solution.

An example is the mysql_escape_string() function, which was replaced by mysql_real_escape_string(), which has in turn been replaced by mysqli_escape_string(), mysqli::real_escape_string() and PDO::quote().

The end result is confusion, especially for novice users, or those coming back to PHP after some time. They don't know which of the several functions to use, or they're using older reference material that suggest the use of the faultiest of the functions.

It's better just to implement things properly the first time around, especially when security or data integrity, for example, are involved. Trying to hack on "safe" versions of functions, or even an entire "safe mode" is the wrong approach.

untog2• 13 years ago

I think load and dangerous_load is a wonderful paradigm for API design.

It is, but I think it ignores the reality of API design. Who would ever start with dangerous_load? If you knew it was dangerous, you'd probably fix it. No, you start with load(), then find out it's dangerous afterwards. The fix will be a breaking change, so you make safe_load().

"You should replace load()!", I hear you cry. But that removes backwards compatibility. And people don't update legacy software. So then old software becomes vulnerable to who knows how many other security problems because of it's obsolescence.

vog2• 13 years ago

> ... ignores the reality of API design. Who would ever start with dangerous_load?

When I wrote the Texcaller library, which simplifies compiling (La)TeX code, I disabled dangerous features such as "write18" from the very beginning.

I you don't take basic elements such as "secure/unsecure" variants into account, you're no really doing API design. You then just have a historically grown API, which is not necessarily bad in itself, but doesn't qualify to be called API design.

> But that removes backwards compatibility. And people don't update legacy software. So then old software becomes vulnerable to who knows how many other security problems because of it's obsolescence.

In that case, at least only old hard-to-update legacy software is affected, and not all the other (good, mostly well-written) software that is written today and in the future.

Of course, don't forget to increase the major version number, as for every API change that breaks backward compatibility.

ufo• 13 years ago

> Who would ever start with dangerous_load?

Haskellers. This is why you get lovely long and alarming names like `unsafePerformIO`

vq• 13 years ago

This is often the case in Haskell APIs, easily abusable or functions with hard to predict side-effects are usually prefixed with "unsafe", for example unsafeCoerce or unsafePerformIO.

tikhonj• 13 years ago

Another cool thing, from a security perspective, is that there is actually a tool (Safe Haskell[1]) that makes it easy to disallow unsafe behavior and operate a whitelist of trusted modules. So not only is unsafePerformIO well-named, it's also trivial to restrict its usage.

[1] http://www.haskell.org/ghc/docs/7.4.1/html/users_guide/safe-...

SilasX2• 13 years ago

So I guess the STD/STI prevention community was actually ahead of the computer security community in having the term "safer sex" replace "safe sex"?

nbm• 13 years ago

This "encode problems into names" approach is something I first ran into at Facebook.

When a particular function or class is found to be a problem (or when it is known to be a problem in advance), it is (re)named something like "foo_POTENTIAL_XSS_HOLE" or "FooMapNonStlCompliant" throughout the code base (using pfff or similar tool), and then the potentially long process of removing the problem can go forward with some confidence nobody will add to the problem while you're working on it.

pfff: https://github.com/facebook/pfff

speleding• 13 years ago

I realise you probably meant this for the more general case, but if you are building an API it should not accept YAML. If you are not building an API and are using YAML to de-serialze stuff you serialised yourself (i.e. use YAML as intended) then there is no issue anyway.

So having "dangerous_load" is not going to help much with YAML: if it's exposed to untrusted input then grep for "YAML" not for "dangerous_load".

vog2• 13 years ago

From the article:

> PyYAML has a .load() method and a .safe_load() method. Why do serialization implementers do this? If you must extend the format with dangerous features, provide them in the non-obvious method. Provide a .load() method and a .dangerous_load() method instead.

I think this is a very good advice that holds in general:

The default should never be the most feature-rich version, but the most safe version. This is also why you should generally prefer a whitelist approach over a blacklist approach. And this is why templating systems should perform escaping by default, forcing you to explicitly disable it, at concrete places, when including raw HTML.

gingerlime• 13 years ago

+1 for `dangerous_load`. If it was implemented this way, rather than `load` and `safe_load`, I doubt we would have seen vulnerabilities like the one in both tastypie and piston, the two leading API libraries for Django.

See https://www.djangoproject.com/weblog/2011/nov/01/piston-and-...

Whilst python might seem safer than the state in Ruby/Rails, it also has its history of vulnerabilities.

moneypenny• 13 years ago

If the original author is reading, those are George Orwell's "pithy maxims", not Allen Short's, but entertaining usage on getting a Big Brother reference into a security/trust context.

devinj• 13 years ago

George Orwell wrote them, but Allen Short was the person who applied them in that way to a security context. Much of the post is apparently greatly due to Allen Short, so he deserves some mention.

abecedarius• 13 years ago

For what it's worth, I applied "freedom is slavery" and "ignorance is strength" to programming back in the 90s in a rambly post on my website. I don't know if Allen ever saw it, and security wasn't much on my mind back then. (We're acquaintances, I admire him, and I'm glad to hear of this talk.)

maxerickson• 13 years ago

I think you are aggressively misunderstanding the attribution here.

The maxims being referenced are the ones about code ('input is an attack', 'small interfaces'). The pithy is writing them in terms of Orwell.

On Python security amidst recent Rails/YAML vulnerabilities

Related Stories