Finding secrets by decompiling Python bytecode in public repositories

https://blog.jse.li/posts/pyc/

by gilad • 6 years ago

165 51 6 years ago

blog.jse.li

maxeonyx• 6 years ago

This would be solved if python used an (OS-specific) cache directory for its .pyc files. I have always disliked .pyc files... here's a concrete reason!

Question: what does python do if it doesn't have write permission in the current working directory? Not write the cache?

eesmith5• 6 years ago

It already (since 3.2, as the article points out) uses a cache directory. Why would an OS-specific one help?

You can also set PYTHONPYCACHEPREFIX if you want to use 'a mirror directory tree at this path, instead of in __pycache__ directories within the source tree' - https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPY...

The PEP mentions your case at https://www.python.org/dev/peps/pep-3147/#case-5-read-only-f... . But honestly, I don't really follow the answer. I think it means "ignore creation and write failures."

deathanatos2• 6 years ago

Why would an OS-specific one help?

E.g., if you want to back up your home dir, and omit caches, since they can be regenerated. It's a lot easier if programs write their cache data to ~/.cache / $XDG_CACHE_HOME than if they intermix it / scatter it about.

globular-toast• 6 years ago

That env var is new in 3.8. I've been looking forward to it to stop docker writing root owned cache files back to a bind mounted file system. I might set it to $HOME/.cache/python globally as well.

pdonis• 6 years ago

what does python do if it doesn't have write permission in the current working directory? Not write the cache?

In the case where the interpreter can't find any place it has write access to to write the cache, it will not write it, yes. That means it will have to re-parse the source file into bytecode (and fail to write the bytecode to cache) every time it is loaded.

jbverschoor• 6 years ago

It makes more sense to use a second stream or named stream.

PeterisP• 6 years ago

That sounds like a recipe for even more people accidentally storing and distributing that bytecode without wanting to do so, because it won't be immediately visible that it's there, many tools don't show or highlight the existence of those streams.

I'd argue that it's more appropriate to follow the 'explicit is better than implicit' from the 'guidelines' of https://www.python.org/dev/peps/pep-0020/ .

shockinglytrue4• 6 years ago

Highly recommend "export PYTHONDONTWRITEBYTECODE=1" in your bashrc and just forget about it. Pyc files are still an important optimization on modern machines in some circumstances (especially with huge oft-restarted apps), but the autogeneration behaviour has always been a pain in the ass

The bulk of your pycs are generated during package install. What tends to remain in the usual case is a handful of files representing app code or similar.

the_jeremy• 6 years ago

Why do you highly recommend this? You can set git (or other VCS) settings to avoid committing .pyc files. They're not large files; I don't see the downside.

shockinglytrue4• 6 years ago

Difficulty with git basics is the least of your worries. For example pycs are fundamentally racy, it is quite possible to have a .py newer than a .pyc depending on how unlucky you were with an in-progress deployment, or some tool that never updated the .py timestamp. Python continues to execute the pyc even though the code changed, since the minimal benefit of the pyc would be rendered significantly inert should Python use any kind of strong check (instead of just comparing second-granularity stat() output) to ensure the cached bytecode matches the source. In this way, without your permission, Python silently plays undesirable code execution roulette with your computer every time it starts, for as long as you have the feature enabled.

I have lost count of the number of times I've seen someone lose an hour due to it. I can also count many instances of QA environments becoming inexplicably bricked by it. The correct fix for this requires opening the .py and hashing its content, at least doubling the amount of IO required to start a program. They were a great feature when parsing small files was noticeably slow, but this hasn't been true for almost 20 years.

It's therefore worth turning the question around: why do you think pyc files are useful?

BiteCode_dev• 6 years ago

When doing dev, there a be other problems with pyc files. On of them is deleting the original py file, but still being able to import the module because Python will import any pyc file it if exist.

So if performances allow you to do so, disabling them when you dev is not a bad idea, as long as you keep them enabled when you run CI.

IMO, when you dev, you should set PYTHONHASHSEED (random is predictible), PYTHONDEVMODE (verbose warnings + tooling + sys.flags.dev_mode=True) and PYTHONDONTWRITEBYTECODE (remove the need for cleaning them).

In CI and prod, you should make sure those are NOT set, and use PYTHONOPTIMIZE=2 (remove assert, __debug__ lines and docstring) if you trust your dependancies to be well written (which I would check by running tests in a pre-push hook).

oefrha5• 6 years ago

Just reading the script from TFA, it attempts to find secrets.pyc and decompile it, but doesn't even check if secrets.py is also in the repo. A glance at search results (I just used GitHub's web interface, didn't bother to run the code) tells me when secrets.pyc is committed, secrets.py comes with it at least the vast majority of time.

I guess the author did find cases where secrets.pyc is committed but secrets.py is not? It's hard to fathom how that could have happened (especially inside "organization" settings). Sounds like the result of absolute rookies in both Python and git following a tutorial with a step "add secrets.py to .gitignore" but unfortunately takes ignoring __pycache__ and ﹡.pyc for granted, which is too much to ask for some people.

it is very easy for an experienced programmer to accidentally commit their secrets

No, it doesn't take an experienced programmer to put __pycache__ and ﹡.pyc to global ignore, or use a gitignore boilerplate at project creation, or notice random unwanted files during code review.

alexchamberlain2• 6 years ago

I actually insist at work that most repositories don’t have a .gitignore; just setup a reasonable global one and you’re done. OSS or repos with a large number of contributors are generally exceptions here.

dastx• 6 years ago

Sure but reasonable for me is different from reasonable for you. In addition, I might miss files that are repo specific. Insisting repos don't have a .gitignore is terrible advice. It doesn't cost much to maintain it if at all.

orf2• 6 years ago

That sounds like poor advice with unclear rationale behind it. There are definitely project specific ignores that you’d want to set up, and if you’re working with projects with multiple different languages, “one gitignore to rule them all” fast becomes a mess.

Newky• 6 years ago

I think you are misunderstanding. The secret does not need to be hardcoded in the python file. If it's read in from an environment variable or some other external source, it will also be in the pyc

orf2• 6 years ago

That’s totally incorrect. .pyc files just contain a representation of the _code_ and not any values that don’t exist in the code.

So a snippet like “os.environ[‘my_super_secret’]” won’t contain anything else than the bytecode to fetch that environment variable.

oefrha5• 6 years ago

Of course not, that would mean env vars are hard-coded into byte code at compile time, which would be completely crazy. A pyc file is just a parsed series of op codes that the interpreter could dispatch directly, so that it doesn't have to parse source files every single time.

It's very easy to verify:

secrets.py:

  import os
  SECRET = os.getenv('SECRET')

Then

  $ python -m compileall secrets.py
  $ uncompyle6 __pycache__/secrets.cpython-38.pyc
  # uncompyle6 version 3.7.0
  # Python bytecode 3.8 (3413)
  # Decompiled from: Python 3.8.2 (default, Mar 10 2020, 12:58:02)
  # [Clang 11.0.0 (clang-1100.0.33.17)]
  # Embedded file name: secrets.py
  # Compiled at: ...
  # Size of source mod 2**32: 40 bytes
  import os
  SECRET = os.getenv('SECRET')
  # okay decompiling __pycache__/secrets.cpython-38.pyc

visarga• 6 years ago

Seems to me quite easy to forget ignoring __pycache__ and *pyc, have the secrets pushed to the repo, then never get to remove them from history.

oefrha5• 6 years ago

Fix your process then. Use a global ignore file. Add a language-specific gitignore boilerplate first thing you create a new project. Scan for files that don't belong in code review (do I even need to suggest this).

never get to remove them from history.

Scrubbing specific files from git history isn't hard.

s/__pycache__ and ﹡pyc/secrets.py/g and people will also commit it in. PEBCAK.

TomMarius• 6 years ago

I think I am an experienced developer (not Python) and this would never cross my mind.

progval• 6 years ago

When you review the list of changes (including added files), you notice the files that shouldn't be there; so you would see them before committing. And even if you forget, git also lists new files after calling "git commit"

thaumasiotes• 6 years ago

It would never cross your mind not to commit .pyc files to source control? They're not even source. Committing .pyc files is to Python what committing .o files is to C.

tucnak• 6 years ago

I read "Finding secrets by decompiling Python bytecode in public restrooms" by accident. It never occured to me that anyone would do THAT in there.

eat_veggies• 6 years ago

there's bytecode sharpied on the walls

scarface742• 6 years ago

Public Service Announcement.

While for the other secrets, I’ve got nothing, there is never a reason to have AWS secret keys in your code or in application specific configuration files.

Every AWS SDK will automatically read your keys from your config file in your home directory locally. Just run

  aws configure

When you run your code on EC2, Lambda or ECS, the same SDK’s will automatically get the keys associated with the attached role.

robbyt• 6 years ago

For others non AWS credentials, just use environment variables

lysium• 6 years ago

They have to be set up somewhere, though.

xvilka• 6 years ago

Radare2 also supports Python bytecode of diffent versions [1].

[1] https://github.com/radareorg/radare2/tree/master/libr/asm/ar...

neurostimulant2• 6 years ago

Another possible place to look for secrets is in public docker images. Bots are scanning github repos for secrets all the time, but what about dockerhub (and other docker images repositories)? I accidentally leaved a secret on my public docker image once and that's made me quite paranoid about it now.

syntheticcorp• 6 years ago

I scanned everything pushed to dockerhub for a few weeks but didn’t find too much interesting stuff showing up

Finding secrets by decompiling Python bytecode in public repositories

Related Stories