Show HN: Vex, a new way to run things in Python's virtualenvs

https://pypi.python.org/pypi/vex

by sashahart • 12 years ago

149 34 12 years ago

pypi.python.org

waitingkuo• 12 years ago

Cool! That's the tool I always need!

sashahart10• 12 years ago

This is based on my personal experience with tools for using virtualenv.

virtualenvwrapper is very powerful with all kinds of hooks but I just never used all that power and needed things like speed more. virtualenvwrapper provided a lazy loader but that would have various problems from time to time as you'll see on the issue tracker.

So then for a long time I just ran 'virtualenv' and source wherever/bin/activate and found that I really didn't miss virtualenvwrapper all that much, which was a surprise because everyone finds 'workon' invaluable, a position I do understand.

Later my wife taught herself virtualenv and then wrote like 5 lines of shell to do the same thing as virtualenvwrapper (the classic environment-modifying thing) which I found hilarious, and I wrote a version of that in zsh with a little more error checking to make me happy and just jammed it in my dotfiles. Then after some months of using that I decided this is nice enough to publish so I worked on rewriting it as portable shell functions with more error checking and was going to put it on PyPI.

Toward the end of writing that I was feeling pretty finished, when some last detail that I forget made me hacked off about modifying the current shell environment, so on a lark I wrote a prototype of the new idea of only modifying a spawned process's environment in about an hour (most of which was learning further about portable shell) and once it was working I looked at implementing more error checking for all the weird conditions that may arise in shell and I said, you know what, I have no reason not to do this in Python and totally cut the dependency on which shell is being used, who is going to use this without Python anyway?

Then I ended up figuring out how to do prompts and completion and pretty much had a publishable project, which was surprising.

It's still faster to directly run the virtualenv's Python or to use your own small shell function but while YMMV, for my uses virtualenvwrapper is a lot of code I will never use and I use vex every day.

Highlights of my todos: pydoc still doesn't work (the reasons for this are lame) and probably it will be nice to implement --make/--remove to substitute mkvirtualenv/rmvirtualenv (which would also let me dump a pydoc script in virtualenv's bin/ or Script\ dir to fix that problem...)

burntsushi2• 12 years ago

pydoc still doesn't work

Are you using `pydoc` on the command line? If so, try pdoc: https://github.com/BurntSushi/pdoc. I wrote it as a tool to replace epydoc (strictly automatic documentation), but it works on the command line just like pydoc. (It can also generate HTML documentation with cross module linking that isn't crap. See an example here: http://pdoc.burntsushi.net/nfldb)

sashahart10• 12 years ago

I do sometimes, more as a matter of muscle memory than volition. I'm going to make a try to use your tool as a replacement for a while and see if I can generate any issues for you :)

However, I still would like to eventually unbreak pydoc for users who might have bad habits like I do.

The issue is that virtualenv itself had an opportunity to fix this (with multiple patches submitted IIRC) and they decided to punt by implementing it as yet another shell function. So for me to unpunt without coupling to the shell, I pretty much have to dump a script into bin/. I guess maybe not that many people use pydoc anyway?

runiq• 12 years ago

everyone finds 'workon' invaluable, a position I do understand.

At first I wanted to praise virtualfish [1] and its ability to automatically activate a virtualenv when I cd into it, but then I realized that vex's way does indeed make more sense. What do we do in a virtualenv? We basically just run pip to do various things, or run the program (in case it's an application and not a library) or test suite.

So yeah, I think I'm gonna switch. The added portability is a nice bonus.

[1] https://github.com/adambrenecki/virtualfish

sashahart10• 12 years ago

Awesome, thank you for trying this. Please make a github issue or email me if you have any suggestions informed by the experience :)

edit: That one feature reminded me of Kenneth Reitz's autoenv (https://github.com/kennethreitz/autoenv).

I can't do anything to futz with cd, but it could be possible to reverse that and cd to a project directory when a virtualenv is activated, I am just not sure yet mechanically the right way to choose which one. Worth considering.

StavrosK2• 12 years ago

Am I the only one who activates the virtualenv by hand? I don't understand what workon and other things provide. These days, I just have a single virtualenv that gets sourced when my shell starts up, all my projects run in virtual machines (I mostly work with Django). Projects that conflict with others get their own virtualenv, but sourcing it once at the start of the day is still not that much hassle.

goblin89• 12 years ago

Well, I use the postactivate hook to set environment variables (for convenience) and connect to project-specific tmux session.

    export PROJECT=`cat $VIRTUAL_ENV/.project`
    export SRC=$PROJECT/src

    [ -z $TMUX ] && cdproject && envopen

Where envopen is my own script that launches tmux (basically tmux -f $TMUXCONF -L $ENVNAME attach -t $ENVNAME with a few checks).

This solution has a lot of flaws, though: it isn't robust and doesn't make sense for non-Python-based projects (my work is not limited to Django). In fact, I'm working on switching to dedicated VMs for my projects—looking for a nice solution currently. I suspect it may involve Vagrant, Docker, perhaps Chef.

acjohnson55• 12 years ago

Ha. I always thought something was unnecessarily awkward about virtualenv, but I couldn't quite put my finger on it. Virtualenvwrapper was an improvement in many ways, but its lack of portability could be a pain. Reading the docs, Vex makes perfect sense. We should use virtualenv basically the same way we sudo. It's kind of amazing it took this long for someone to do it. Thanks!

ProblemFactory• 12 years ago

Vex seems very convenient, but it is also possible to avoid the "magic" with just standard virtualenv.

If you provide the full path to the python binary in the virtualenv, it executes with the virtualenv loaded:

  /home/acjohnson/virtualenvs/foo/bin/python myscript.py

This is useful for crontabs, system service config, etc. which don't load a regular user's bash setup files.

shuzchen• 12 years ago

Similarly, anything you've installed in that virtualenv that installs itself in a bin folder (pip, mercurial, django-admin.py, etc) will be in that bin folder, and calling them from that directory will execute them in the context of that virtualenv as well.

sashahart10• 12 years ago

Fully agreed, and the README for vex suggests this if you are worried about overhead from the wrapper especially for non-interactive purposes where you are repeatedly incurring python startup.

Deejahll• 12 years ago

I'm very happy more people are coming around to this idea, despite multiple independent discoveries. :)

Here's another project that attempts something similar: https://github.com/berdario/invewrapper

Here's a sortof-blog-post and some more discussion about the superiority of modifying a subshell rather than sourcing a bunch of janky shell-functions that modify your current shell: https://gist.github.com/datagrok/2199506

sashahart10• 12 years ago

I did not know about invewrapper, thanks. pew is another decent name.

edit: after looking at the doc, my impression is that invewrapper seems to follow virtualenvwrapper's design more closely with a large number of subcommands with names similar to virtualenvwrapper's, most of virtualenvwrapper's options and features (except hooks and maybe some of the project stuff). I would describe the options as "comprehensive." I specifically wanted something with a dramatically simpler interface and feature set.

invewrapper also seems primarily or only intended to run a shell. I get personal use out of running arbitrary commands under vex as if it were sudo or something.

It is an improvement not to modify the current environment and couple tightly to specific shells either way, though.

One thing I do envy is the PowerShell prompt ;)

berdario• 12 years ago

(invewrapper/pew author here)

I just discovered your project, so I haven't tried it yet, but since the approach seems similar, it should work just fine on Windows (maybe with some tweaks)... and you can just add the Powershell prompt to your docs, no need to envy it :)

I agree that pew is intended as a virtualenvwrapper replacement, obviously you can get something similar to `vex env cmd` with `pew in env cmd`...

Unfortunately I've neglected a little bit the project lately: the most serious thing I want to do is rewrite the test suite, to be able to run it on windows.

Since you mention PowerShell maybe you use it quite often? I'd like to get some feedback on Windows (I added Windows support out of completeness, but I seldom use that OS), since I don't know anyone who uses pew on windows :)

BTW, I discovered some new tools to manage environments in the last year:

pyenv (I actually already knew rbenv), modules[1] and I even started to use Nix

These have not the same scope, and in fact I appreciate the latter 2 idea of using the same system to manage dependencies and versions for all different kind of tools. I don't have an opinion yet on what's the correct way to do these things so, in the meanwhile, even tools that are python-specific (like ours) have a niche to fill

[1] https://archive.fosdem.org/2014/schedule/event/hpc_devroom_e...

mmelin• 12 years ago

This looks great. Any way to have a system-wide .vexrc?

sashahart10• 12 years ago

Yes, there CAN be a way :) but would you do me a favor and write a spec for how you want that to work and jam that in a github issue? https://github.com/sashahart/vex/issues

tudborg3• 12 years ago

Looks good.

I currently use http://tudb.org/articles/2014/03/31/vpython/ for pretty much all of my python projects (also, i am the author of vpython).

sashahart10• 12 years ago

Ah, discovering it from the directory. I considered this in analogy with the way vagrant works.

tudborg3• 12 years ago

Downside is, ofc, that compared to just running something inside a virtualenv shell, vpython takes forever (up to 100ms-ish on my machine) to find the virtualenv, and set the correct env variables.

Most of my projects are long running though (webservers, number crunchers, etc), so the extra miliseconds it takes to load is really not that important.

deathanatos• 12 years ago

Well huh.

Vpython is a tiny(-ish) bash script to help with your day to day virtualenv needs.
* You don't have to worry about sourcing the activate script.
* You don't have to point to your virtualenv path.
just use vpython instead of python to invoke your scripts.

That's basically why I wrote vpython: I'm the author of this vpython[1], which isn't the same as yours, nor what the article is. (I suspect mine is a little more immature, as I've not worked on it for as long.) I guess similar needs beget similar utilities.

  [1]: https://github.com/thanatos/vpython

tudborg3• 12 years ago

Ha. How fun :)

We take different aproaches though.

Repo link for reference: https://github.com/tbug/vpython

Mine expects you to know virtualenv and how it works, and also that you explicitly install it. Same goes for dependencies.

My needs where to avoid sourcing the virtualenv, so i wrote the tool to fix just that. Also, i built mine to resolve symlinks and be usable as the shebang in your python scripts, which is pretty neat when writing command line tools.

schwuk• 12 years ago

I've never bothered with virtualenvwrapper - I create virtualenvs alongside my projects, and call `$virtualenv/bin/python` or `$virtualenv/bin/pip` etc. directly. Sometimes I activate the virtualenv, sometimes I don't.

StavrosK2• 12 years ago

That's what I do too. I mostly activate them by hand nowadays, just to avoid calling python. Fish is smart enough that it knows that, if I type a period in a directory that contains a virtualenv, I want to activate it.

selectnull2• 12 years ago

This is cool, I like tools that focus on one thing.

Just like the author, I never used all virtualenvwrapper's features and was always happy to create and delete VEs on my own. The only thing missing from standard virtualenv workflow is easy way to activate the VE; so I wrote that on my own and been using it since ever. It's trivially small shell script but it does the work well, you can check it out at pypi:

https://pypi.python.org/pypi/ave/

sashahart10• 12 years ago

What is truly horrible is that I wrote my shell function, named it ave, thought it was a great name and then months later (edit: after I released vex) realized you had written a package named ave. Oh well.. :)

This is also worth checking out if you just want 'workon' type functionality without the fuss.

selectnull2• 12 years ago

I guess I was faster picking ave :) but vex is good too.

crdoconnor• 12 years ago

I usually do this just by cd'ing to the project directory, which has the virtualenv inside it and running './venv/bin/python'.

That eliminates the possibility that I'll accidentally run the wrong version of python - e.g. a system python or a python from another environment that didn't properly deactivate.

Most other issues are easy to deal with (e.g. non-existent virtualenv, being in the wrong folder), but the issue of accidentally using the wrong environment is awful because it can manifest itself with subtle bugs or weird exceptions rather than an obvious error.

e12e• 12 years ago

I mostly do this, along with a little hack for certain commands I need, such as mercurial: I have a virtualenv under ~/opt/venv/py-2.7-misc, and make a symbolic link from ~/opt/venv/py-2.7-misc/bin to ~/opt/pybin -- and then my .bashrc checks for a ~/pybin and adds it to the PATH if it exists ( [[ -d "${HOME}/pybin" ]] ...).

So I can just go "pip install -u mercurial", or "pip install <some useful utility>" without worrying too much. And (my) "hg" is in my PATH -- and I can still run stuff in custom virtualenvs on a project basis, just as you say, by specifying the full path (./proj/venv/bin/python ...).

I've never understand why people use anything more complicated.

I do see occasions where activate is useful -- certain packages, I think scipy and pygame are among them -- are not only rather difficult to install via pip, but also tend to be very picky about their environment. Normally I get by by just installing those (and other packages with strong c/c++ library dependencies) via the package manager though.

neotrinity• 12 years ago

one of the things I do in virtualenvwrapper is cdsitepackages to do some patch/workaround on the installed libraries. Not a lot though but its a nifty command to have.

isbadawi• 12 years ago

I like this. I've used both virtualenvwrapper and pew before, but they both have lots of features I don't use. Using this plus some small shell functions like the below to create, list and delete virtualenvs is enough for me.

  function mkvirtualenv {
    virtualenv "$HOME/.virtualenvs/$1"
    if [ $# -eq 2 ]; then
      vex "$1" pip install -r "$2"
    fi
  }
  
  function lsvirtualenv {
    ls "$HOME/.virtualenvs"
  }
  
  function rmvirtualenv {
    rm -rf "$HOME/.virtualenvs/$1"
  }

sashahart10• 12 years ago

It looks like there is a lot of demand for --make/--remove options to vex so I will do those (and that should nearly approach the end of the new features I will do on vex).

Probably good to do some error checking on the arguments to mkvirtualenv and rmvirtualenv (sleepily hitting enter too early, etc.)

Show HN: Vex, a new way to run things in Python's virtualenvs

Related Stories