Python Patterns (2013)

https://taoofmac.com/space/blog/2013/08/11/2300

by dedalus • 10 years ago

165 45 10 years ago

taoofmac.com

snuxoll3• 10 years ago

Can I just state how awful this two-column layout is to read?

BinaryIdiot• 10 years ago

I'm glad I'm not the only one. Decided to skim it instead of reading it all the way through because it was annoying.

Fuzzwah• 10 years ago

I found the "toggle columns" option (top right, under the search box) and it made me happy.

dr_zoidberg• 10 years ago

While it did away with the horrible 2-column layout, it looks like the article was writen with those columns in mind, so it uses short paragraphs that look good on 2-column, but feel a bit weird on a wall of text.

My (barely existant) OCD is going haywire over this article as I read it...

afarrell• 10 years ago

It goes away if you decrease the width of the viewport

praneshp• 10 years ago

Also, the picture with the directory tree got cropped out for me, and I couldn't figure out how to scroll it horizontally. Good thing the columns can be toggled.

agumonkey• 10 years ago

Way better than the single column one on my 1024x768 lcd.

mixmastamyk• 10 years ago

Bottle, peewee, gevent workers, eh? Interesting choices. SimpleTemplate looks interesting, I hadn't seen it before.

Would be nice to hear some discussion there, instead of the column layout.

coleifer3• 10 years ago

I'm the author of Peewee and I've been running my blog with Flask, Peewee, gevent and sqlite. It's a very low-impact combination, lightweight and fun to use, although I'm a bit biased.

raymondh• 10 years ago

Thank you for your work on Peewee. The code is nicely written and is easily read start to finish.

jqm• 10 years ago

I'm using peewee for the first time on a project starting yesterday.

Not using the ORM (I like writing SQL for some reason) but the connection pooling on Postgres is nice and I'm really enjoying working with it. Thanks for your work.

emehrkay2• 10 years ago

I've learned a lot about Python from reading Peewee's source, thanks.

bendmorris• 10 years ago

Interesting as it's quite different from how I generally structure my own projects. A couple questions:

1. Why store dependencies locally? Seems a bit wasteful and unnecessary with tools like pip/virtualenv. (OP does mention virtualenv, but also "Include ALL the dependencies locally" so I'm not sure what to make of that)

2. You've chosen to separate your code into MVC-inspired directories instead of by function. Flask/Django have ways (apps, blueprints) of making application subcomponents modular. It seems like a single controllers directory and a single templates directory, etc. could get cluttered pretty quickly. What will you do as projects grow?

zphds2• 10 years ago

Not the OP, but w.r.t. [1] vendored dependencies are especially helpful if you happen to deploy the software on premise where you might not have full system access and in some cases, no internet or also strict policies on what can be installed.

snuxoll3• 10 years ago

You can vendor your dependencies inside a proper OS package as well. Write an rpmspec that creates a virtualenv, runs your setup.py with the virtualenv python interpreter and boom - your dependencies are bundled without needing to go through acrobatics with manually vendoring them.

If your application isn't able to be installed at the system level I would really recommend this for handling deployment, regardless of whether you have full access or not - deploying tarballs with configuration management tools like puppet are hackish at best, RPM's/DEB's really are the best way to roll your software out.

EDIT: As an alternative to virtualenv which is actually a PITA since --relocatable is always broken, buildout works great.

Loic• 10 years ago

What you can do is get wheel packages for all your dependencies and install them from a local "wheelhouse":

    pip install -r requirements.txt --no-index --find-links=/path/to/wheelhouse

If for each requirement in `requirements.txt` you have the corresponding wheel file in `/path/to/wheelhouse`, this will not touch the network and complete. I have been using this approach the past 2 years with great success.

You can learn a bit more about wheel in this very good article: http://lucumr.pocoo.org/2014/1/27/python-on-wheels/

pc86• 10 years ago

Great example of this - my current client, who requires on-premise development, blocks Fastly. Guess what runs on Fastly? NPM.

So no development if you need to use npm. I can download e.g. Grunt from GitHub directly, but installing the local package immediately tries to download dependencies from npm.

pm902• 10 years ago

1. Why store dependencies locally? Seems a bit wasteful and unnecessary with tools like pip/virtualenv. (OP does mention virtualenv, but also "Include ALL the dependencies locally" so I'm not sure what to make of that)

This is a good question, and I think it depends on your toolchain and also the nature of the code you are working with. Not every project you work on will be OpenSource and might be inaccessible from the public containers/vms where you deploy. So its much easier to bundle the whole application into one package and push that into the destination. e.g if you want to deploy your app to heroku/cloudfoundry/bluemix.

rcarmo5• 10 years ago

OP here.

I use dokku/CloudFoundry these days and just toss in dependencies to requirements.txt, but at the time I often had to deploy stuff on boxes with _zero_ outside access.

rat87• 10 years ago

1. Its useful if you don't want people to have to run stuff to install it, just check it in side by side and they can run it directly from checkout.

hcarvalhoalves• 10 years ago

Project scaffolding like this feels very 2005.

Terr_• 10 years ago

Well, because JSON is about the only format I can read on all the languages I use without ambiguity or extra dependencies

Alas, there is a downside: The JSON spec does not allow comments, which are often important for configuration files. (Though given how much that simplifies parsing, it may have been the right decision.)

So you either omit comemnts, or end up trying to shove placeholder key-values into the closest place you can to whatever you want to comment about.

creshal• 10 years ago

For configuration files I just put in comments anyway and strip them before feeding the file to a JSON parser. (Like here: https://github.com/creshal/yspave/blob/master/yspave/pave.py... )

Hackish, but beats having to bother with YAML parsers and their regular RCE vulnerabilities.

sanpan• 10 years ago

here's a link to the utils mentioned in the article: https://github.com/rcarmo/python-utils

rcarmo5• 10 years ago

Hi, OP here.

Brief clarification: some of those patterns were a consequence of us having to run Python 2.5 and 2.6 on older boxes - it's a survival technique, if you will - and having to vendor some dependencies to make it easier to deploy stuff under some other interesting constraints the article won't get into.

On the whole, these patterns make it easier for other people to not break your code when they want to add, say, another set of REST endpoints (which was my main occupation at the time).

You might also want to read my short piece on SSE: http://taoofmac.com/space/blog/2014/11/16/1940

And yeah, I've been meaning to change the blog layout. Hardly any time for it these days, honestly.

emehrkay2• 10 years ago

Adding overflow: auto to your code blocks (.syntax) would allow us to see the whole snippet. I love decorators too, glad I'm not the only crazy person :)

rcarmo5• 10 years ago

Riiiight. Let me see if I can toss that in without breaking something else in the CSS.

euphemize• 10 years ago

My ground rule is that anything I easy_install will go into env, and env will quite often be (re)built on a separate machine that matches my target environment. So I treat it as a disposable folder, and it’s never committed to any repos.

Was wondering, do you pip freeze > requirements.txt? I don't see the file in there, but I assume that's what you mean since you mention that none of the env folders make it into version control.

On a semi-related note, I used to have tons of projects like this a year ago, and having moved most of our python code to AWS lambda, our repos have shrank to maybe 25% of their original size. Granted it's not fabulous if you need to run a JS app on top of your Bottle framework, but we used to have a lot of small services here and there, the provisioning and deployment was so much work.

andy_ppp2• 10 years ago

I have to say the using Pika and RabbitMQ directly has been a lot easier than trying to use Celery. Just my experience.

neuroid• 10 years ago

Just out of interest, what were the main challenges you faced when using Celery?

Celery is a task queue. Pika could definitely be used to build a task queue (similarly to how Celery is built on top of Kombu), but I can't think of many good reasons why that would pay off. Unless your use case was not a task queue...

andy_ppp2• 10 years ago

Flask application context problems, hiding exceptions within tasks, Json Serialisation sometimes does weird things and hard to configure (the default, pickle, seems to be warned against as a security vulnerability), weirdness with Gevent/Socket.io flask stuff, difficulty (impossibility??) to emit socket events from within a worker, finally the canvas stuff seems great on the surface but I found it mind blowingly difficult to debug weirdness with it.

My Pika/RabbitMQ stuff was much easier to build, unit test and debug (i.e. it actually works). There is every chance the I am just a bit retarded when it comes to using celery, but the fact I've built the Pika version maybe these were just real issues? Dunno, it might all work flawlessly for you in which case it's a great tool.

Python Patterns (2013)

Related Stories