From Python to Go: migrating our entire API

http://blog.repustate.com/migrating-entire-api-go-python/

by ainar-g • 11 years ago

71 33 11 years ago

blog.repustate.com

cdnsteve• 11 years ago

How did debugging change, for the better or worse? Did your tests speed up?

mikesf888• 11 years ago

my favorite quote: "Writing software in languages like Python or Ruby often seduces you into being ignorant of what’s going on under the hood because it’s just so easy to do pretty complex things, but languages like Go and C don’t hide that."

erokar• 11 years ago

Damning with faint praise. Often, you don't need to know what's going on under the hood, and often it's not very interesting.

marcosdumay• 11 years ago

To tell the truth, Python is an easy target here, but I doubt 99.9% of the projects in C or C++ care to manage memory as closely as GHC does under the hood. (The same way as they don't manage concurrency.)

Some times, it's even better not to look.

bjt• 11 years ago

I read that but I can't see what he's getting at. Go is garbage-collected just like Ruby and Python, right?

azth• 11 years ago

The original comment didn't make a lot of sense. Go still has many things going on beneath the scenes, including GC and interfaces.

cowsandmilk• 11 years ago

They cite their main reason for migrating as being self-hosting, which is hard in Python.

I've had a huge amount of success using pyinstaller for this problem, shipped many products into hugely diverse environments.

https://pyinstaller.org

melling2• 11 years ago

I think performance ended up being the big win:

"We migrated our entire API stack from Python (First Django then Falcon) to Go, reducing the mean response time of an API call from 100ms to 10ms We reduced the number of EC2 instances required by 85%"

medecau• 11 years ago

I'm usually left wondering if anyone else know about the multiprocessing module. Specifically the Pool class.

Scarbutt• 11 years ago

tl;dr - Python is to slow for us. (its about the implementation, not the language).

simeonf• 11 years ago

I expect the performance/parallelism complaints. It's interesting that the tl;dr also includes "packaging and distributing an app is a reason go > python".

Python needs a .war format and I'd like to see more people get behind https://github.com/pantsbuild/pex

WorldMaker2• 11 years ago

The original Python .war equivalent was the .egg, but that's slowly losing ground to the .wheel.

bliti• 11 years ago

I wonder why they didn't go with Java. It would have been a good fit with the additional benefit of not having to pay the premium Go devs go for (Go dev here).

microtonal2• 11 years ago

Did someone gather salary statistics? I could imagine it could also be the other way around: hordes of Go fans competing for a relatively small amount of jobs drives prices down.

eatonphil• 11 years ago

Am I the only one who thinks it's crazy that people would compete for jobs because of a language? Especially a new language? Would anyone seriously give up a job just for the opportunity to program in another language?

Perhaps this is a bigger thing among corporate programmers? I've only worked at smaller companies where they are more open to experimenting if you talk them into it.

vkat• 11 years ago

Pretty neat trick porting over python unit tests to go. Any plan on sharing more details or open sourcing that part?

dv_says• 11 years ago

Formerly using Python, motivated to migrate to a modern stack, and so recently attempted to write a simple web back-end for my apps in Go and then in Elixir. Same logic, same unit tests, two languages.

Really liked: the simplicity of Go's build, not having to spawn N processes on the server side, the compiler catching so many issues, the very strict formatting rules, all a major win over Python. However, didn't like how verbose the code turned out, especially for parsing or any data manipulation tasks. For server-side debugging/tools, it's also a bit of a build-your-own adventure.

Elixir I found to be much more pleasant to write and subsequently read, more well-suited for back-end tasks, great for generating and parsing data. Also, more things in the toolset for back-end deployment, in large part due to the OTP stack.

Perhaps Go's ideal use case is local system tools, compilers, or anything else that requires concurrency (let's say, creating tools like "ls," or Apache bench, or gcc, or cURL), and something like Elixir is more ideal for building actual servers.

gtaylor• 11 years ago

It sounded like they were gunning to shave the maximum in response time from their HTTP APIs. From my understanding, Elixir/Erlang probably wouldn't be able to match an optimized Go process in raw efficiency. I get the sense that scaling horizontally is the big draw of the Erlang VM, not so much raw responsiveness.

If simplicity were the primary concern, they could have stuck with Python. You do get a much more distributed element with Elixir/Erlang (which certainly helps with volume), but I'm not sure they'd reach that 10ms response time average with it.

BuckRogers4• 11 years ago

If latency were the primary concern nothing would beat the BEAM platform. It was built for that purpose, low latency is absolutely critical when you make a phone call.

Of course that's a bit like a benchmark (meaning, it means absolutely nothing to you/your code). If there were any more latency in the Erlang solution it would be the same algorithmic issues that could pop up in any software.

Go is more efficient for CPU throughput, but BEAM has been fine tuned for decades to keep latency down and wouldn't be able to top for responsiveness.

My personal opinion is that they made a mistake. Reporting about swapping out a stack like this is weird because it looks like they don't know what they're doing. What's next month's blog? We're Moving to Rust? I'm glad they're satisfied, but you gain improvements anytime you do a rewrite- even in the same language. I wouldn't have migrated my existing stack anywhere. I'd use PyPy which would've done wonders for responsiveness on the existing product and adopt containers for deployment.

nailer• 11 years ago

10ms HTTP response times in Phoenix (on elixir) is pretty normal.

BuckRogers4• 11 years ago

I see a lot of these types of blogs but I'm always left with the same questions. Did they not try using PyPy? Using LXC and PyPy would be my first step, not swapping out my stack. It just seems more sensible to me considering that Go as a platform isn't perfect either. Nothing is.

But I'd be hesitant to leave a massive, thriving ecosystem like Python (especially if on 2.7 which has everything). Win some things, lose some others when you can just adopt containers and PyPy and gain the benefits that Go has without losing Python's advantages.

eloff• 11 years ago

They tried Cython first, so PyPy wouldn't have been faster than that. The big speedup for them was in making the code parallel, which is tricky in python and typically requires processes instead of threads and copying instead of sharing, either of which may eat most of the gains in this case.

gcb0• 11 years ago

their problem was using weird and heavy frameworks.

i wouldn't consider djangob for anything other than a heavily cached content site.

seems like it's a company that just plow through problems. i bet whatever the problem is in go they will find only next week.

python 2.7 and 3 have a marvelous, but not very well documented, multiprocessor module. it gives you more performance for that the cython and other shenanigans.

but i guess using the existing tool right doesn't give promotions.

... yeah I'm making a lot of assumptions about then but that's just because I'm salty today. they should still learn to use python multiprocessing module. :)

Randgalt• 11 years ago

"Due to the similarity between Python and Go" - huh? Go and Python are almost perfectly orthogonal.

From Python to Go: migrating our entire API

Related Stories