A parable about problem solving in software development

http://www.drmaciver.com/2013/03/what-the-fuck-are-we-doing/

by Zarkonnen • 13 years ago

55 29 13 years ago

mattmanser2• 13 years ago

I honestly couldn't make head nor tale of this.

It's certainly not a parable as there's no lesson drawn.

Without any code specifics this comes across as a bunch of smart people who thought they knew how to program but didn't constantly reinventing the wheel even though everything described was a solved problem.

To me anyway.

andyjpb2• 13 years ago

How about "Obviously this is how we should have done it in the first place. It’s not just obvious in retrospect, it should have been obvious in the beginning. We were just too focused on fixing this one problem with our current system rather than calling the system itself into question to see it."

DRMacIver6• 13 years ago

What, you expect us to read the article to derive the lessons contained therein? What lunacy is this?!

blablabla123• 13 years ago

To me each of the "grand visions" were legit and valid somehow. (Maybe except for the 12-step solution ;)) Sometimes it's better to just stick with one solution and make the best of it.

So the lesson I see is to not change your framework every month.

seguer2• 13 years ago

I was headed down a similar path myself, but luckily I was writing a completely new library that would abstract a lot of key-value stores. Before we used it in production, I performance tested it - HTTP is really slow. The end result was a client library that talked directly to the stores using adapters for the abstraction.

Patient03• 13 years ago

"Before we used it in production, I performance tested it - HTTP is really slow"

Slow compared to what? direct memory access?

If you're going to say "HTTP is really slow" I contend you really probably meant "communication over a network is slow"...

jeremyjh• 13 years ago

Well, he didn't express this very well, but I don't think he is quite that naive. Bytes move across a network media at the speed of light regardless of whether the electrons and pulses encode HTTP headers or not. And the latency of network layer switches and routers is similarly enencumbered by the presence of slashes in resource names. However, it is typical that HTTP-based client/server applications have more layers of serialization and a heavier payload that you'd find with a typical binary database protocol. Plus of course your HTTP server probably turns right around and talks to a database adding a whole extra layer of serialization and network data transport.

You are correct that this is not the case per-se, and I think there is an implication here that a lot of people really do not think very much about the massive overhead of serialization in general - instead they use a heuristic like "HTTP is slow". And while that heuristic does not capture an accurate model of the problem, it does often point in a direction that will prove fruitful as they grasp about blindly in possible solution spaces.

seguer2• 13 years ago

In contrast to direct database access, the HTTP implementation of the abstraction layer performed significantly worse. This will have also been part of the abstraction library having to parse/run the logic as well (nginx + PHP).

From memory, after trying the entire abstraction layer, I boiled it down to as small a file as possible for testing purposes (ie. a single PHP file that took the call and stored it into the DB, no configuration, removing as many "variables" as possible).

EDIT: Direct database access over the network.

NateDad2• 13 years ago

Wow, when did this guy start working at my company?

This is like 75% of what happened at my company, except we actually made a whole (custom, of course) framework for dynamically loading code on the server and client.... we also never did the last step of having the clients talk to the DB (the server federates access based on a complex authorization model that can't reasonably be done directly against the database). We are currently rewriting the client-server communication stack (I'm still not sure why).

And next month we're starting a full rewrite from scratch to put it in the cloud. Whee!

DRMacIver6• 13 years ago

May I extol the virtues of quitting and finding a better job to you? :-)

Patient03• 13 years ago

He lost me at:

"At some point it occurred to one of us that the reason our HTTP API was slow was of course that HTTP was slow. "

Why "of course" is HTTP slow? There's nothing intrinsic to HTTP that I am aware of that would make it "slower" than any other protocol going over a TCP socket...

DRMacIver6• 13 years ago

For our cousins across the pond, this is an instance of a type of humour called "Sarcasm", in which you state something directly in order to imply that it is obviously wrong.

NateDad2• 13 years ago

Yeah, FWIW, I missed the sarcasm, too. Such is the trouble with a text only format.

andyjpb2• 13 years ago

I think that understanding this kind of reasoning is the central point of why they made the mistakes they did and how they could have been avoided.

contingencies2• 13 years ago

This post is a logical mess. The author might do better writing another one in a few years or learning to use diagrams to convey architecture (but for god's sake don't bother with UML!).

Architecturally, the changes discussed seem to be (1) From a standard protocol to some customization for premature optimization purposes (unmeasured results), and back to a standard protocol. (2) From service abstraction through higher-layer communication to service-dependency through direct database connectivity. (3) From code duplication between client/server to the use of a centralized model. (4) From multiple codebases to a single codebase.

My take: (1) Is premature optimization, needless complexity and was ultimately a waste of time. (2) A change so completely fundamentally different with a view towards system architecture (security, resource allocation, proxy/WAN/potential-client-range, etc.) that it is hard to analyze; ultimately it suggests a complete misunderstanding or rescoping of the original problem. (3) Generating code or behaviour from models is very often a great idea. While up-front complexity may appear slightly larger, it means you can retarget your code to other languages/platforms much more easily, without the inefficies and syntax-learning overheads of partial-but-similar strategies through runtime abstraction libraries. (These are occasionally useful for very-specific requirements, but that's a tangent.) (4) See 2.

rossjudson• 13 years ago

Your insult is a logical error. You might do better writing another one after googling David Maciver.

Always remember! Google before insulting!

My take: You failed to to note the confessional tone of the piece. You also failed to note that even very smart people (see Google point above) can sometimes get sucked into wise-by-step, terrible-in-aggregate decisions.

contingencies2• 13 years ago

No insult intended. However, congratulations on exhibiting lowest form of wit. I was just trying to extract some meaning from the piece that was practical. Sorry to ruffle anyone's feathers... I'll get back to duckduckgo-ing 'totalitarian panopticons of childlike presentation'.

drewcrawford• 13 years ago

This was a very strange story to see, because my life is almost a mirror image of the author's. Except my harebrained ORM-powered RPC message-queue rest-was-too-slow experiment is working so far. Consider this a counterstory.

The author lists two problems with the old system: code re-use and speed. But he doesn't really say how the solution is going to improve on the status quo, other than just by throwing programmers at the problem and hoping they improve on REST. Now maybe they had clear solutions and maybe they didn't--but the post doesn't say.

I sympathize with the author's problems. But I also started with specific clarity about the problems and strong solutions about how in practice to improve. For example, the speed problem for my use case was very specifically that latency was too high. Latency was too high specifically because of fast-lived connections, the SSL handshake, and TCP overhead. So I designed a system very specifically around long-lived connections, a non-SSL based crypto layer, and UDP. It's a full order of magnitude faster; it actually changes the way you use network calls because they are so cheap now. As time has gone on I have extended the speed improvements, but it was way better than the status quo on day one.

Similarly for code reuse, I had a very narrow set of problems surrounding reusing across client/server only very specific parts of very specific classes. As a result, rare is the change that requires moving client and server in lockstep. I don't think I did it in the month of February. And the productivity gain was great from day one.

I am the first to admit that I've had some bugs in the complicated pipeline that require Deep Magic to comprehend. I have one or two open right now. But, if you have a 10x runtime improvement and a 3x development improvement, spending a day or two learning the ins and outs of Deep Magic is not the worst thing you can do.

I think the moral to this story, if there is one, is don't fly by the seat of your pants. Don't write a system that is faster, write one that is exactly 92.56% faster by the application of two specific techniques. Don't write an RPC system that eliminates the barriers for sharing arbitrary code, write an RPC system that eliminates two particular barriers for classes that meet three known requirements that most people in the organization are annoyed about. If you don't know where you're going, you will never find out if you get there.

DRMacIver6• 13 years ago

Did you miss the part where I explicitly said in multiple places that there was an obvious simple solution and that this post was an example of what not to do?

triplesec3• 13 years ago

Quote of the week: "What follows next is the point at which it all starts to go completely tea party."

(para) "At the moment what we have here is a slightly baroque architecture, but it’s fundamentally not that much worse than many you’d encounter in the wild. It’s not good, but looking at it from the outside you can sortof see where we’re coming from. What follows next is the point at which it all starts to go completely tea party."

markokocic• 13 years ago

I hate to read that "one caffeine fueled weekend later ..." in a story like this.

Is a weekend the only time one is allowed to commit fundamental code changes to the project? Is a "lone cowboy" coding all alone without any consultation and coordination with the rest of the team really a good role model in any development environment?

This is just a small nitpick. The rest of the story, however, is good and educational.

A parable about problem solving in software development

Related Stories