Programmer as wizard, programmer as engineer (2018)

https://www.tedinski.com/2018/03/20/wizarding-vs-engineering.html

by matfil • 8 years ago

166 74 8 years ago

revskill• 8 years ago

Universal, component-based application is the answer of the "boundary" in the article. You got both wizard and engineering solution, which is simple, easy to delete and FUN.

moltar• 8 years ago

Is it what DLL files on Windows are/used to be?

sebcat2• 8 years ago

My understanding is that it is more like COM, CORBA and NetBeans. I guess it is one of those things that can mean different things to different people. At its core, it's just separation of concerns, usually branded and (re-)packaged. Often times, that core idea seems to get lost somewhere around the way...

bboreham2• 8 years ago

Brad Cox described this in “Object-oriented Programming”, a book I first read in 1989. Building applications. simply by plugging together off-the-shelf components remains a beautiful vision, and it is almost entirely unrealised 30 years on.

marcodave• 8 years ago

“Object-oriented Programming”, [...] Building applications. simply by plugging together off-the-shelf components

I've read just here on HN that microservices architecture is none other than an implementation of the original concept of OOP.

jgoodhcg• 8 years ago

Cute way to put it. Just like many I'm in the midst of painful transition from quick and dirty mvp to well engineered end product.

Everything Rich Hickey has been giving talks on and clojure itself seem to be the best _solution_ I've seen.

quantombone• 8 years ago

Using PyTorch (and the broader space of machine learning algorithms under the “deep learning” category) really makes me feel like a wizard. But the downside to being Python-dependent is that putting PyTorch stuff into products not easy. I hope PyTorch 1.0 will change that.

Note: This article was not written with Machine Learning in mind, and I will have to re-read the article to better articulate my thoughts on “Machine Learning Wizardry” and juxtapose my own ideas with those of the article author.

Kudos to author: The article’s main metaphor is excellent because it got my creative juices flowing (i.e., brain working at 110% for a few brief moments).

mlboss• 8 years ago

You can always use ONNX to convert PyTorch trained model to any other format. https://onnx.ai/

fullstackchris• 8 years ago

I wholeheartedly agree that software development as a whole, especially in the past few years, has been more in the spirit of 'wizardry'.

But with the speed at which the sheer amount of new software 'stuff' comes out each year, there's simply hasn't been enough time to develop rigorous engineering specs or best practices for all of those tools.

Perhaps we're starting to see an initial version of a universally accepted model, at least for the frontend, with the mentioned Typescript + Flux (I'll just say Redux).

Many assume that Redux is only a state container, and it is at face value, but more semantically it is (when implemented correctly) a very logical boilerplate where it puts everything related to state in its proper place, so any developer can look at the code base and immediately get a general idea of where state is set, what events exist and how the state is used throughout an application.

I'd like to see things like these very strict patterns emerge for other tools, like Node. For example, I could imagine a snippet of THE universally accepted express boilerplate for a login, given a specific backend... (can an email + password login really get _that_ customized?)

...argh, on second thought I suppose it can, but even then such a boilerplate could have sections with a freebie spot for customizations

Eh, it's late and I wonder if such universal pattern ideas are a pipedream... I suppose only time will tell...

markmiro• 8 years ago

It would be interesting if programmers took "sketching" to be a valuable and necessary part of programming. It's common practice for painters to make a pencil draft first. It's common in industrial design to produce prototypes.

However, when it comes to code we treat it similar to writing. We may have a first draft, but the final draft is often nothing more than a cleaned-up draft. I could be wrong. I never wrote professionally.

It would be interesting if we had languages that would be great for prototyping but designed to be unusable in production. However, I'm having a hard time imagining properties that don't already exist in languages like Python and JS. You want weak typing of course, but you'd be ok with poor security. Maybe we'd some nice features that would make the language run slowly since it running in prod would be a non-goal.

supercleanbro• 8 years ago

I mean, some writers like to make up outlines and stuff to plan out books, though so do stream of consciousness style and clean it up in later drafts. I can see that planning out your code can be beneficial and honestly ideal. Just break it down in a sort of outline what all the program needs and then what each part of the code would need, etc.

DonHopkins2• 8 years ago

How about all the libraries and tools turn over every two months? Oh, there's JavaScript!

strken• 8 years ago

Node.js turns 10 in a few months. TypeScript is 6 years old. Webpack is 6 years old. React is 5 years old. Redux is 3 years old.

Front end JS and TS went through a phase of rapid change 2 to 6 years ago, but I don't think that rapid change is still happening. Certainly there are new libraries, and maintainers still like to play fast and loose with backwards compatibility, but it's not obvious that you have to keep up to date with every single new framework anymore.

taneq• 8 years ago

Part of the reason I like C-style header files for declarations is that I use these as a 'sketch' / 'story map' for the piece of code I'm writing. I tend to spend much more time per line writing my headers than my code, because once I've thought through the public interface and it's close to its final form, the internal code pretty much writes itself.

Tarean• 8 years ago

I notice that I have two modes of programming. This is clearest with haskell.

For simple code I make type errors into runtime errors and let the compiler figure out type signatures. This emulates dynamic languages with a great linter pretty well.

For complex code I write rigid type signatures first and then play type tetris. This works best with dependently typed languages and is called type driven development. Idris can fill in large parts of the program with this - since type signatures are equivalent to propositions and programs to proofs this is basically proof search.

southerndrift• 8 years ago

Don't programmer do this when they create a prototype and reiterate from there until they have a fully functional product? Unlike painting, programming can refactor parts and rearrange everything. There is no need for an additional sketch if the sketch can become the painting.

SketchySeaBeast• 8 years ago

Unlike painting, programming can refactor parts and rearrange everything.

You've never seen a "//I don't know why this works, but don't touch it" in code?

nv-vn• 8 years ago

I'm a big fan of literally sketching. One of my favorite things to do when approaching a hard programming problem is to enumerate a list of things I already know and then try to literally piece those together into an algorithm. I typically do this on paper, making diagrams of how things will fit together and examples.

But my "notation" here is always miles away from what code would look like, and I think "sketching" in pseudo-code or real code would fail to provide this same springboard. One of the biggest things with writing code is that data structures (especially in the form of objects) quickly lock you down to a certain design and it becomes progressively harder to think of the problem in any other way than your initial view of it. Plus, the hard parts typically require huge amounts of infrastructure to be in place before you can run them for the first time, which works much worse than an abstract brain model in my experience.

It would be interesting if we had languages that would be great for prototyping but designed to be unusable in production.

The reality here is that people would find a way to make it usable in production and this vision of a good first draft would quickly fall apart :/

goodcanadian• 8 years ago

Well, I sometimes start by writing psuedo-code first before turning it into actual code, later.

biztos• 8 years ago

There is a saying, "real writing is rewriting."

Ideally the final draft is something that has been very aggressively refactored, multiple times, with input from a Refactoring Engineer.

(Did I just invent a new job category? I don't think there's currently any equivalent of an Editor in the Software Engineering world; code review is a chaotic approximation.)

Unfortunately there is usually pressure, and maybe also desire, to just make it work, maybe with tests, and move on to the next thing, at least in companies.

LoSboccacc3• 8 years ago

"Build one to throw away"

- the mythical man month

ilovetux• 8 years ago

I believe that a Refactoring Engineer would be similar to a contractor specializing in performance, security or systems optimization. I've had the pleasure of working with a couple of these people, and I have to say that I felt an initial (pride|arrogance)-based resistance but they were all friendly and had some really valuable insights especially since their expertise seemed to be lower-level than mine and they were able to look at working code and find places to improve.

In essence I agree with you that this should be an on-staff role, but perhaps the reason we are not seeing this is that the job is usually fairly quick (a couple of weeks/months) and for the business its hard to justify a full-time staff member to perform this service. There is also something to be said for how people in this role are able to gather a wealth of experience by working on a high number of code bases instead of being stuck continuously working on the same few software projects all year.

bmj• 8 years ago

The issue is that in art or industrial design, someone can't reasonably sell the prototype as a completed work. With software, particularly anything that is web-based, as soon as a sales person sees anything that vaguely resembles something they can sell, they will sell it. Sales means bookings/income, so that prototype becomes the MVP.

ivanhoe• 8 years ago

I think that TDD is in a way "sketching" for programmers, you draw the outline (the expected results), and then you fill in and refine the details one by one to make the code work. When dealing with the complex code I also have a habit of structuring the code with comments first: I write a comment for each step in the flow (like a subtitle for that block of code: e.g. parse input, calculate this, get that, output result, etc.), and only when that logic as a whole makes sense I start to implement the actual code below the comments one by one. I find this very helpful to visualize the logic flow.

joker3• 8 years ago

Isn't that just domain modeling in UML?

kelnos• 8 years ago

What do we do?

For me, it's "simple": I never ever ever ever write one to throw away (and I do mean that in a literal, absolutist sense, which is rare for me). This does mean that I often can't use wizard tools (so it can be less fun to build). I never use dynamic languages, and build on the JVM (Scala or Java), because I know it will scale, and there are battle-tested libraries that do nearly everything under the sun floating out there. (If your org has a different "blessed" platform for production services, then use that.) It isn't quite as quick to build the MVP as if I was just hacking something together. But I can do it fast enough, and still end up with a maintainable, evolve-able code base.

It's not a perfect process. My first version usually has only a few tests that verify behaviors that I had trouble modeling clearly in code and didn't feel confident about. Sometimes I miss error cases here and there that someone else has to find and deal with later. Also note that, because I use strongly-typed languages, I can push a decent amount of correctness verification onto the type system, so the compiler catches a ton of errors that I'd need a giant test suite to catch using many dynamic languages. The tests that I do write focus on logical correctness, not code correctness.

But at the end of the day, I deliver products on time that I feel much more comfortable being robust in a production environment than build-one-to-throw-away prototypes. Stuff that I'm fine holding a pager for if I need to. I have several "prototypes" that are still running in production several years after my first release, maintained by other people after I've moved on. And by and large they still contain a lot of the original code, and the design remains close to (and/or continues to be heavily influenced by) the original design.

On the flip side, I've had to deal with code that's been thrown together with the expectation that it could be thrown away later (of course it never can be), and it's incredibly difficult to bring it up to a robustness level that would be deemed acceptable for a generally-available product. These code bases constantly set off pagers for dubious reasons and write unactionable crap to logging systems... and it doesn't have to be that way!

jrs95• 8 years ago

In my experience even things which are explicitly prototypes can be dangerous, as it can appear to be in the short term interest of the business to take a prototype and modify it as little as possible to get it shipped. I've seen this result in massive headaches and thousands of wasted man hours, and for what? Shipping an "MVP" that can't be effectively iterated on 2 weeks sooner? The worst case of this I saw, the technical aspects of this were bad enough that it was (in my opinion) what caused the product to fail.

sethammons• 8 years ago

If you go the prototype route, it is critical that management knows that it is an exploration and no running code will result from it - just knowledge. I am a super big fan of prototypes in my domain. They take under a week usually. We use them to crack the hardest nuts on an upcoming project and usually learn something(s) very valuable that guide the design of the v1 product/solution. Usually these are the learnings we would not have gotten to until mid-way or more in a real project, but we can prototype it, preventing all the potential rework.

zimablue5• 8 years ago

I don't think that this is a constructive post. It is probably totally true - in whatever seat you're sitting in, there's no situation where code is exploratory, but unless you're saying something about how common that is then it's not interesting for the problem. I think you might be implying that there are no/few seats where an exploratory approach is useful, in which case I really disagree.

pweissbrod• 8 years ago

I paint (mostly acrylic) and code. To me, the notion of building a practice run is very liberating; it enables me to think about components, how they should be named/organized, how they interact without the pressure of "getting it right the first time". The time-box for throwaway work should be small. Small enough such that you feel a firm confident grip on the plan at hand. If you have something fully end-to-end operational then more than likely the throwaway has been overworked.

I encourage you to consider trying out more throw-away work before writing the real thing. You find yourself with not only a more lucid vision of what goes in to the "real thing" but a little more muscle memory in getting started down the right path.

AstralStorm3• 8 years ago

Unfortunately you cannot throw away a building with a fresco, and many useful applications are of this sort of size.

(Even something as "simple" as email.)

The best you can do is paint the fresco on paper, digitize, design the building outline and iterate 3D building designs. And you have not considered materials and structural design at this point and do not even have a scale physical model.

Prototypes work for small, enclosed apps with limited functionality. Not even video games most of the time and these are relatively small and specific. Word processor? Good luck. DAW? Oh my. CAD/CAM tool? You've got to be kidding me. IDE? Nope. A compiler? You'll rewrite it a few times. Desktop environment? A good recipe to lose users. Even CRUD...

You cannot iterate a painting into a building with a fresco.

The thing is that currently an MVP is definitely huge, as much as startups would like everyone else to believe otherwise.

dmichulke2• 8 years ago

A really good alternative to Python and C: Clojure + Java

Mostly because

- Clojure is very very terse

- Java has the battle-tested libs

- they run on the same (J)VM - so no FFI required in your code

zimablue5• 8 years ago

I think the problem with this is, if I'm going to say "completely rewrite this [function/class/module] into another language", then there's a good chance you want that language to be about as fast as possible. I guess because a lot of problems fall straight from "speed absolutely doesn't matter" to "this is the bottleneck of the whole thing".

I think that is a reason why there's a lot of python/c++ in hedge fund land. I've written some clojure but don't know the c interop story for it.

dmichulke2• 8 years ago

completely rewrite this into another language

You just opened the box of pandora for a million reasons most of which are unbeknownst to the both of us :) But for the sake of argument I will continue under this assumption

language to be about as fast as possible

If you're interested in Mathy stuff like Machine Learning, FFT, ... then maybe. But even for those you usually have JNI bindings, so it's easy to use most of those mathy C libs if necessary.

But I guess that 95% of all software isn't about speed but about something else (correctness, maintainability, safety against threats, portability, ...) because costs today are usually dictated by manpower costs or those arising from safety/security incidents and much less often by hardware costs compared to say a decade ago.

kenhwang• 8 years ago

Arguably you could mix and match any of the jvm languages to your liking. Jython, JRuby, Kotlin, Scala are all reasonable alternatives to Java while being easier to hire for than Clojure.

LoSboccacc3• 8 years ago

Back then I used to use beanshell for the most changing part of code (prototypes/logic)

rgoulter• 8 years ago

"Wizard, Engineer" reminds me of Yegge's "Software Liberal/Conservative" approach to risk. https://plus.google.com/110981030061712822816/posts/KaSKeg4v...

Albeit, rather than "wizards like implicit/magic, engineers prefer explicit/boilerplate/maintainability", the difference Yegge suggests was management of risk.

matfil2• 8 years ago

Thanks, that's a pretty interesting take.

(Also a reminder that we're probably going to lose some interesting stuff when Google+ goes kaboom...)

Adamantcheese• 8 years ago

A wizard then should have a spellbook, one filled with all sorts of spells written out for immediate use. Maintainable hacks if you will. Those are probably just scripts though I suppose?

bryanrasmussen2• 8 years ago

The problem with spellbooks is one seen in media on the subject of wizardry, that the wizard spends a lot of time memorizing spells. This is why the most useful spells get turned into artifacts that the wizard does not have to always memorize to know how to cast correctly.

What is needed is an artifact like spellbook, that the wizard when faced with a situation could describe it to the spellbook and get back the correct spell or combination of spells to solve the situation. Attempts have been made to create such an artifact, but unfortunately the resulting spellbooks still take a long time to find the correct spell and when reading the spells you often find that there are missing ingredients or a complicated set of gestures that must be performed to make the spell work, and you have to read all about these gestures in turn to figure out which ones you really need.

thisiszilff• 8 years ago

Have you heard of hoogle (Haskell's type signature search)? It almost exactly feels like what you've described.

bitwize• 8 years ago

Lisp is the ultimate "wizarding" programming language.

But when I miraculously was called upon to maintain an enterprise code base in Common Lisp, it was an absolute joy. Because whenever they encountered a roadblock in maintenance, the Lisp wizards who had come before me just wizarded up a solution. One of the things that stuck out was that it had its own custom test framework, that was head and shoulders above XUnit, Mocha, or any other commonly-used test framework. Adding a new test was virtually a one-liner; the test would generate test data , send it to the server, and check the server's response against an XML template provided by the test case.

zimablue5• 8 years ago

I'm sceptical of his argument that "we've gone from dynamic being trendy back to typed(java), because 'people' had to maintain dynamic codebases".

An equivalent but probably equally not-the-real-explanation argument would be that we've gone from an era of opportunity into an era of oligopoly as the internet titans have emerged and so the "coolest kids" who everyone cargocults have gone from being fast-growing startups to members of the big-5 elite. Kind of the same as his argument just with detail on who his "people" are and why, but it changes the implications.

I think it's more as he touches on that the two are converging. I think eventually some sort of pluggable typing will win (the "proofs" on your system won't be a single compilation pass but maybe different typing for different parts of the system, and specific proofs run between compile and runtime as the two blur), which will look more like gradual typing.

jondubois2• 8 years ago

According to the definitions provided, it seems that engineering is the wrong approach for the vast majority of systems (especially popular web-based platforms). We should be leaning more towards wizarding.

Most systems change all the time. So long as a company needs executives to make decisions and steer in different directions depending on the economic conditions, companies also need the flexibility to change their code.

Even the Linux Kernel which is now decades old is still being changed all the time. If you use tools which assume that every line of code you write is not going to change, then unless you're programming an aircraft or a medical hardware device you're probably using the wrong tools.

You should not assume that just because some low level module is deeply nested within the code, that it means that it should not be changed or thrown out.

That's why I prefer dynamically typed languages for web systems; they start with the rigth assumption about the ever-evolving nature of the project.

If JavaScript was designed to be statically typed from the beginning, web browsers would not have attained the usefulness or popularity that they have today.

zimablue5• 8 years ago

I'm not sure to what extent I believe this.

There are a lot of overlapping distinctions, some blurry, I think. He chooses some for wizardry vs engineering but I don't know whether they're best. For example he puts "magic"=implicit, but I would normally say highly implicit code is "engineering" because it implies that someone has understood and explored the problem enough to write a very tuned framework or library. I think reasons for putting very implicit code in with wizardry could be that it might be that the person using it doesn't understand it, and that it's somewhat in opposition to strong typesystems

wizardry/engineering high uptime/downtime possible efficiency important/efficiency not important large codebase/small codebase not implicit/implicit (I disagree most with this one) problem is understood/code is exploratory (I think this is the most important distinction) high specialism/low specialism of coders changes slowly/changes quickly typed/untyped

I would say websites are engineering because the problem space is well understood, they need high uptime and they tend to be written by specialists (React guy, django guy etc).

ellius2• 8 years ago

Another problem is highlighted by Joe Armstrong and Rich Hickey, which is that as soon as you move beyond the confines of a single program and start dealing with a system of programs, type guarantees become less relaible. So you have these types that may make complete sense in your program's world, but they drift out of sync as the larger system evolves. This is a general problem with any sort of proof mechanism. I still like types and think they can be useful, but they are only a single tool in designing a robust system. They have limitations and drawbacks like everything else.

acidbaseextract2• 8 years ago

That's why I prefer dynamically typed languages for web systems; they start with the rigth assumption about the ever-evolving nature of the project.

I agree with your assessment that we need to design for evolution.

I disagree with your assessment that dynamic languages provide that. Dynamic languages certainly do hit a sweet spot for fast iteration on small projects. As soon as your project gets large or long running, the assurances that static types provide get really nice. Witness how much effort has gone into building type systems for dynamic languages: Typescript (MS) is exploding and had vigorous competition from Flow (Facebook). Python has mypy plus a bunch of corporate backing: MonkeyType (Instagram) and PyAnnotate (Dropbox).

I feel much more comfortable aggressively refactoring in static languages.

dtech• 8 years ago

I disagree with your assertion that dynamic languages are easier to change. In my experience programs written in a typed language are much easier to change, because IDEs can exactly tell you how something interconnects (and thus where breaks might happen) and the compiler can give you some confidence that you didn't miss anything, or alternatively throw an error if you did miss something.

You can partially provide the same benefits in dynamic languages with tests, but at that point you're paying the same cost as with a type system.

sly010• 8 years ago

Most systems change all the time.

The way I think of it is "wizardry" has good initial velocity (as it relates to change), but velocity decreases with time, whereas for "engineering" velocity is terrible at first (initial investment), improves a bit over time, but most importantly it settles to a constant level.

Engineering is O(log(n)), wizardry is O(N*N).

coldtea• 8 years ago

That's why I prefer dynamically typed languages for web systems; they start with the rigth assumption about the ever-evolving nature of the project.

The inverse can be said: a statically typed language makes it easier to replace parts later, do automatic refactorings, etc., and know you got everything right.

Dynamic just let you have less ceremony (no type definitions) but forces you to keep this all in your head (you still don't want to pass the wrong type of thing to the wrong receiver) and doesn't give you any assurances.

With type inference and/or autocomplete, statically typed languages are faster to write and easier to get right than dynamic.

zozbot123• 8 years ago

Gradual typing does not seem to work very well. It requires a runtime check every time you pass data from a "dynamically-typed" to a "statically-typed" part of the program, and then a conversion step from statically-typed to tagged data for the converse. These things eat into performance, in the end it's no better than what dynamic types would give you, and a lot worse than what one could achieve with static types.

jbergens• 8 years ago

It sounds like optionally typed languages like Typescript should be really good if you want to start in "wizarding" mode and then change the code more into an "engineering" solution.

luord• 8 years ago

There are quite a few false assumptions here.

People have certainly managed to create test suites that make it harder to maintain the code.

Sure, but pretending that this is the norm is just not true. And arguing against an extreme case can be done against anything .

I get the impression Google has been able to migrate a lot of C++ and Python to Go using this approach.

Not only is this heavily suspect (I'd love to read a citation for this) but if there's a language that represents the "engineering" side (?) pretty clearly, it is C++.

Gradual type systems has started to garner a lot more interest.

No they haven't. Those have existed for over five decades . Another problem with these articles is how they seem oddly ignorant of the history of computer science. This is specially odd coming from a PhD (assuming that I looked up the right person in google).

And, last but not least, some of the best engineered pieces of code are precisely shells, dynamic languages and frameworks.

It's things like these what makes these articles seem like they were written by java developers annoyed because java is no longer the trendiest toy.

marcosdumay• 8 years ago

No they haven't. Those have existed for over five decades.

There is nothing in "started to garner more interest" that implies something is new. It's even the other way around.

casper345• 8 years ago

"Python + C" I am taking this approach working/learning with the raspberry pi. Although it can do both python and C, I know in electronics working with low level C will be a lot beneficial for my learning and code itself. When I migrate to more advance topics to scripting and ML, I will use both PYthon and C to implement what is needed.

Programmer as wizard, programmer as engineer (2018)

Related Stories