Domain Knowledge or a Lack Thereof

http://jacquesmattheij.com/domain-knowledge-or-a-lack-thereof

by ajhai • 13 years ago

203 110 13 years ago

kayoone• 13 years ago

I am currently working on a web software project in the medical/health space. Of course its really good to know how the people that use the software work and what they expect and i am eager to learn those things, but i will not dive into a sector i am not really interested and absorb any and all information i can find, since i probably wont need it for the next project and it personally does not interest me a lot.

For technical domain knowledge i agree, but it also happens that this is often of personal interest to me anyway.

kevingadd• 13 years ago

The lack of this kind of domain knowledge - either from being unwilling to go find it, or not being given the time and the resources to do so - is a big problem for tools development in games. People doing the scheduling and planning tend to assume that if you put enough smart programmers in a room and give them detailed specifications, they'll be able to solve all the problems faced by some artists and writers in another room, even though they don't know anything about art or writing. Sometimes this results in the kind of obvious mistakes that you might otherwise assume only happen when you're writing software for the government.

In a sense it's kind of sad: In building traditional software, understanding the customer is one of the hardest steps because you can't simply walk over and observe them going about their daily work, or ask them questions on a regular basis to understand their issues - you're separated from them by a sales process and probably some automated support tickets and a PR guy who won't let you have open discussion in public. People building tools for game development can have their customers sitting a few desks away!

calpaterson• 13 years ago

> In building traditional software, understanding the customer is one of the hardest steps because you can't simply walk over and observe them going about their daily work

One of the nicest things about the Agile fashion at the moment is that my customer (or a reasonable representative of him) is a couple of desks away and I really can go and ask him about what he wants. It's incredible how much time is saved by avoiding doing things he doesn't care about and how much happier he is with the end result.

jeltz3• 13 years ago

> Sometimes this results in the kind of obvious mistakes that you might otherwise assume only happen when you're writing software for the government.

My personal guess is that the reason there too is lack of domain knowledge. That many government projects are ran by handing out specs to the people tasked with implementing them.

Jare• 13 years ago

I always asked my tools programmers (and do myself) to spend some time regularly sitting by an artist or designers while she works, and observe. HUGE difference.

kenkam• 13 years ago

This particularly resonates with me. I have been working in a bank as a graduate developer for the last year and a half, getting by without much care about how instruments are priced, traded, recorded, audited, etc. To me, I was moving a button around and colouring things using WPF. To me, the "Yield" column was just another column with numbers that were supposed to be 2 decimal places and turns red if negative. All the while I was dreaming I wished I worked at Google on 'technology'.

Which, in hindsight, was a little misguided. There is so much value in learning and understanding the domain in which we are working in because it makes us so much more effective. The realisation came when I became frustrated in meetings where we would discuss ideas on financial models and I could not add any value to the discussion.

I then imagined working at a company like Google, let's say, doing stuff I wouldn't normally be interested in. Would I bother learning the domain knowledge? I couldn't say yes with 100% conviction.

I love programming but I also want to be effective in what I do. To me now, understanding the problems we are solving is probably just as important as writing maintainable code.

wanderingmarker• 13 years ago

Ah yes, the tragedy of studying computing to the exclusion of everything else. Graduates with computer science degrees have an in-depth understanding of... text editors. And maybe compilers, databases, and operating systems — all items of zero value until they are applied to solving some real problems in some other field. Problems which the computer science graduate knows nothing about.

Then these graduates go into the working world, and they typically can't help solve problems in economics, physics, applied mathematics, or medicine except by coding up some system to someone else's specs. The more dedicated ones make an effort to understand what they're doing. They exceedingly rare, gifted, and motivated ones might learn enough about an industry to make real contributions.

I hope that our field eventually evolves to the point that programming skill becomes an accessory. People will then learn to program just like they learn to write, and use the craft to help them solve their actual problems. The profession of programmers who only know how to program should disappear like the profession of scribes who only know how to write.

GFischer• 13 years ago

I hope we'll have specialized careers someday (Medical CS, Economics CS, etc...).

In many careers, it's the other way round - most physics I know taught themselves how to program, for example, but they're lacking some tools that would probably help them a lot (and physics are very close to CS). I've seen accountants do a lot of programming in Excel macros and Access too.

Doctors and Economists might have less interest in learning, but academy-minded ones will really appreciate the modeling they can do with the use of computers (I think I saw a medical paper where they reinvented some basic CS concept :) ).

Some of the domain training comes from practical experience - I started out working for the credit report sector, and then moved on to the insurance sector, so I've picked along some domain knowledge in those areas. Other colleagues have worked in logistics, and now know about their problems, etc...

eru• 13 years ago

> (I think I saw a medical paper where they reinvented some basic CS concept :)

Perhaps more shocking:

https://fliptomato.wordpress.com/2007/03/19/medical-research...

pytrin• 13 years ago

Programmers are not like scribes, they are like authors and writers - which are still here. Programming is a creative discipline, it's not just formatting one written format (specs) into another (code). Of course, the bad ones might make it seem like it...

blindhippo3• 13 years ago

"the bad ones might make it seem like it..."

Or the ones with highly uncreative managers who lack sufficient domain knowledge themselves.

alemhnan2• 13 years ago

"Ah yes, the tragedy of studying computing to the exclusion of everything else. Graduates with computer science degrees have an in-depth understanding of... text editors."

That makes a lot of sense. To me is that because we are using "languages". With a language you write. And your problems are writing problems: syntax, commas, naming, verbs, etc etc.

What if we just get rid of "languages" ? We actually do not "write a skyscraper". We make a project for a skyscraper. Or we "draw" a skyscraper.

Are languages the best tool of choice to solve the problem in connection with the information (and its calculation)?

(not that I have a definite answer, just a question).

fallous• 13 years ago

Languages exist to convey complex information, nothing else comes close to them in that ability.

As an exercise, try to convey the idea "I'm diabetic and I need some insulin" via !language. Charades? Interpretive dance? Drawing?

And yes, we do "write a skyscraper". That's what blueprints are for.

jiggy20112• 13 years ago

You can draw a skyscraper because at it's core a skyscraper is an object that exists strictly in a 3 dimensional world. Therefor you can convey a lot of important information about the skyscraper by showing it's shape and dimensions in an autocad drawing.

In reality however I expect that most skyscraper projects involve many many pages of writing and formulas in addition to the drawings.

OTOH a computer at it's core is a machine that sequentially follows instructions. So any description of a computer program must be able to reduce to this.

Since around the dawn of computer (or at least in the 10+ years I have been involved) there have been attempts to remove the requirement for computer programming, but in reality nobody has come close to anything as powerful as a programming language.

The most effective progress has been made via abstraction, where you allow one instruction to be used to imply many. For example making a REST API call with 1 line implies a lot of operations on a lower level (TCP/IP etc).

I think this is fundamentally because any system that you propose to replace a programming language will either be:

1) Turing complete, in which case it could be classified as a programming language itself and will require knowledge of algorithms/abstractions etc to operate effectively. This is where you end up with flowchart driven development tools where the flowchart ends up literally resembling a plate of spaghetti for any reasonably complex system.

2) Not Turing complete, in which case you lose a lot of power as there are things the system will not allow you to express and thus lose the ability to deal with many edge cases. Since you are developing software and not just using it, the reason for the development activity in the first place is probably to deal with some edge case than an off the shelf program does not.

When I was younger I used to play with various "code free" game development tools and ran into the same fundamental problem.

For example, I have a shooting game and when the player is hit with some projectile he takes some amount of damage based on the projectile he was hit with. This was usually very easy to do.

BUT then say I want to add an "armour" powerup to the game. If a player with the armour powerup is hit then he takes damage but at a reduced rate that is proportional to his armour value. The armour value is also reduced based on itself and the armour piercing value of the projectile which hits. However if the armour piercing value of the projectile is below some value proportional to the armour and health values combined then no damage is taken at all. Oh and the armour powerup cannot stack with certain other powerups except under some specific circumstances.

That type of logic was impossible to express with these tools, or required finding creative ways to abuse the tooling in ways not intended which would in turn make other functionality more difficult/impossible.

raverbashing2• 13 years ago

Yes

And some are almost proud to not know anything beyond their field.

Newsflash: You can be the best writer in the world, but what are you going to write about then if you know nothing besides writing?

Programmers often complain about managers knowing nothing about the field. But it's the same thing when the programmer doesn't know anything.

There is a lot of money in doing specialist applications. Most of these are made by people who know a lot about their field but only a little bit of programming.

onemorepassword• 13 years ago

This. The same programmers that complain about clueless micromanagement are usually also the ones with the "I don't want to know all this business shit" attitude.

Guess what? You're being micromanaged by MBA's because otherwise, despite all your l33t coding skills, you're f-ing useless.

thejteam2• 13 years ago

Your point about writers is a real problem. Way too many literary novels where the main character is a writer.

Too many people, especially worker bees like engineers and janitors, tend to focus only on their part of the job and not how it fits into the big picture.

henrik_w2• 13 years ago

Yes, reading HN it often seems like programming skill is the only game in town. It is obviously important, but it's nice to see something on the importance of domain knowledge as well.

So coding skill is one dimension along which you have to be good. Domain knowledge is another dimension.

A third dimension is knowing the code base you're working in (which takes time, since most production systems tend to be big). To be a really effective developer, you need to be good along all these dimensions.

yareally2• 13 years ago

> Yes, reading HN it often seems like programming skill is the only game in town

I always felt most of us on HN were well rounded individuals--interested in most areas of knowledge. Looking at the random articles that get upvoted along with ones on programming, it seems to coincide with that premise. Sure, we talk quite a bit about programming, but it's a just a tool for automating the repetitive work we do and to make ours and other's lifes easier. It's a means to an end and also something that happens to interest us at the same time.

uchi• 13 years ago

One of the big detractors that made programming very difficult for me was the fact that I couldn't find any practical application to what I was learning. As a result I felt unmotivated and dragged my feet for a very long time. Double clicking on Eclipse and following book instructions on how to make an array in Java for a class wasn't particularly interesting or fun to me. I always chucked it up as not bothering because I wasn't a 'math person', and this was probably something that only people who liked math would do. (I know I know)

It was only years later after I began delving into Linux and found a need to make something a bit more powerful than a bash script that I began developing an interest in programming again.

I'm sure there are many other capable and willing folks out there right now who feel the same way that I did. They want to learn but they don't see the point of it. Most courses at least don't really teach any real world applicability to what is being studied.

henrik_w2• 13 years ago

Exactly. Programming is a means to an end. One of the great joys of programming is making something that is useful, especially if it is useful to somebody else (points 1 and 2 here: http://henrikwarne.com/2012/06/02/why-i-love-coding/)

mindcrime4• 13 years ago

I had a similar experience, but from a slightly different perspective. I always found programming for the sake of programming to be sufficient motivation to spend time writing code. But when it comes to learning math, I have a really hard time caring about math in the abstract. Sitting me in a math class and having me memorize equations and formulas and operations, and solve a few word problems here and there (two boats leave point A, one going west, one going east, after one hour, blah, blah, blah) leaves me bored shitless for the most part. But if I have an application that matters to me, for the math I'm learning, I'm FAR more interested and willing to put in the time to learn the stuff.

Now that I'm trying to get deeper into Machine Learning stuff, I find myself motivated to go pick up books on Statistics and Linear Algebra and dig in, when I had very little interest in that stuff before.

Natsu2• 13 years ago

> The profession of programmers who only know how to program should disappear like the profession of scribes who only know how to write.

I think it would be better to say that the new programmers would replace the old as writers replaced scribes. Because we do have people who only write, it's just not quite the same.

gcv• 13 years ago

Writers are not merely new scribes. Writers are people who use the written word for some specific purpose. Very coarsely, I claim that novelists are storytellers, poets are artists (who use words as the medium, like sculptors use marble or bronze), biographers are historians, textbook authors are teachers, journalists are investigators. For all these subtypes of "writer," the process of getting words on paper (or into a Word document, or an Emacs buffer!) is simply a means to an end. They certainly don't "only write" — at least not the ones who produce anything remotely worth reading.

jacques_chester5• 13 years ago

> The profession of programmers who only know how to program should disappear like the profession of scribes who only know how to write.

Everybody "knows" how to write.

There's a difference between literacy and mastery.

Without fluency in the solution domain, the problem domain will simply remain that: a problem without a solution.

The Other Jacques's point was that as developers we often see lots of problem domains and that learning about them is useful.

I tell my customers that part of my job is becoming an "instant expert" in the part of the world they've called me in to help with.

The only subject I can think of with similar professional requirements would be law, where deep knowledge of the solution domain is required and quick acquisition of the problem domain is a predictor of success.

yareally2• 13 years ago

I guess it all depends on the student in question. Me, I majored in Computer Science, but I love all areas of science and study them on my own. Same with History, Art, Philosophy, Music (played violin & clarinet) and Literature. I was probably one of the few odd students in Computer Science at my school that actually loved the creative writing and art courses he took. I am not a great artist, but I can talk to someone who knows way more about art than me and relate to what interests them to a point. Such things lead to forming connections with people and exchanging great ideas across knowledge gaps between them. It also helps you to form better analogies and metaphors for explaining difficult, technical concepts.

I would have actually gone into teaching History if the United States school system was not as limited as it is these days in what one can teach students. I probably read and study history as much as I do about software development. However, I love Computer Science and have no regrets about going into it instead.

Any student that does not strive to be well-rounded in as many areas of knowledge as they can outside of their field is really missing out on life. Entrepreneurship in general is all about knowing what will make life better and it's hard to know what needs improving if one does not have experience in areas outside of their field.

A person does a great disservice to themselves when they choose to isolate their realm of knowledge to a narrow field, no matter which field it is.

wpietri• 13 years ago

If I were to get all novice programmers to learn something else, it would be some practical anthropology. How to observe. How to interview. How to see and record behaviors, triggers, social interactions, power dynamics. How to discover motivations, attitudes, and they way people understand the world. The one thing all of our software has in common is that it is supposed to make a difference in somebody's life.

If I could demand a second thing, it would be the basics of business. A large portion of software is directly about business, and almost all of it is somehow shaped by how business works. E.g., I'm volunteering at a non-profit this month, but the developer I sit next to is hip deep in a risk management problem.

martinced2• 13 years ago

And yet another cheap stab at many people reading HN. HN is filled with negativity --if not downright hatred-- towards nerds lately.

Items of zero value? A certain sci-fi author would beg to differ with you. Even someone that would have zero knowledge of anything else but would be mastering programming is still a technopriest. Even with zero knowledge about anything else but computers one still understand more about today's world than most people out there.

I'm not advocating refusing to learn other domains (I'm the kind of person who, when writing a book, decides to learn typography and typesetting software so I can typeset the book myself) but your cheap stab at young programmers (I'm 40) is uncalled for.

The world is going to shift to even more 0's and 1's. Many things are going to disappear and be replaced by 0's and 1's.

If there's one thing common to every single profession out there is that people need to read, write and communicate. And you know what? The tools allowing to read, write and communicate are written by the technopriests.

It's even more "scary" than that: the very infrastructure in place allowing people to communicate (like electricity and Internet) are, themselves, run using software. Not to mention very basic need like the entire infrastructure providing clean water to fullfill one of the most basic Maslow need.

There's no need for someone taking care of a network to understand what's inside the packets of information running through the network. All they need to know is that if there's no more Internet packets travelling worldwide, then there's no western civilization anymore.

The very foundation the entire society now relies on is software. If a worlwide EMP blast would render all computers useless tomorrow the world would probably turn to a mad max horror scenario in a few hours. No more water. No more electricity. No more food. Our logistics would be so screwed that "advanced" countries would fall into chaos. It's probable that in some places people would die after a few days because of the lack of drinkable water and it's very probable that in a few weeks many would die due to lack of food. Not to mention the nuclear facilities which would leak radioactivity left and right and break havoc.

Doctors aren't forced to understand how computers works to do their job. Most doctors are utterly clueless when it comes to technology: they're bound to live in fear of being hacked and not understanding what's going on when they connect to their online banking account and not understand why this or that mainstream article tell them to disable Java in their Mac. I've got doctor friends: they simply don't understand the world they're living in. We're already in the technopriest age. And they're asking me questions, lots of them.

The most computer-wise knowledgable doctor I know was thinking that the latest state of the art way to diagnose using computers was still to use an "expert system", filled by doctors. That's not how it works. These "experts" aren't able to process the amount of information a computer can process and can't find the correlation a computer software can find. Which is why now machines are trained with gigantic amount of information and suggesting exams and deceases to look for that are way more relevant than what doctors would suggest.

I think you got it totally backwards: it's a lot of the other professions that are going to disappear. Replaced by robots, machine learning algorithms, software and... The technopriests.

The ones still able to somehow understand the world they're living in and still able to write the world they're living in.

P.S: regarding mainstream economics I'd like to point out that Krugman, Nobel Prize winner, recently said that it wasn't a bad idea to mint a one trillion coin to deal with the U.S. debt SNAFU. Sure. Why not mint 26 of them and be done with the U.S. public debt!? That is the kind of economics most economics and most universities do teach worldwide: Keynesian economics. I happen to think Keynes was all wrong and keynesians have never been able to predict the consequences of their interventions more than a few months/years forward. There are economists from other schools who predicted 15 years before it happened, before the first euro was even circulating, that the euro would lead to: too many houses in spain, too many public servants in France, too many industries in Germany and who explained by which mechanism Greece and Spain would eventually default. Not a single Keynesian has ever been able to do a prediction that precise. That's all I need to know about "economics": that the mainstream is all wrong.

I do also think you put way too much faith in doctors: these people really aren't nearly half as expert as you think in their field. And they are the ones getting replaced today by sofware and robots. There's simply no way a surgeon can be as precise and have as many axis of rotation compared to today robotic arms (explained to me by a surgeon friend). There's also simply no way a doctor can read blood samples and other exams results and be able to correlate that with hundreds of thousands of other results and hence make more correct deduction as to what's going on.

If we eventually end up one day with true AI then everybody becomes meaningless because that day there ain't going to be doctors nor economists anymore.

Until that day programmers (even the one knowing nothing besides 0s and 1s and how 0s and 1s move around) are going to be more and more like the technopriests.

hef198983• 13 years ago

I didn't read it as a stab at nerds or young programmers but more like an advice that domain knowledge still is important beyond pure programming skills. Maybe the comment was formulated a littel bit over tthe top, but well, I'm doing that quite regularly.

I completely agree that the world is running on software right now, but the world isn't software ITSELF (well, not yet).

On the other hand, people being cracks in there field claiming that there field is the only one that counts are just arrogant. No mater if they a programmers, accountants, mech. engineers, doctors, car mechanics or what not. And yes, software written by people without the slightest idea of the purpose of said software tends to be not so good.

TheCoelacanth• 13 years ago

The trillion dollar coin wasn't supposed to deal with US debt, it was a trick to get around the debt cap rule that Congress imposed on the government.

If they actually spent the trillion dollar coin, that would likely cause devaluation of the dollar; as long as the coin remains locked up in the treasury it's just an accounting trick to get around their own arbitrary rule and wouldn't cause devaluation, because everyone buying dollars knows it's just a stupid accounting trick and not actual money.

potatolicious• 13 years ago

I think you two are coming two extreme ends of the spectrum. He's coming from the "clueless one-dimensional engineers" end of the scale, and you're coming from the "we are digital gods" end.

The truth lies somewhere down the middle, though I think you're a bit more extreme on your end of the scale than he is.

It's hard to take your views seriously when you describe your kind as "technopriests", and claim that people understanding nothing but computers "understand more about today's world than most people out there". This is a laughable claim.

I'm sorry, I can't get over this. Technopriest? This is the real world, not Snow Crash.

> "Many things are going to disappear and be replaced by 0's and 1's."

Yes, and they are being replaced by 0's and 1's in specific spaces where those doing the replacing must understand non-computer things about the space. Want to automate the entire traffic control system (LA just finished doing this)? Sorry, you have to know a great deal about transportation.

Want to automate food production? Sorry, you must know a great deal about both manufacturing and food. You can know all you want about computers, but unless you know these other topics you're useless. Well, you could be coding to a spec from someone who actually has a broad base of knowledge - a cog in the machine at that point.

The future belongs to people who have a deep understanding of computers and [insert relevant field]. Those who understand just computers have only a future pounding keyboards and not making decisions.

You know what, you're right - there will be "technopriests" in the future - but they will be the ones with a broad base of knowledge on top of technological mastery. They will not be the ones who understand tech and nothing else - those will be the digital equivalent of janitors.

Though they might fancy themselves technopriests while the world passes them by.

MrMan• 13 years ago

Is an incoherent rant like this an uncharacteristic indulgence for a technopriest? Or are you just a fan of technopriests, not being one yourself?

Everyone in our increasingly automated gift economy lives in fear of being cut off from the support of the layers of technology they rely upon for all their needs, all the way up your vaunted Maslow hierarchy.

You shouldn't romanticize this state of affairs, and you should not crow about the special role of the priestly class. In my opinion you should think soberly between now and your 50th birthday about how you can contribute to a global society that is both more stable and more technologically advanced, and try to forget about the futurism that was peddled to you ( and all of us) by Wired magazine in 1994.

bjhoops1• 13 years ago

I love the "takedown" of Keynesian/mainstream economics in the middle of this rant.

Any other centuries-old disciplines your knowledge of Big-O notation and compilers qualifies you to discredit? Oh, modern medicine? Good for you.

This guy takes the "hubris" as a virtue line way too seriously.

reactor• 13 years ago

Scribing was easy from the onset. But programming as it is today is not easy. Unless it evolved into plug-and-play-the-building-blocks, the profession of programming is going to stay.

alemhnan2• 13 years ago

My take is that one of the main improvements in programming languages will be to reduce the gap between domain experts and programming experts.

Now we should face three cases: 1) the programmer needs to learn the domain; 2) the domain expert needs to know how to develop; 3) something in between.

Most of developing is done in 1). There are some brilliant cases of 2). I once talked with a biologist doing some multi sequence alignment that told me that some routine in C was quite slow and was far better rewrite that routine from scratch in assembly. I was quite speechless.

In order to create better programming languages we could: 1) express domains in a better way. Actually, just be able to express domains would be really cool (and not so easy). 2) provide higher levels of abstraction.

jacquesm2• 13 years ago

Every programming language from the dawn of time up to the present has made that claim at some point. This time it would be different.

I think it is just a people problem. Programmers tend to seclude themselves from their customers. Whole IT departments tend to seclude themselves from their users.

When you are looking for underwater welders for the off-shore industry, what is better, teaching a welder to dive or teaching a diver to weld?

(that's a serious question with a real answer).

It is much easier for a programmer to acquire the required domain knowledge than it is (usually) for a domain specialist to learn how to program. But you can only do so if you're prepared to 'dive in'.

unwind• 13 years ago

My guess would be the first, teaching a welder to dive.

I base that on the fact that the core value, the welds, will be higher-quality and that's more important than the diving execution.

The diving might be off, but it shouldn't be that hard to learn well enough to survive and be effective. Of course, I'm neither a diver nor a welder.

Did I win?

caw• 13 years ago

Funnily enough, I know someone who was an underwater welder. From the conversation with him, it was a multi-month certification course, so I went ahead and looked it up online.

https://www.commercialdivingacademy.com/underwater-welding.d...

That's just one program, but it appears diving is the first thing you learn.

Edit: If you look at it from the context of a normal welder, they have an understanding and control of their environment before they start welding. You don't realize it because you've been understanding your environment since you were born. To do underwater welding, you'd first have to understand how you operate in that work environment, and then they can teach you how to start welding in it.

jeltz3• 13 years ago

> I think it is just a people problem. Programmers tend to seclude themselves from their customers. Whole IT departments tend to seclude themselves from their users.

My experience is that it generally has been managers shielding the programmers rather than the programmers shielding themselves.

hef198983• 13 years ago

Maybe that is a little bit of the mark right now, but wouldn't that domain knowledge be a great opportunity for a non-tech co-founder to add value to a start-up? Just guessing, but there are a lot of domains out there that are quite diffcult to learn but that are also (or maybe because they are difficult to learn) ripe to be changed by software.

I could imagine the right match of domain knowledge and the capability to learn how to develop in a non-tech guy and programming skills and the capability to learn the domain in a tech guy would be rather powerfull combination. And pretty rare one, I guess. At least for me, the actual programming part is... difficult. ;-)

On another point, this applies pretty much things like lean or six sigma, too. Both are quite usefull (or can be, that depends!), but people tend to see them as domain by itself. In practice, and that's only my personal opinion, this comes out short. If you add this to existing expertiece in a certain domain, then it works!

erichocean• 13 years ago

In order to create better programming languages we could:

3. Create another EUP paradigm. It's been 34 years since VisiCalc came out, and despite it's success, I don't see a lot of effort to get end-users to program, especially in models, like the spreadsheet, where they don't know that's what they're doing. ;)

If you're interested in that sort of thing, ping me (see profile). My company does R&D in this space.

Nursie4• 13 years ago

Actually I think domain knowledge is overrated. A good hacker can pick up the domain knowledge needed to contribute to a project very quickly.

Now, if you're designing a product to solve a problem in a particular domain you need the knowledge ahead of time, sure. But if you're a shit-hot coder and a fast learner, brought in to make things work, who cares?

rcfox2• 13 years ago

Because, as the article clearly mentions, there's a difference between making something work and making something work right.

Nursie4• 13 years ago

I disagree that it's important unless you're the person designing the solution. If you're not (and in many/most teams you won't be) it's far less important.

qxf23• 13 years ago

The line I found most insightful: >>"Software is much more intertwined with the institution commissioning it and the people using it ..."

Given that, I'm surprised at the emphasis on 'reading' as the primary source of gaining domain knowledge. There are just 2-3 mentions of speaking/collaborating with the domain experts that are driving you to build the software itself. If you are lucky enough to work with acknowledged domain experts, spend time talking to them. Pick their brain. Two techniques I use: a) forward interesting articles to them with a question or an analogous concept in my domain b) resist the urge to show them a 'better' way. E.g.: In the HealthCare domain, I learnt people don't really want quicker or lesser clicks. They would prefer a set, stable workflow. If you cater to senior-citizens, keyboard shortcuts are less important than instructions in large font.

I still suck at engaging domain experts. So, I'd love to learn more techniques to hold an interesting conversation with domain experts. Ideas?

jacques_chester5• 13 years ago

I can read a book faster than I can speak or hear words.

In addition, books can usually be found, on almost any topic, which are intended to bring a novice up to a basic level of conversancy. Reading one (or two) such books will make discussions with the experts much more productive.

Plus: it's fun to learn about new things.

qxf23• 13 years ago

Oh, no argument on the fact that reading is extremely useful. I read too. In expressing the balance between reading and collaborating, I felt that your article emphasized too much on reading - but I fully recognize that you do collaborate.

>>"I can read a book faster than I can speak or hear words."

I disagree that this is an argument in favor of reading. Reading is easier but I think its not always as high in quality as collaborating.

>> "Reading one (or two) such books will make discussions with the experts much more productive."

Totally agree. I try my best to read as much with the end goal being picking the brains of my colleagues.

majc22• 13 years ago

I'm surprised about the emphasis too. In terms of ideas, I used to work in a large corporate on the technology side - now work for a software company, so YMMV with these:

One of the first projects I worked on involved re-writing an entire admin system, both the technology and the underlying processes. One technique that worked well for the team, was to work-shadow; we would each spend time watching how a couple of people worked; compared notes and would go back and ask questions if we didn't understand it. Work shadowing doesn't have to be project related either; I visited nearly every department in our division of the over the years (1 every couple of months). I even managed to swing going out with the sales guys and visit clients; the clients were happy to spend an hour talking about their business and problems. Anyway, my experience with large corporates if that if you ask - people are generally happy to help you out - but I think many people just censure themselves from asking in the first place.

One other technique that I think works quite well is to get the strong end users seconded to beef up the QA team. Again, this doesn't work in all environments. But in the context of developing an internal system, it worked really well. For an external facing system, we did get a sales guy seconded onto the team too - which again really helped build up the knowledge of the whole team.

A variant on reading is podcasts and forums - I've used both of those to get to grips with whats going on in an industry.

For the software company, most of the above doesn't work, we're adapating Steve Blank's mantra of "get out of the building" and just trying to talk to as many people (especially customers) as we can. If they agree to talk to you, they're generally going to be helpful.

adrianbg• 13 years ago

Programmers should learn more about other domains and domain experts should learn how to program. Programming is becoming nearly as basic a skill as writing.

mathattack2• 13 years ago

This is a very true post, but I don't think the solution is at the university, at least for undergrads. Schools barely have enough time in 4 years to create entry level programmers. Lopping off half the coursework in exchange for domain knowledge leaves grads unprepared.

My observation is if you can learn to program well, you can learn any field of knowledge. A programmer can learn consumer products easier than a brand manager can learn Python. Similarly a CS major can learn bond math easier than a Finance major can learn database design. (Yes, these are generalizations)

What's this mean? That bridging the Tech/Client divide is on the shoulder of the technologist. In addition to learning the latest frameworks, we have to learn our customers business. In addition to tech credentials, we should add theirs too.

jacques_chester5• 13 years ago

The degree gives you a tour of the foundations of the solution domain.

Mastery of the solution domain and knowledge of various problem domains cannot be taught in a 3 or 4 year degree.

mathattack2• 13 years ago

Agree completely.

marpalmin• 13 years ago

Perhaps we should all work in compilers ;)

dschiptsov• 13 years ago

Another metaphor for necessity of Domain Knowledge is translation. Give an average translator a technological text full of idioms from this technology domain and you will laugh your ass off.

I remember reading reports about some SAP installation process (written by idiots) full of stupid special terms such as "System Landscape" and "User Privileges" translated by an English teacher.)

Why we have this as a top news? Isn't it obvious that you must know the slang before you're trying to speak, let alone translate?

Well, most of coders use a slightly different strategy - jump right into IDE and pile it up, using all that containers and design patterns they have heard of (without understanding, why bother?) until it satisfies the manager (compiles) who know absolutely nothing (this is why everything must be statically typed). Then test it like a black-box - correct input produces correct output (never try incorrect or unexpected) and then ship and forget.)

euroclydon• 13 years ago

I've held programming jobs in: Water Treatment Process Control, Workflow Management, Clinical Trials Management, Electrical Power Monitoring and Simulation, and Medical Imaging. In each case my natural curiosity led me to learn as much as I could about the field. I always enjoyed talking to the domain experts at each company and asking them as many questions as they could handle. There are usually one or two people who relish answering these questions and I especially like the ones who talk fast, mock astonishment at my ignorance, and proceed to try to fix it.

In water treatment, I learned about oxic/anoxic bacteria and how they're grown and recycled in the process to treat waste, how to write chemical metering logic in a PLC because I was the only one left on-site to do it, how motors are controlled with variable frequency drives, how proportional-integral-derivatives are used to prevent oscillation in feedback based controls.

In clinical trials management, I learned how double-blind studies are made, the process of randomization and dispensing drugs, medical coding and the life cycle of the trial.

In power systems simulation, I learned about three-phase power transmission, resistive and capacitive loads, active and reactive power, the challenges of hooking up a lot of solar panels our existing grids, how data centers are build and cooled and powered, microgrids, power meters, intelligent breakers and more.

I feel privileged that my skills at software development have exposed me to a variety of fields and that I know how several important processes in our word work. And I got to take home a nice salary while gaining that exposure. I do lament that, unless I pick a field and stick with it, I'll never get to work on the difficult programming problems that require a decade or more of experience and knowledge.

danboarder2• 13 years ago

This article is talking about what I learned as classic Systems Analysis and Design. It is an interdisciplinary intentional practice of seeking to fully understand the business processes involved.

See http://en.m.wikipedia.org/wiki/Systems_analysis_and_design as a starting point.

treerock• 13 years ago

So have you ever tried what the articles suggest? Ever spent a week working on the 'shop floor'?

In my experience of public sector software projects, the programmers seem to be kept as far away from the end users as possible.

danboarder2• 13 years ago

As a independent contractor / systems analyst, yes I have. However I agree, most solutions providers don't bother.

markbao• 13 years ago

Turning the tables around: perhaps the software (or even physical products) that we use where we think, "this is stupid! why would anyone build it this way?" were similarly built by people with no domain expertise in the product.

Introduce a software engineer with an interest in the subject matter into that environment, and provided it isn't too big (like government), it might be self-correcting.

smoyer• 13 years ago

I believe this is actually the major problem faced by outsourcing projects. When I was in the cable industry, I saw several projects fail when the "Systems Engineers" (who were in charge of writing the specifications) were unable to convey enough information to the developers.

There's a real limit to the amount of "truth" you can put into a specification, and at some point you have to trust the developer to "do the right thing". If they don't have domain knowledge in the industry they're writing for, they don't stand a chance ... and so your project fails even when everyone works hard and has good intentions.

martinced2• 13 years ago

Sadly in some domain it's not possible. I've had cases where to understand a particularly complex business logic case it turned out nobody had any idea how that case should be resolved. Lawyers / jurists, employees, managers, etc.

We tried to go up the chain up to someone, anyone, who could explain how that case should be resolved but nobody could tell.

Why? Legalese. And just like most legalese documents: there's more than one interpretation possible because the very concept of legalese (based on thousands years old tradition) is inherently flawed and open to interpretation.

So, somewhere, some politicians, probably out of bikeshedding fashion, decides to vote an amendment exceedingly badly written and exceedingly complex. Then it becomes law. And law has to be followed. So that case has to be taken into account by the software but isn't (yet).

Everything is all and well until that specific case appear. The day it appears all bets are of. It's not working and nobody knows why. Nobody knows what is not working and why it's not working, nor how it should work. The law is bogus and the business case cannot be resolved without contradicting with the law(s).

Trying to determine this ended up with the programming team receiving a PDF with tens and tens of pages of legalese text that nobody up the chain (including lawyers) could explain to us. Probably that the politicians who voted these changes couldn't even themselves explain it.

We simply couldn't make the changes.

How did it end up? Someone very high up saying: "That's what should be done in this particular case" and then everybody covering its arse.

And there came the technopriests: fixing the production DB...

raverbashing2• 13 years ago

> How did it end up? Someone very high up saying: "That's what should be done in this particular case" and then everybody covering its arse.

This is usually how it ends up.

I've been close to similar situations (but while creating something, not changing something that already exists), but usually, if nobody understands how this is to be done, the fact that you can't code it is second in importance.

jedbrown2• 13 years ago

In science and engineering, it's common to hear of "Computational X" where X is "Plasma Physics", "Geodynamics", "Ocean Science", "Electrical Engineering", "Aerodynamics", or any of a myriad of other disciplines. In these fields, the primary task is formulating Science/Engineering questions that can be effectively answered using computation, then writing the software to do so.

There is also the more generic field "Computational Science and Engineering" (CS&E) [1], which is primarily the development of methods and software that is applied in many specific disciplines. I work primarily on the library development side of CS&E, working with many applications to understand their needs, often finding ways to satisfy those needs with better software reuse and better ultimate flexibility, then implementing any components that belong in libraries. Additionally, we do basic research by recognizing unexploited common themes and designing new algorithms that will benefit existing and potential users.

There is currently a great deal of demand for Computational X, for CS&E through the postdoc level, and for CS&E in industry and national labs. It is recognized that a versatile practitioner of CS&E can effectively assist Computational X for many values of X, since there really is a lot in common. Once you've put in 10000 hours of CS&E, plus 1000 hours in a handful of disciplines, you should be able to get orientedy quickly in new applications.

At the faculty level, support for CS&E is highly variable. At some universities, X departments have the view that all their faculty should "X first, computation second", while the mathematics department wants to see theorems and isn't overly concerned with following through to see the method applied to obtain a new science or engineering result. When hiring committees say "we need CS&E, but some other department should have the CS&E faculty", it drives a lot of talent away from teaching roles (e.g., into industry), perpetuating the relative lack of early education in CS&E fundamentals. On the other hand, the trend is for universities to have more direct interest in CS&E (sometimes recognizing it with a dedicated department or inistitute). It helps that computation stimulates many interdisciplinary collaborations, leading to more diverse funding possibilities.

[1] We have a big meeting next week: http://www.siam.org/meetings/cse13/

pmahoney• 13 years ago

What sort of backgrounds do people working in this field have? More specifically, I have a bachelor's degree in materials science and currently work doing run-of-the-mill web development (were the typical background of my peers is a bachelor's in CS). What steps might I take to move into computational engineering?

jedbrown2• 13 years ago

There are a couple journals dedicated to computational materials science. Two active fields are phase field models and molecular dynamics.

Phase field models are mesoscale continuum (PDE) models based on the Cahn-Hilliard and Allen-Cahn equations, and are useful for things like predicting crystal grain growth and reorganization under strain, as well as fracture and many other uses. In phase-field modeling, the interesting physics is a consequence of the free energy functional, which is phenomenological and different for each material, so software should make it as easy as possible to turn specification of the free energy into an accurate and efficient massively parallel simulation. One such approach is outlined in this paper [1] from some colleagues. The resulting discrete systems are stiff and have positivity properties that must be preserved, so scalable implicit solvers (like multigrid) as well as methods for variational inequalities are necessary.

In molecular dynamics, there are many parallel scalability and expressivity challenges. Fast algorithms for long-range interactions (such as the fast multipole method) lead to less regular computational, but huge asymptotic (O(n) versus O(n^2)). Since better algorithms become more important as our hardware allows us to keep scaling up the problem sizes, Fast methods need to be applied in more places.

There is also a lot of activity in multiscale modeling and numerical homogenization, where explicit microscale models (often kinetic) are used to provide missing components of meso/macro-scale continuum models.

[1] http://dx.doi.org/10.1016/j.commatsci.2011.07.028

hawleyal• 13 years ago

OP might be right if they choose to write software without collaboration with clients or subject matter experts.

IMHO, with collaboration, it's more a matter of providing a hyper-rational process by which the purpose and priorities of the target audience(s) are focused and honed, and then producing the software as feedback, automation, and a usable compliment to their needs.

keithpeter• 13 years ago

Does anyone here find the work of Lave and Wenger ('situated learning') of any use in their attempts to understand a problem area for a customer?

http://www.infed.org/biblio/communities_of_practice.htm

Basically, some organisations make their processes easily visible to most employees, and some don't. The ones that do can train people more quickly and improve their processes more quickly. Organisations that make 'peripheral participation' easy might be better customers for developers than those that don't. You might not have to make many sails or use a height gauge too often to 'see' what the issues are, just interview knowledgeable people carefully and cross reference the results with observation.

As a concrete example, the original author invested a considerable amount of (quite expensive) time in learning about sail making and lathe operating. In some of the 'business applications' I have to use daily, it is pretty obvious that the customer was a manager higher up and that the interface we have to use to populate the database that generates the wonderful reports has not really had much time spent on it!

DirtyMonkey• 13 years ago

I agree, but this is something I've found resistance to in the past. For example, last year I was building an automated shipping system for a client and I wanted to spend a few days in the shipping department - this was flat out refused as the management didn't see it how it would be useful, since the process was fully documented. Turns out, much had to be rewritten because the shipping staff were (predictably) doing 'undocumented' things to speed up their workflow, the new shipping system messed this up and made them slower.

mark_l_watson• 13 years ago

Jacques is right-on. For a lot of the work in my life, most of the effort was coming up to speed on a domain rather than just technology. Much of my work has been in 'artificial intelligence' (for some definition; never liked that term because it is too general) so domain knowledge is central if you are trying to get human level or better performance in some narrow domain.

Having to spin up on new domains also keeps work and life interesting.

bwooceli• 13 years ago

So I don't work in a scientific field, and I'm not a "programmer" or "software engineer". I am a business analyst. I am a business analyst who CAN code. My code isn't always pretty, and I recycle a crap ton of boilerplate and hack the hell out of django apps. And I read the heck out of HN and have spent 5 years slowly refining my design chops. But most importantly, I know top-to-bottom the ins and outs of my business. So when our call center shopped around for a software solution to provide a call center full of agents realtime access to their stats, we were given a 3 month time frame and quoted $100k+ for a solution. So I wrote and deployed a web based results dashboard of realtime stats over two weekends. And we were just quoted a price on a quality management tool for $225k. Guess what's next for me? I've been an agent on the phones. I've sold in a retail environment. I've been the corporate trainer dealing with the LMS. Domain knowledge plus tech chops has proven a powerful combination for me.

jiggy20112• 13 years ago

I agree about the importance of gathering as much domain knowledge as possible.

One thing that I have always had issues with when gathering requirements around something for an unfamiliar industry is that people who are in that industry will often neglect to mention things that are both important and non-obvious/counter-intuitive to someone unfamiliar with the industry.

This is probably because they are so familiar with the quirks of the way that the particular industry works that such things seem so obvious as to not need mentioning (and they probably don't see how it could possibly be any other way). Basically similar to how we are often amazed by assumptions some people have about the way that computers/software works.

The paradox of this though is that to deliver truly good software for a particular industry one must really know that industry as well as somebody who has years of experience in that area. At which point you start to feel that you could do their job too.

Another alternative is to have a very tight feedback loop with someone who has tons of domain knowledge and is the target user. Basically sit next to them as you develop the software and have them review every single change you make.

vicbrooker• 13 years ago

I think in order to create good software that fits the industry's current best practices then you do need a really in-depth domain-specific knowledge. If that's the goal then learn everything possible about that industry.

To create good solutions that are also innovative, I think that programmers only really need enough knowledge in that domain to be dangerous - to know more or less what happens but with enough gaps to not really know what's doable and what isn't. But, this knowledge is only useful if you can combine it with bits and pieces from several completely separate domains.

If innovation is the goal then I think there is a point where domain knowledge actually hurts you.

Of course there still needs to be at least one specialist to keep everything on track though - and I completely agree with you about having someone with deep knowledge on board. I think it should be someone who has been in that industry and will stay in that industry.

I can't see the point in programmers trying to become experts in areas that they're not interested in working in for the long haul. It's easier to learn (and actually apply) a little about a lot than a lot about a little.

east2west• 13 years ago

HN is understandably filled with programmers who value programming above all else and believe domain knowledge can be picked up quickly. It would be true for someone hired as a programmer who implement a published algorithm or port existing software to newer systems or different platform; in other words, improving existing systems. I believe it requires equal depth if not not more for domain specific knowledge to innovate in non-computational fields. For my field of bioinformatics/computational biology, many open questions cannot be answered because of computational hurdles. Reading textbooks won't lead anyone to them, nor would reading survey papers. It requires expert knowledge in both biology and computer science to even know what is feasible and interesting. We want to find the driver mutations for cancer, discover mechanism of gene regulation, uncover subtle interaction of heredity and environment. Computation is intimately involved because observation is indirect, noisy, and massive. Only with knowledge in both fields can answer questions that domain experts care about but cannot do because of computational/quantitative obstacles.

I find researchers (I am in academia) in both fields cross-talk too much. Computational/quantitative experts try to shoe-horn biological questions into their small corner of research area, while biologists don't care about computational efficiency, statistical robustness/efficiency, and the mathematical implication of their simple methods; no one cares about software quality. This is unfortunate. It is also unfortunate hiring committees do not look favorably towards multi-disciplinary researchers because one is not a trained geneticist/cancer researcher. I definitively do not encourage computer scientists go into support roles in academia; they are poorly paid and get little respect.

joshwa• 13 years ago

Reminds me of the old "what the customer actually wanted" comic:

http://imgur.com/NXHbZ

trvd1707• 13 years ago

I think domain knowledge is important, but sometimes it can be a trap that hinders inovation. As Pierre Bayard put very nicely in "How to talk about books you haven't read", when we don't have domain knowledge we open ourselves to questions that we wouldn't make otherwise.

Domain Knowledge or a Lack Thereof

Related Stories