Ask HN: I have a mental block designing software. Time to quit?

https://news.ycombinator.com/item?id=10169937

by porker • 11 years ago

56 85 11 years ago

news.ycombinator.com

This is embarrassing to admit after 15 years building web applications, but every project I hit a brick wall decomposing software.

I know the data inputs; I know the outputs. My prototype works. But my mind freezes when I try to rewrite the prototype into something maintainable that will run for 5+ years.

High level planning is fine, it's getting from what needs to happen to how to decompose into classes/methods/libraries, dependencies (logging everywhere) and organise message passing.

My current project is choreographing 3 enterprise systems to work together. Once past the lack of documentation it's conceptually simple; my code however is tangled.

Related, I have a mental block with drawing flow-charts. I can't isolate tasks, decisions, actions. The chart becomes spaghetti, or too detailed/too sparse to be useful. This is difficult when people want the system described on paper.

Either:

1) I need to learn something 2) It is an innate mental block, and I need to change fields

If I'm going to continue in software this needs to change, as I don't feel I'm delivering quality. The more senior I get the less I can stick with prototyping or "Just Works" as an outcome.

sebastianconcpt• 11 years ago

Sounds like a challenge on synthesis and perhaps design problems. I suggest you to not do anything drastic but use a pair of fresh eyes on your project. Manage to pair with someone and observe what this new guy finds challenging. You have to be open to refactorings or code style improvements. Also the guys must have to be good otherwise it could misguide you. What about using 3 instead of one? if they don't know each other you know the biases are under some control. After getting that quality feedback, you can ask yourself those questions again. You'll know better

porker23• 11 years ago

Good advice & I'd like to, my challenge is I work freelance/direct and don't have other programmers around to ask.

I think paying for code reviews and some mentoring is the only way I can get this feedback.

moberemk• 11 years ago

Why pay for review though? If you're working on open-source stuff you can try http://codereview.stackexchange.com/ for help, or reach out to some trustworthy people in your network as a fresh pair of eyes.

KuhlMensch• 11 years ago

https://www.youtube.com/watch?v=GAFZcYlO5S0

https://www.youtube.com/watch?v=iukBMY4apvI (javascript module workflow that helped me (frontend dev) to transition to modules)

help?

porker23• 11 years ago

Watching Simon Brown now & will look at his book too, thanks.

To take the current project, I have a CMS, Salesforce and ExactTarget. I'm writing the glue to automate taking new content from the CMS, finding SF users with the right permissions to see it, and emailing it to them via ExactTarget.

All good & working until... do I model it as 3 objects? Which object is responsible for creating the DataTable at ExactTarget (populated with SF data)? How is data going to be passed around cleanly between my representations of all 3 services? Or should I represent the system more abstractly as EmailToSend, EmailContent and use all three services within these classes?

mistermann2• 11 years ago

Code it each way and post it online and someone will tell you with absolute certainty that you are Doing It Wrong because <some fashionable pattern>. Take this person's recommended implementation and post it online and someone else will tell you, again with absolute certainty, that you are Doing It Wrong because <some other fashionable pattern>.

All the while never mind that in 9/10 cases you could have written it either way and it would be of sufficient quality and performance, run more or less without issues for years, be conducive enough to periodic refactoring, etc.

I have a similar feeling as you about software today, although I'm not sure if for the same reasons. I look around me and everything is just so damn complicated. Writing code that is small and understandable is ruthlessly mocked, while a gigantic, complex marvel of engineering (that does the exact same thing) is looked upon as The Right Way. Functionality that would take a week or two to implement 15 years ago now takes a month, or three.

rmetzler• 11 years ago

I think, you overthink it.

Make 3 scripts

1. get-content

2. get-emails

3. send-content-to-emails

call this in a cronjob. Have logs.

The second day you will realize, you'll send the same content to the same people, so you'll need to add a filter. A simple file or database-based filter will be sufficient. You should be able to generate this filter from the logs.

Add this steps before sending emails:

* filter-emails

* filter-content

* generate-email-to-content-relation

gdy• 11 years ago

In my opinion it's better to define criteria first and then evaluate possible solutions against them. What is more important for your code - to be fast, correct, easy to understand, fix or extend? How each possible solution scores on these scales?

stdbrouw• 11 years ago

Relax. Anyone who writes software will code themselves into a mess sometimes. Learn to love refactoring.

jordz• 11 years ago

Edit: This. One of the things I see a lot of people running into is this mis-conception of "prototype" -> "production". The prototype is your proof of concept, you take this application and you refactor it and optimize it as you move into your production life cycle. You take the bits out and put new bits in that are either un-optimized or don't work for the entire scope of that specific problem.

Working code is better than nice code that doesn't work. Refactor small bits as you progress.

Raphmedia• 11 years ago

You could stay in your field, a field that you obliviously love, but try at another company where you would have better ressources / coworkers at your disposition.

Take days off too.

eigenrick• 11 years ago

What you're describing to me is you've built a very solid structure for reasoning about programs, but it is too rigid, and too focused.

I would heartily recommend learning a new language. You need to shake things up a bit.

Two reasons:

1. https://www.wikiwand.com/en/Linguistic_relativity - Our language shapes our thought. This is true for spoken and programming languages.

2. You cannot properly understand a language unless you learn another. This is partially based on point 1, but it needs to be said otherwise.

Since you're in web, I would recommend learning Elm for the front-end, and Haskell in the back-end. Why? They will force you to design your systems differently? Is it better or worse? That's subjective. The important thing is that it is different.

For data design, have a look at things like datalog. Datomic is a good playground for that.

kovek• 11 years ago

... you've built a very solid structure for reasoning about programs, but it is too rigid, and too focused.

When I read "focused", I thought about deconcentration-of-attention.com . I don't know if it will solve porker's problem, but it is a very interesting read.

The idea of the text is that by doing deconcentration of one's attention, it is easier to notice specific details in the whole area the attention is spread over. Meaning, if one deconcentrates one's attention over a whole painting for example, one can more easily find a specific detail on said painting. The alternative to deconcentration of attention would be to scan through the painting with effort. Deconcentration of attention is essentially the reverse of the tunnel effect.

captn3m0• 11 years ago

This is the first thing that came to my mind as well. Learning a new language surely shakes things up.

porker23• 11 years ago

I would heartily recommend learning a new language.

I've installed a nice selection of languages tonight. Can anyone suggest sample projects to do as learning exercises? My ideas a) don't need the language, b) are too complex or c) involve CRUD, and I do far too much of that every day :)

TIA!

tayo42• 11 years ago

Isnt your reason 'a' the point of doing something like that? To learn how to solve problems a different way?

estefan• 11 years ago

Read "Functional Programming in Scala" or any other good book on functional programming to see things differently if you've just stuck with imperative so far. There's also http://iloveponies.github.io/120-hour-epic-sax-marathon/

Read books on concurrency ("Learning Concurrent Programming in Scala" if you carry on with scala) and then you'll find different ways to write performant concurrent code - for CRUD sites or other stuff.

I'm not pushing scala rather exposure to other ways of doing things you might not have come across - functional programming, futures, akka, etc.

But yeah, also design patterns. And have you read http://highscalability.com/ for architecture ideas?

jaredhalpert• 11 years ago

Time to take a break. Decrease the amount of time you spend programming, and increase the amount of time you spend with friends, family, being outside, etc. You need to remove yourself; I recommend balance.

mikekchar• 11 years ago

It's hard to tell without looking at your code, but I suspect that you are stumbling in to a wall that many do. What's in your favour is that you recognise it, so you will probably be able to do something about it.

Systems grow over time. At the beginning, they are simple, which means that they are forgiving. If you make a mistake, it is easy to see and correct. As the software grows, it gets progressively more complex. Problems hide for a long time and when they present themselves it is often very difficult to see the actual problem.

What I'd like to do is challenge you to look at it a bit differently. Think about the number of lines of code in the OS, system libraries, language, libraries, frameworks, etc that actually go into your code. Even if you only write 2 lines of code, there are probably hundreds of thousands of lines that your 2 lines are sitting on top of. Why are your 2 lines easy to understand?

Obviously because there is a great separation between the complexity of the underlying systems and your code. What you need to be able to do is to provide that same separation in your own code.

As you mentioned that you don't work with others, I will recommend some other approaches. However, if they don't work, I recommend that you change to an office job where you can get the mentorship you require before you leave the field altogether. 15 years is a lot of learning to give up.

#1: Read good code. One of the biggest problems that programmers have is that they rely on their own code for examples of "good code". You are at a huge advantage because you recognise that you need to improve your code. What should you be reading? Read library code from whatever libraries you are using. As you are doing "web development", I would advise you to shy away from reading the source code of frameworks like Rails, but rather read the underlying class libraries for the languages you use. If you are not using languages that are open source, then it will make your life much more difficult, but try your best to find open source and free software on the internet to study.

#2. TDD. Every time. All the time. Not BDD (although BDD is an excellent practice in its own right). You want to do TDD. This will force you to think about your interfaces and to decompose. There are many books about TDD. Some of them are good and some of them are bad. Read Test Driven Development: By Example by Kent Beck. It is probably not the best book on TDD, but it will give you the basics without confusing you.

If you are working with legacy systems and don't know how to TDD with those legacy systems, read: Working Effectively With Legacy Code by Michael Feathers. It is a very, very difficult book. Learn it all.

You may hate TDD. Many people do, but using it will make you a better programmer. Personally, I don't think it's bad if you abandon it eventually, but master it before you do. This will simplify the process for you and break it down so that you can consider things in more manageable pieces. It will move you away from "Just Works" because it is god awfully slow (or at least it feels that way). Just pay attention to the effect it has on your code.

From where you seem to be now, to where you want to be, I estimate you've got 2-5 years of effort. Don't get discouraged if it doesn't come right away. This is a discipline where you can get orders of magnitude better over time. Try to be as patient as possible.

I hope this helps.

insanebits• 11 years ago

I agree with a TDD, but it really depends on the framework/cms you're working, some of them are incredibly hard to write tests agaist. One of the examples would be writing unit tests for the WordPress plugin, if you're sticking to the WordPress way of coding unit testing is practically impossible because there is lot's of database access and events which do mess up.

You can write plugin folowing OO principles and abstract things out to be able to mock them as nessesery. It really depends on the coding standarts you're folowing.

As for unit testing. In my personal experience it enforces you to write cleaner code, because when I started I had to do heavy refactoring to be able to test my prototype, but as I wrote more testable code I started separate unrelated parts which reduced amount of spagetti code.

Another way to achieve cleaner code in my opinion would be design patters which lets you separate code logically(for example MVC), and make some things cleaner(I'm big fan of laravel's Facades for example).

porker23• 11 years ago

Thanks for your thoughtful comment Mike.

#1 is a good idea. Defining 'good code' is hard, but I know bad code when I read it, and certainly code with a philosophy that I disagree with!

I am definitely going to have to learn TDD after all the comments in this thread. Thanks for the recommendation of books, esp Feathers - a lot of my work is with legacy code.

Thanks again!

rbrogan• 11 years ago

Whatever it is, it is only so different from your past work. It may help if you sit down and try to put together an argument either that you can or you cannot do the work. I would be skeptical and rigorous about it and try to force it into defining exactly what it is that you cannot do. Chances are it is a matter of framing up the work, and in trying to prove the work is impossible, you will probably frame it up and make it be possible.

omginternets2• 11 years ago

If at all possible, take a vacation where you do anything but code. Pull the plug, if only for a week.

porker23• 11 years ago

I hate to say, I'm just back from a week's vacation. I struggled to switch off & went down with shingles (brought on I am convinced by the work stress)

coda_• 11 years ago

Give yourself some time to get back to healthy... any kind of health issue can cloud your judgement. I always try to refrain from making big decisions when I'm not feeling good. Shingles sucks, and the pain can linger... hope you're doing better now!

omginternets2• 11 years ago

Maybe a week isn't long enough? I know I barely start coming down after a week.

Edit: obligatory Vacation is like violence. If it isn't working, you aren't using enough

cableshaft• 11 years ago

It sounds to me like you've dug yourself into too much of a rut of how software should be made that you can't just dive into it anymore, and get bogged down in doing everything 'perfect' that you don't get anything done at all.

I tend to just dive in, make a bit of a mess, and refactor/clean things up before I check things in. I think it helps that I have tackled learning quite a few different languages over the years and have done some working for startups (which has no time to get things right, it's just got to get done before the whole ship sinks).

Just pick a little piece of what you want to work on, get that working, then add another bit, and another bit, until you have a full feature but still runs, then clean it up, check it in, and move on to the next tiny piece. Don't worry about documenting every little thing.

Learning a new language, especially a scripting language that can get things done in a few lines of code and people don't obsess over diagrams and graphs of the system could help. Python filled that niche for me.

Good luck!

zaphar2• 11 years ago

jaredhalpert says take a break and he's right. eigenrick says shake things up a bit and learn a new language he's also right. Sebastienconcpt says get a fresh pair of eyes, he is also right.

You just need a jolt. If it's feasible take vacation and build something fun in a brand new language that is really different from anything else you've ever used. Set goals like I'll work on this no longer than X hours a day where X is small. Don't set deadlines for completion. The goal here isn't a finished product it's to have fun tinkering with something that will break you out of your block. And pick a language with a welcoming but passionate community that can give your feedback on your code.

I do this all the time to keep myself fresh working on open source projects or playing with new things. It expands my toolbox and gives my brain a break from the constant legacy enterprise spaghetti-ware we so often work on.

And when it comes to your money earning work don't discount the Just Works outcome. When you are glueing together legacy systems sometimes there is no "beautiful and elegant" solution. Sometimes due to time and legacy constraints your only option is duct-tape and bailing wire. Having an outlet to write more elegant stuff in your spare time can be a valuable sanity saver.

Side Note: I wonder if there are any good places online to have virtual design reviews done? Somewhere where the high level goals and constraints are listed and people comment on the best way to achieve them? Language and platform agnostic preferrably although some suggestions will inevitably work better in some languages than others.

porker23• 11 years ago

And pick a language with a welcoming but passionate community that can give your feedback on your code.

Any recommendations? The PHP community (where my commercial work is done) is not that. And far too fad driven.

And when it comes to your money earning work don't discount the Just Works outcome.

You're not the first person to say that to me. I should start listening!

Having an outlet to write more elegant stuff in your spare time can be a valuable sanity saver.

Agree - if I despise what I write every day, it wears me down. I haven't worked on anything fun (or open source) for years, focusing on making money.

Thanks mate!

imakesnowflakes• 11 years ago

The PHP community (where my commercial work is done) is not that. And far too fad driven.

If you have the insight to recognize this, I think you are doing fine.

I would recommend Python as it's community seems to be a bit better. It sure is not as 'design pattern' infested. If you have a lot of time to spare and tenacious enough, you can also try Haskell. Although you should not feel disheartened about not understanding 90% of things being said in the irc chat room even after wrestling half year with the language. But I think it is worth giving a shot. I have already ported my side project which was in PHP to Haskell using a framework called Spock and PostgreSQL as backend..

Also, I think the php thing might be causing you this conflict. As I myself have gone through a similar stage. (8 years of professional PHP experience). When I started python 3 and after a while I unlearned certain things from PHP. Things became quite a lot simple.

For example, this is the same static site generator I wrote in PHP and Python3.

PHP - https://bitbucket.org/sras/vodka

Python - https://bitbucket.org/sras/feni

Writing a bit complex stuff using PHP is so damn tiring, but it only became clear that the problem was PHP after I started programming in Python and unlearned enough PHP stuff...

zaphar2• 11 years ago

It's hard to say without knowing what's already in your current toolbox. But a partial list might be:

* Erlang - Novel approaches to fault tolerant system design. Fail fast and recover somewhere else. Functional without the Type wizardry so you learn about immutability, pattern matching, and the typical functional language tools like folds, maps and so on.

* Clojure - A lisp that runs on the JVM. Learn about Code as data and how that empowers meta-programming. Also a functional language with interesting takes on how to share data in an immutable by default world. Lazy collections of a sort so you can learn about laziness.

* Haskell - Don't let the Category theory stuff scare you. Very interesting type system that can open your eyes to new ways to abstract things. Speaking of abstraction Haskell takes abstraction to a whole other level. Sometimes you can't see the program for all the abstraction going on. Militantly functional.

* Forth - Stack based programming, very small and compact. Radically different than anything else.

* Rust - Maybe you want to go a little bit closer to the metal for a change? Forces you to think about exactly how you share data in a multithreaded world. Targets the same world as C++ but perhaps with an easier to enter community.

* Python - Cleaner more elegant language in the same family as PHP with a very welcoming community. Batteries included so it's easier to get started than any of the others.

* Go - Concurrency done right. Lots of tooling builtin to the distribution toolchain.

eternalban• 11 years ago

Your #2 is drastic. Your #1, reasonable, and Martin Fowler is your friend [1][2].

[1] http://www.martinfowler.com/articles/enterprisePatterns.html

[2] http://www.amazon.co.uk/Enterprise-Integration-Patterns/dp/8...

[edit: beefed up]

weavie• 11 years ago

it's getting from what needs to happen to how to decompose into classes/methods/libraries

Perhaps the problem doesn't fit into an object oriented solution. You could try learning a functional language and see if you find it fitting better. I struggled for years with OO and was never happy with my solutions. Learning functional was my road to Damascus experience.

If you have to go OO, there have been plenty of people who have thought long and hard about how to make OO a suitable solution. Some classic books you may want to consider :

- Design patterns : elements of reusable object-oriented software (Gang of four) - there are people who swear by this and turn it into their hammer for which everything becomes a nail.

- Patterns of Enterprise Application Architecture (Martin Fowler)

- Refactoring: Improving the Design of Existing Code (Martin Fowler and Kent Beck)

- Growing Object-Oriented Software, Guided by Tests (Steve Freeman)

porker23• 11 years ago

Perhaps the problem doesn't fit into an object oriented solution.

Or if not the problem, then my brain.

You could try learning a functional language and see if you find it fitting better.

I come from a physics/science background; what I've seen of functional languages feel natural.

I bought my first book on design patterns 12 years ago (Head First Design Patterns). I worked my way through and it didn't make sense. I've dipped into PoEAA and Gang of Four since, with the same problem. The flow-chart-esque (UML) diagrams, the way messy life is meant to fit patterns... it didn't gel.

Time to try again, but I fancy learning a functional language first :)

nerdy4• 11 years ago

I reject the idea that this is an OO problem. These same problems would exist in the form of functional programming, except it would be "functions" instead of "methods"

mistermann2• 11 years ago

I struggled for years with OO and was never happy with my solutions. Learning functional was my road to Damascus experience.

Is there anything you could recommend reading that might show how a non-ridiculously-trivial example might be done in a functional language compared to an OO language? The "OO world" seems to me to becoming increasingly unintuitive so I'd be interested to look into something that makes me feel less crazy.

haxeny1• 11 years ago

I would add

- Object Oriented Software Construction (2nd. Bertrand Meyer)

I would recommend Eiffel if you want to elevate OO design. The issue with the GOF book that I have is that there is a slant towards composition as opposed to inheritance. When done right inheritance does result in compact code. Multiple inheritance cannot be avoided (Scala proves this by trying to bring in traits). Repeated inheritance actually helps reduce the code size as one could inherit the same method twice to implement a doubly linked list from a single linked list.

Note: In functional paradigm (Haskell) composition over inheritance makes sense, though in object oriented languages choosing an approach that has a compositional bias can impose some constraints that could be avoided otherwise.

porker23• 11 years ago

Object Oriented Software Construction (2nd. Bertrand Meyer)

That was the first programming book I ever bought! Sad to say it's sat unread on the shelf for 13 years...

themullet2• 11 years ago

Design Patterns! As you said your problem is "how to decompose into classes/methods/libraries"

Book wise, I learnt them from "Head first design patterns" which is useful and fun or for a quicker overview - http://www.java2s.com/Tutorials/Java/Java_Design_Patterns/in... .

Also remember if you have other people around you delegate / meet them / bounce ideas together.

porker23• 11 years ago

I bought my first "Head First Design Patterns" 12 years ago, worked through it, and it didn't make sense. I've dipped into PoEAA and Gang of Four since, with the same problem. The way messy life is meant to fit patterns... it didn't gel.

Any thoughts?

And thanks for your great reminder about meeting up to bounce ideas around. I'm going looking for them.

themullet2• 11 years ago

Difficult problem, I do know what you mean about fitting messy life into patterns, that said some can still be invaluable at times.

Something like a singleton would be handy to stop multiple creation of an object, for web design mvc just makes life a lot easier by decoupling different parts and in theory for your project something like an observer pattern could be created to notify other systems to a state change. http://www.java2s.com/Tutorials/Java/Java_Design_Patterns/01...

In the end there's many ways and approaches, maybe even brushing up some class design stuff in another language might help.

Good luck on your journey, sounds like you'll be fine.

chipsy• 11 years ago

Everyone's style is going to be differ on the high-level stuff - and you are surely well beyond the point where any random blog post blabbing about software architecture can guide you.

Keep walking back to the user and business requirements when you're in doubt. In senior technical positions, you aren't focused on immediate issues in the code, so much as on developing the right processes to attack those problems. Software isn't expected to last very long, and when it does it's both miraculous and ugly, as the tools and techniques become more and more arcane over time. If the solution you develop can satisfy all parties for an extended period, it is as correct as you can hope for - and part of that involves choosing a technical process that can be kept running in an environment of some ineptitude.

When you know the requirements, model the data first. Data comes first because data lasts longer than code. Flexible data lets you do more with less code. And if the data is very well defined you can usually use fewer language features, too - which is a bonus to maintenance later. Think of features in data terms first and lifecycles later.

The best-practices cruft you've learned over time - that's going to have to be sacrificed when you identify ways that your designs can be simpler. You may identify a pattern that is useful but not "with the grain" of the languages you're using. That doesn't mean it's wrong, just that you're breaking past some known boundaries of programming.

As you know, most projects get dirty in order to ship. Not doing that means doing basic research to find the ideal technique, whereas there's always room to tolerate a few more hacks, global flags, mystery-meat callbacks, etc. Your best defense is for the data to be so good that all the code can remain disposable and the system will still turn out more-or-less the same way if rewritten from scratch, with a different-but-equally-awful set of hacks and kludges.

Under the pressure of strong data, code that sticks around after several cycles of disposal becomes the library code by default - and internal APIs will become strong enough to prevent a "lava layer" from taking root, because they'll consistently solve the immediate problem faster than any wrapper layer. The maintainers will feel like geniuses because they'll keep finding clever ways to use the existing data better, instead of mashing out another travesty of reflection and classes-on-top-of-classes.

If data isn't sufficient, you may also have to define a protocol. Protocols, if they work correctly, compact the state management into a small part of the app, and leave you with more data.

porker23• 11 years ago

When you know the requirements, model the data first. Data comes first because data lasts longer than code. Flexible data lets you do more with less code. And if the data is very well defined you can usually use fewer language features, too - which is a bonus to maintenance later. Think of features in data terms first and lifecycles later.

Boy am I SO GLAD to see someone say that! I think of my approach as "Data-Driven Development", rooted in my days of scientific programming. The data is what matters. On my largest project to date (8 months, entire business system) I spent 25% of the time just on the data store schema. Needless to say it worked & evolved without issue over 5 years.

Yet (at least in the PHP community) the fad is currently Domain-Driven Development, BDD, and architectures ("another travesty of reflection and classes-on-top-of-classes").

The maintainers will feel like geniuses because they'll keep finding clever ways to use the existing data better

Oh yes, that's how my oldest systems are. Spaghetti code in parts, but the data keeps on giving.

Thank you for taking the time to write this, you've ignited a small flame of enthusiasm in me and reminded me why I enjoyed doing what I do!

ihsw• 11 years ago

Quick question -- do you use unit tests?

porker23• 11 years ago

No. I missed the boat, then I felt functional testing was more important. Ian Cooper sums up my thoughts better than anyone else: https://vimeo.com/68375232

nerdy4• 11 years ago

If the code is tested then refactoring the ideas contained within is the way to go. Do the smallest refactorings possible. Even well-respected programmers who've authored tons of books and given talks do not sit down and 1-shot a program. Elegant programs are the result of refining and designing.

The earlier in the development process you get to the refactoring and isolating minimal testable components, the less of a tangle you have between components to refactor; it's a lot like the wire tangles we're all so familiar with). This not only means less refactoring, but it also means that the existing code is easier to understand.

The tests are code. Treat the tests just like the rest of the code. I prefer test-first but if you want to pull your cart in front of your horse then go for it.

If the code is not tested I'd recommend testing it. There are really only 3 options to go from untested code base to tested code base: - Try to put tests under existing code (often inefficient) - Refactor the system incrementally as time goes on and features are added and bugs are fixed (slow process but over time test coverage grows) - Throw it out and start over (ouch but... greenfield; if you do this you should _NOT_ add any new features to the live/production system or you'll end up in a rat race between the two forever)

Also if you're able to take any time away (preferably measured in days) the fresh perspective might help too. Sometimes that little bit of total separation is enough to come back and read a method name and wonder why the hell it's even in that particular class. This scenario could be an early warning for burnout too, take care of yourself!

porker23• 11 years ago

Testing. I have never done TDD; I put it off in the beginning as an extra cost to clients (or time lost for me), and then when it became trendy I (as ever) disliked the trend :) Ian Cooper sums up my thoughts better than anyone else: https://vimeo.com/68375232

I'm doing a rewrite of a small project and that seems ideal to give it a go. To me functional testing is much more appealing, testing a simulated user's experience through the website, yet TDD seems more popular - as easier?

This scenario could be an early warning for burnout too, take care of yourself!

I've been in a burnout cycle for the last few years - a holiday in July was the first time I switched off in 15 years, then a health scare last week and a friend in palliative care have pushed me to reconsider. The lure of money isn't as strong as it once was.

jerf3• 11 years ago

I think TDD is a great exercise to do a couple of times, and a great way for a new developer to grow some good habits very quickly. I'm much, much less convinced of its utility for people with experience.

However, you really do need some automated testing. One of the reasons I love automated testing is that it permits me to make (nearly) monotonic forward progress on my tasks. If part of your burnout here is that you are just [expletive] tired of making changes in one bit of the code just for it to create 6 more bugs in the rest of it, automated testing is the core solution to that. I used to get that type of burnout... I don't anymore, except when I'm stuck crawling around somebody else's code base where I can't add testing for some reason.

Also, rather than posting another comment, rather than a new "field" you may just need a new job. It isn't all CRUD coding. Mind you, there isn't really much out there that is fascinating cutting-edge research every day, either; even that involves a lot of grunt work. But it isn't all CRUD work out there, either.

nerdy4• 11 years ago

First, that alleged "extra cost" is the #1 myth. It might be true of test-last (code then test, anti-TDD; whatever it's called)-- I am not sure because I've never tested that way and that process is transparently ineffective if you take a very structured approach to the process (more later). I've been at a place where I also believed the same thing but since then I've built larger and larger systems and felt these pains. While TDD will slow you down initially as you learn, once you're comfortable with your testing framework, you'll notice you don't really go slower because you have near zero debug time, feel absolutely empowered to refactor and clean code, clarify intentions and just generally make everything crystal clear without breaking anything. You'll also notice that in most cases, as projects grow large and cumbersome, you keep that same steady methodical pace

Second, TDD is not E2E testing. There are tools for E2E but when we're talking about refactoring or structural organization it isn't so much a matter of UIs as it is a matter of what happens within the logical decisions of the system, which hopefully is not contained within the UI for non-UI logic. The description of your problem does not sound like you have an issue with the website's frontend but with the complexity of its brains behind the scene.

So you'll want some tests to cover the application code units individually (unit tests), and something which makes sure you can glue all of those pieces together and they still work as expected (integration tests). An integration test is not end-to-end, it's simply multiple components. Each of those components should already have passing tests, so you should have very high confidence that they work in isolation. The integration test simply makes sure they're connected the right way. I prefer as few integration tests as possible (but at least 1). If you have lots of integration tests you're probably performing at least some testing which should be covered by unit tests within those integration tests. Integration tests should just make sure the components are compatible and connected, not really test their internal behaviors.

I personally do the highest level of finalized testing (V&V) manually because I've proven the components work and are wired correctly, so the UI is usually just a dumb wrapper dumping out HTML. I open a browser and click around, make sure it works, think about how I can break it, beat it up for a few minutes and appreciate my work. You could automate this part of the process but I haven't found sufficient reason to (yet).

TDD is simply writing an assertion and then satisfying it with code. At the end, you've independently computed and verified that some tiny granule of code has some certain behavior.

When I started I'd think to myself "ok, I'm going to write the sorting code" and then I'd run off writing tests to make sure some simple structure was sorted after being passed through the method. That's way too fast. The best way I've heard the proper approach described is when Bob Martin says "test the most degenerate case first".

So my sorting would start off with sorting null. What should happen if that unit gets null instead of the expected array? What does it return? Does it throw an Exception?

Refactor time. Why does the code suck? How would it be easier to understand? Any methods too long? Any repetition? Don't add tests, don't change behavior. Only change the way the result is reached.

Test subsequent cases like null/false/zero/undefined. Refactor after each, if appropriate. Don't skip refactoring! You don't always have to do it but you should consider whether or not it is currently appropriate.

After those cases you can test the simplest things; an empty array. It should probably return an empty array. Then refactor. Then a single-element array, refactor, then two... then three, then the three in various orders.

At a certain point you can't write a test that fails. You're done with this unit of code. Everything provably works. I realize all of this seems like an overengineering process that's too low-level to be remotely sane. Seriously, I understand why you might think that. The thing is, all code is written one line at a time. Every second spent on the tests is paid for by the absence of debugging, dumping variables to screen, viewing source, etc. I'd argue that time spent debugging is time least effectively spent, particularly when in vein.

BTW money isn't my primary motivator either, I just enjoy programming. Make the best of it; do what you enjoy and get paid for doing it but don't look at the money as a carrot. If you can't stay motivated then maybe it isn't a matter of career but a matter of the project or industry?

lukebuehler• 11 years ago

I recently hit a wall like this as well after designing many apps and enterprise systems.

It was a problem that seemed simple at first, but I just kept banging my head against it. I did exactly what some people described here, switched to a functional language, but that didn't help. I started doubting myself.

I'm though it now, and I like the solution, although it could be better. Here are some points that I've picked up along the way:

- Remind yourself that you are not only generating value when you actually write code. When you take a walk and think about the problem, draw stuff on paper and so on, this is still work. You are still doing exactly your job.

- This gets hard when coming up with a design takes longer than usual. It's likely that you have run into a complex problem without knowing it. Endurance & patience is key here.

- Attack from different angles, don't consider the time wasted if you try something and then end up throwing it away, you will have learned something, even if it just eliminating one possible design option.

After you are through it, you might end up cherishing the experience.

ArekDymalski3• 11 years ago

How exactly does the "mental block" look like? Does it mean that you see several possible approaches and struggle to decide which one to use? Do you feel the pressure of the "into something maintainable that will run for 5+ years" requirement?

If the answer to 2nd and/or 3rd question is "yes", your description sounds like a specific phase in the process of your professional development.

A lot of people grow with their competences (be it knowledge or skills) to the moment when they start to realize the limitations of their competences. The moment when they see a bigger picture of their work, the broader consequences of a possible error, they feel bigger responsibility etc. It's a moment when doubts start to creep in and self-esteem plummets. The paradox of this phase is that objectively such people are very competent and capable to solve the problems they face. But they become paralyzed by the pressure and self-doubt.

If this reminds you your situation (at least partially) then the solution is called coaching - a method created to help competent people who lost their faith in own competences, overcome the difficulties and grow further.

Other (often proposed, even here) solutions like switching a job, going to some seminar, being more controlled by your boss or taking a vacation do not solve this specific problem, and quite often lead to bigger decrease in self-esteem.

Hope it was helpful and coherent - it's quite complicated subject to present in a short comment.

porker23• 11 years ago

How exactly does the "mental block" look like? Does it mean that you see several possible approaches and struggle to decide which one to use?

Sometimes yes (paralysis by choice), but in this case no. In this case defeat and "I can't do it" are my initial response, and I shy away from programming.

A lot of people grow with their competences (be it knowledge or skills) to the moment when they start to realize the limitations of their competences. The moment when they see a bigger picture of their work, the broader consequences of a possible error, they feel bigger responsibility etc. It's a moment when doubts start to creep in and self-esteem plummets. The paradox of this phase is that objectively such people are very competent and capable to solve the problems they face. But they become paralyzed by the pressure and self-doubt.

That fits. I know I know nothing. Then I wonder what it means to know. What is meaning? You get the picture - it's a destructive deconstructive spiral brought around by doubt. I inhabit "Imposter Syndrome", still trying to find a way out.

If this reminds you your situation (at least partially) then the solution is called coaching - a method created to help competent people who lost their faith in own competences, overcome the difficulties and grow further.

Do you have any pointers for finding a competent coach please, vs someone who's overconfident in their own abilities?

Thanks for taking the time to comment, you've been really helpful.

SignMeTheHELLUp• 11 years ago

Do you have any pointers for finding a competent coach please

I second this request. Finding competent, egoless developers who are willing to share their knowledge is extremely hard.

ArekDymalski3• 11 years ago

defeat and "I can't do it" are my initial response, and I shy away from programming.

Was it always the case? Also: how do you finish your projects then?

a competent coach please, vs someone who's overconfident in their own abilities?

I'd say that anyone demonstrating a strong confidence either in the results (ie. guaranteeing to help you) or own skills (ie. talking a lot about self) has a giant red flag over his/her head. Good coaches have an interesting mixture of modesty (no bragging), genuine focus on other people (more questions than statements) and assertiveness - healthy interpersonal strength when it comes to the coaching process, tools etc. Ideal coach will:

1. Be very clear upfront about the distribution of responsibility between him/her and you - it should be ~50-50. That means that being a coaching client is a quite a challenge and it isn't always an easy experience. Anyone painting a sunny, broad staircase up for you is a red flag :)

2. Recognize you as an individual - without jumping to conclusions too early.

3. Describe you the idea of coaching and how it is different from advice, counseling, therapy and other forms of help which are sometimes confused with coaching. Anything what smells new-ageish or like proven receipt for success = mega red flag :)

4. Describe you the process you will go through. Usually it's something like: analysing your needs->setting development goal(s)->performing some action->discussing the action+feedback->setting next goal and this repeats until your main goal is reached (or the agreed upfront number of sessions is finished). It's very important that coaching is like a good project: it has clear, measured goal and is meant to end at some point. If the end is lost in a distant fog - yes, it deserves red flag.

5. Be very precise and metrics-oriented - there's no place for unclear mumbo-jumbo in coaching. Unless someone likes red flags of course :)

So, these are the essential characteristics. Also it is very important that you should feel comfortably in contact with that person - imagine openly discussing your struggles, doubts as well as receiving feedback form him/her. Regarding formal requirements: respected certificate (ICC/ICF/IIC/IAC) is a proof that someone was trained enough to know the process and toolset. But certificate isn't necessary because the process&tools are actually quite easy to learn - everything depends on mindset/attitude and that's something what could be shaped during the training but as usual - no guaranties.

Feel free to ask if something requires more info.

jacooqs• 11 years ago

5+ years? That's crazy for web applications. I cant imagine any code lasting more than two years nowadays

blt2• 11 years ago

I'm going to take a wild guess and say you don't really care that much about the purpose of your software. You're not a game developer who's been an avid gamer since age 4, or a silicon valley entrepreneur who truly believes their web app is going to change the world. You've identified a niche that's liable to make some money, and are diligently plugging away at an implementation.

Your lack of inherent motivation from the product domain means you don't get any satisfaction from just getting the requirements met, so you hunger for the satisfaction of making a beautiful architecture. But all architectures have flaws. As a competent engineer, you notice the flaws in whatever you design, so your only source of satisfaction is compromised.

I would try to find work on a problem that excites you. When you get that giddy rush just from seeing your program give a correct answer. I think most software developers have felt this way about some project during their learning process. Look back and try to remember where it happened for you.

You're in trouble if software craftsmanship is your only source of motivation. It's a great motivator, but it is not enough by itself to carry you through a big project.

Now that you're in this project, it may be impossible to quit. I would find a side project to work on that provides more satisfaction. That should make it easier to plug-n-chug away at your current project without becoming depressed at the inevitable small failures in elegance and design.

I've never understood how developers can be happy writing CRUD apps or wiring together a bunch of pieces made by someone else. You gotta search for meaning. Even if you take a pay cut, it's worth it to find work that you care about.

porker23• 11 years ago

I'm going to take a wild guess and say you don't really care that much about the purpose of your software. You're not a game developer who's been an avid games since age 4, or a silicon valley entrepreneur who truly believes their web app is going to change the world. You've identified a niche that's liable to make some money, and are diligently plugging away at an implementation.

Spot on. I walked into this career by accident, the money was good and I stayed. No, I don't know what I actually want to do - something I've never known :)

You're in trouble if software craftsmanship is your only source of motivation.

Spot on again. Pride in what I do is my only motivation at this point. I don't get to see happy users, find out how I've improved lives, get user feedback and make things better for someone.

Now that you're in this project, it may be impossible to quit.

It was impossible to turn down, as they're a good client and provide somewhat interesting work (always unusual tasks). Otherwise I wouldn't have touched it.

I've never understood how developers can be happy writing CRUD apps or pushing ads. You gotta search for meaning. Even if you take a pay cut, it's worth it to find work that you care about.

CRUD is the most soul-destroying code to write as doing it with good craftsmanship is impossible (there I said it). Concerns overlap between the front and backend. I do share your sentiment, but pay cuts are hard (I suspect I undercharge at the moment, making them harder still).

Thanks for commenting!

richardboegli• 11 years ago

Try your hand at Game programming, its great to mix things up and the communities are great:

Unreal Engine 4: https://www.unrealengine.com

Maybe a mod for Unreal Tournament or Ark Survival: https://www.unrealtournament.com/blog/

http://www.polygon.com/2015/8/14/9153379/ark-survival-evolve...

Unity: http://unity3d.com/

Construct2: https://www.scirra.com/

ktt3ja• 11 years ago

I'm not the OP, but what you say resonates with me as well.

May I ask what domain or field you work on?

blt2• 11 years ago

I used to work on desktop apps doing interesting stuff with large and/or soft-realtime data. Now I am working on a Ph.D. in CS. I decided I would not be happy in the long term unless I'm doing research-level work. Even though I was developing products much more interesting than the typical CRUD app, I was still spending lots of time on GUIs, undo systems, file IO, etc. I want to avoid that kind of work as much as possible in the future.

abathur• 11 years ago

When you explain your problem, I feel like we might share a trait, which I could pithily describe as difficulty separating/ranking/comparing/organizing/categorizing things. So, I find it very taxing to make the kinds of decisions you describe. I usually get there, but it's heavy lifting for me.

I don't think there's a silver bullet for you; you probably aren't incapable, but you may need to suffer through sub-optimal organization to hone this skill. What I will say is that, if you haven't worked on an existing project with highly complex organization (ideally one that has problems), doing so might help you. While I envision working directly on the project when I say this, there are cases where working with complex dependencies you need to get custom results out of can also be useful.

The point is that you want to be working with complex software (not simple interfaces) that requires a strong understanding of its internals to accomplish what you need. You'll know you're working on something on roughly the right magnitude of complexity when you regularly realize that your understanding of how the project or dependencies work is wrong, and your solution doesn't account for some hitch in how they work. The meta-point is that you're working with sub-optimal decisions other people made, and you either just have to deal with them and simmer to yourself about how _you_ would reorganize this mess if it was within scope, or you have to start refactoring their code (or building your own interface with it) that solves the problems with their design. The meta-meta point is that we're better at seeing problems in work other people have done than our own, but with effort we can train our eye on other people's work and (eventually) learn to focus it on our own.

At it's core, I think this is really about the cognitive overhead associated with working on and understanding a given implementation, so working on things that are cognitively overwhelming provides you an opportunity to learn about how to manage cognitive overhead. The result is hopefully that you learn to wrangle that complexity into an understanding that helps you do what you need without doing surgery, you build an interface that reduces the cognitive overhead enough that you can do what you need, or you refactor to accomplish the same.

manyxcxi• 11 years ago

Learning something is always helpful in some ways but it may not be exactly what you need. It sounds like you may be at a point in your career where you may be suffering some imposter syndrome when it comes to making the decisions to move the PoC code to something more final. I hit a stage as well, years ago where I was paralyzed by the 40 different ways to architect a Java web app and all the arcane crap that came along with it. In general patterns only serve to better understand and maintain your code. That should be your primary goal during this time, is this readable and maintainable? After that, think about ways you can make it more slick. Eventually, you get over the hump and those patterns become part of your writing style and you don't even think about it anymore, until you hear about some new cool pattern (MVC became THE THING during my professional time) and you figure out how to incorporate it. After this similar block I had in my career I never really had another, until...

I had a different type of block earlier this year. For the first time in my career (I'm 32) I left my employer to work 100% for myself. I spent the first six weeks so twisted up I'd go days without a line of code written- luckily there was enough in the way of infrastructure and Ops tasks to give me something to do. I think the stress of it all just kept me from being able to focus like I need to in order to develop.

I started taking breakfast with the wife and kids, going to the gym every single morning, walking around for at least 15 minutes at lunch, being home by 4:30 (instead of 6:30/7). I basically focused on everything else and after a week or two it just started coming back and I feel like I'm faster and less stressed than I've ever been. Even if I'm not I'm happier and my family is significantly better than before, so I'd do it again in a heartbeat.

porker23• 11 years ago

It sounds like you may be at a point in your career where you may be suffering some imposter syndrome when it comes to making the decisions to move the PoC code to something more final.

Definitely! I'm the imposter that imposter syndrome was defined around ;)

That should be your primary goal during this time, is this readable and maintainable?

Oh, that's an interesting twist. Move the focus away from craftmanship of the present-day to future-proofing. It may be more ugly but if I can fix/extend it in 12 months without missing a beat then my job is better done.

I spent the first six weeks so twisted up I'd go days without a line of code written- luckily there was enough in the way of infrastructure and Ops tasks to give me something to do. I think the stress of it all just kept me from being able to focus like I need to in order to develop.

I'm 31 and that's how I've spent a lot of time recently. Especially when not working flat-out on a project to meet a deadline. Give me one of those and I'm happy: head down, pounding it out.

eyeinthepyramid• 11 years ago

For me, a lot of this stuff only comes through trial and error. It's not ideal, but I think over time, both the trials and errors improve in quality.

Basically as I work in the code, I see little things that could be easier or make the code clearer. Then sometimes I get ideas for big changes. Sometimes the changes make things worse, but that's why we have revision control right?

I think the same goes for drawing. Just get something on paper and look for small changes. As you make changes, eventually you might see a larger change.

palidanx• 11 years ago

Do you have any training in UML? I feel that learning basic things like domain diagrams, sequence diagrams, and state diagrams help decompose complex systems into maintainable chunks.

joeld42• 11 years ago

You're thinking about it wrong. Don't think of it as "prototyping" or "just works". If it just works, you're done, there's no need to rewrite it to make it enterprisey.

What about it needs fixing? If it works for 100 items but not a million then fix that. If it works but is difficult for others to change the code, rearrange things until it's clearer. If it works by using some weird 3rd party dependency that wouldn't be easy to deploy and update, just remove that dependency, etc. If it works but you don't have an easy way to build or deploy it, then work on the build system. If it's buggy, fix the bugs.

Don't make things more abstract just for the sake of abstraction. Don't add message passing or enterprise logging just because they're enterprisey. Ask what problem you are trying to solve by adding these.

quantum_state• 11 years ago

It seems to me it may help, if you have not done so, to refresh yourself with some basic abstract math .. starting with set theory, topology, abstract algebra such as group theory and group representation .. it would illustrate how to walk up and down on the abstraction ladder to get to the essence of the problems at hand .. once you have got the essence .. you would be liberated and can spin it any ways you like ..

davelnewton• 11 years ago

I'm confused why you list only two options.

MalcolmDiggs2• 11 years ago

Is it possible that you're suffering from "Analysis Paralysis"? (https://en.wikipedia.org/wiki/Analysis_paralysis)

If you have 15 years of experience, I'm fairly confident that you know how to solve these problems. In fact you probably know 10 different ways to solve these problems. Maybe the overabundance of choice is becoming the bottleneck?

If so, perhaps there's a way you can simply force yourself to commit to a methodology/problem-solving-method before you start tackling the actual problems. Or, you could try breaking the development cycle into smaller, more digestable, iterative chunks (so that you only have to think about the next small step instead of the whole path forward).

porker23• 11 years ago

If so, perhaps there's a way you can simply force yourself to commit to a methodology/problem-solving-method before you start tackling the actual problems.

I really like that idea. So for a given project where there are 5 ways I can say "I'm going to solve it using approach 4. It may turn out to be better or worse than the others but that's not what matters; using this method is what I'm doing" and have that as my goal, not finding the optimum method amongst the 5 possible ways.

MalcolmDiggs2• 11 years ago

Exactly. It's almost like recreating the pseudo-efficiency that newer developers seem to possess (I only know one way to solve this, so I'm going to do it that way, and do it fast). In your case, the loss of choice is artificial, but it reaches the same result (getting it done).

cturner• 11 years ago

About 2004, I reached a stall state. I did some serious wilderness time and it took about six years to settle down into a new zone.

This was the trigger for me: (1) I had a development career where I tried to build every application centred around databases; (2) I now had enough experience under my belt to join the dots on where this ended up, and to see that this was futile. But I wasn't able to see past my everything-is-a-hammer.

There is a technique I took a lot of value from in moving on, and I still use it most days. It is this: think hard, ten times, about a problem that is tickling you. I went through a notation phase, but now I just type out my stream of conscious into a text document.

In the past, I would have spent a year building the first thing that came into my house. Now I can consider a large number of ideas, and have context for judging them to be good or bad ideas. The first thing I invest development energy in is significantly better than it would otherwise have been.

It's cool too in that you don't need to be at your desk to do it. You can be in a restaurant, or sitting on the beach, or on the bus, or on an exercise bike. My favourite spot at the moment is lying on a couch. The good thing about doing it away from your desk is it takes away the pressure to rush through the uncertainty.

"choreographing 3 enterprise systems to work together"

How do you think about message-passing between them? Why do you do it that way? Think ten times about different approaches you could use. You could use the exercise to force you to question your passive assumptions. You may find that you are scared by uncertainty. Your subsconscious could choose to fix on something familiar, than to admit that it does not know. All the more reason to do this away from your desk. Take a walk.

Feel free to ping me via my account email if you want to discuss your situation in more detail.

nostrademons• 11 years ago

Don't decompose up-front. Instead, take your prototype and and start adding the polish necessary to turn it into a finished product. When a subsystem starts getting so complicated that can't understand it when you're changing it, look at that subsystem in isolation and figure out how you might refactor it to eliminate code duplication, create simple interfaces, etc. Eventually you end up with clean system boundaries separated by message passing, but each individual step doesn't require that you hold the whole system in your head.

Ask HN: I have a mental block designing software. Time to quit?

Related Stories