"How hard can it be to implement?"

http://answers.37signals.com/basecamp/52-how-can-i-move-or-copy-to-do-lists-between-projects#reply_6986

by adamhowell • 16 years ago

134 97 16 years ago

answers.37signals.com

Is there really no simpler way to solve this problem?

"Moving a message needs to move all of the message's comments, and all of the comments' files, and all of the comments' files' versions."

Why can't you just change some top level reference in the database? I'm imagining a Projects table and a TodoLists table. Each TodoList has something like a projectID foreign key right? Why can't you just change that and automatically have all of the messages, etc, come along with the TodoList?

"But I hope you will see that sometimes even the simplest feature can be much more complicated than it looks from the outside."

I agree there, just wondering why the simple solution doesn't work in this case.

SMrF• 16 years ago

From the response;

"We can't use database transactions because performing a big move would slow Basecamp down for everyone. So we have to log the process of each step of the move, and make it so any failure in the move can be rolled back gracefully. That means a move is actually a series of copies and deletions instead of just changing a field for each moved item"

xentronium• 16 years ago

What's the point of having ACID when you don't use its advantages, one might wonder.

jules5• 16 years ago

Why is a transaction necessary? Just change the one field project_id in the table todolists to point to the new project. Because the messages point at the todolist_id instead of at the project_id you don't need to modify anything else.

gaius5• 16 years ago

How amusing. Absolutely not an issue in any grown-up database that implements row versioning/MVCC a la Postgres or Oracle. The sad thing is he (and most MySQL users) probably believe this is inherent to all RDBMSs.

russell• 16 years ago

It can be if your data structure is a simple tree. But usually it's a complicated, maybe inconsistent, multi-rooted graph. It got that way because of incremental implementation. As the article illustrated, you then have to tease out the pieces.

I once inherited a large enterprise system that had been designed from the ground up to allow that kind of flexibility. The problem was that there were so many levels of abstraction and indirection in the database and the ORM model that performance was abysmal.

weaksauce• 16 years ago

Might be trickier to do this if the foreign key is across a shard or if the db record for the comment wasn't structured like this. One situation could be because they store the threaded comments with the thread key in each record and also the parent comment key. This would be useful to pull down all the comments associated with a thread in one db query but also allow you to have the nested comments.

smiler• 16 years ago

37signals don't use sharding. They use one database server with enough RAM to load the db into memory

toddheasley• 16 years ago

I also agree with the sentiment that nothing is ever as simple as it seems. But, yeah, isn't this what relational databases are for?

grandpa• 16 years ago

That might - might - solve the first 4 points, but certainly nothing after that.

amix3• 16 years ago

The strategy you usually take with scaling a relational database is sharding, partitioning and de-normalization. And given that Basecamp has millions of users I don't imagine their database structure to be pretty or normalized.

This said, if they have their database sharded by project_id then it would not be that big an issue, but it can be that their database structure is very complex or messed up...

jemfinch11• 16 years ago

> And given that Basecamp has millions of users I don't imagine their database structure to be pretty or normalized.

Millions of users is not really that many. It's certainly within the realm of what can be reasonably vertically scaled.

dasil0038• 16 years ago

You're falling into the same trap all programmers do when they perpetually underestimate the time to accomplish any task. The main difficulty in estimating time is that you don't know what you don't know, so you have to actually get into the nitty gritty of implementing it, and if you're smart enough you'll hopefully catch all the requirements before you actually launch it.

I'm going to take Sam's word on this that there is not a simpler way, because he's the one working on it, and I trust the competence of the 37s dev team. That's not to say that some armchair analysis here on HN might not have valuable insight, but without actually seeing the code, and in light of the full list of points Sam mentioned, I doubt very much that there is a much simpler solution of any form.

jemfinch11• 16 years ago

> I trust the competence of the 37s dev team.

I don't, in large part because this shouldn't be a complicated problem to solve.

dugmartin• 16 years ago

Yes, to paraphrase Mr. Rumsfeld, when estimating large system changes you can see the known knowns and have an idea of the known unknowns but the unknown knowns and unknown unknowns are hiding like the underwater part of an iceberg. Bad estimates are usually caused by not taking into account the unknowns.

How is that for a mixed metaphor?

DougWebb3• 16 years ago

> The main difficulty in estimating time is that you don't know what you don't know, so you have to actually get into the nitty gritty of implementing it, and if you're smart enough you'll hopefully catch all the requirements before you actually launch it.

That's certainly true for a low-level, high-precision estimate, which you can only make when you've got detailed requirements. Prior to that you still need to be able to deliver an estimate, and the best approach to take is to provide an estimate with Quantified Uncertainty. The phrase is important, because the estimate is for managers and managers are all about quantifying risks, and uncertainty in a development estimate is definitely a risk.

When you say "I don't know how long this will take, because I don't know how much I don't know about the solution" you're giving an estimate with uncertainty, but you're not quantifying the uncertainty. It's got no bounds, it's limitless. Managers can't make any decisions with that, so it's useless information. But if you say "This will take 4 to 8 effort weeks, and I'll be able to give a more precise estimate after 2 effort weeks of investigation" that gives your manager something to work with. You're still uncertain, but you've quantified the bounds on the uncertainty, and decisions can be made. For example, solving the problem might not be worthwhile unless it happens within the next two weeks (eg: to meet a release deadline) so the manager can just delay the work until later.

The trick with this approach is setting the bounds. The man who taught me this style of estimating called it a Surprise Range. The lower bound should be an estimate such that you'd be surprised if it took less effort than that, and the upper bound should be an estimate such that you'd be surprised if it took more effort than that. Given whatever information you have about the requirements, you just keep pushing your bounds until you can honestly say "I really don't think it would take less/more effort than that."

As you get more experienced, you'll naturally start accounting for the unknown unknowns, because your past experiences with them will be nagging in the back of your mind while you're trying to come up with an estimate for some new task. This is especially true if you're estimating work on a project you've worked on before; you'll have a feel for where the trouble spots are and whether or not the task in question is going to get into those spots.

The other trick with this approach is the "more precise estimate after some investigation" bit of the statement. You need to provide some estimate now, buy some investigation time, and commit to another estimate later. Most of the time the second estimate will fall somewhere within the bounds of the first one, but not always, and your manager should be aware of that possibility. But with experience, you'll usually wind up within the original bounds and narrower than the original bounds too, which is quantifiably less uncertainty than before so you can show progress.

vog5• 16 years ago

Also, even if there was a good reason for that strange database scheme, they could have simply specified "ON UPDATE CASCADE" on their foreign references, and the database would have taken care of the nasty details. That's what databases are good at. (Or did their ORM prevent them from doing so?)

So this is more an example of "How to make simple tasks difficult" rather than "Even simple tasks have their pitfalls".

ams6110• 16 years ago

Maybe we're seeing a downside of the YAGNI, shoot from the hip, do the simplest thing design philosophy.

Vitaly• 16 years ago

agree. most of the described changes seem to follow from a bad db design.

lets see:

- Moving a message needs to move all of the message's comments, and all of the comments' files, and all of the comments' files' versions.

Not really. comment should only by 'tied' to message, i.e. there should be comments.messages_id command, so moving massage somewhere else shouldn't require moving comments (unless you fucked up the db schema to begin with)

- Moving any file needs to move its thumbnail, too, if it's an image.

No need to move the file to begin with, see above.

- Moving a milestone needs to move any associated to-do list and messages. And of course those to-do lists can have to-do items with comments, and attached files, and multiple versions of those files.

This is probably the one that indeed needs special treatment even in reasonable designed db. There are ways to skip this if you anticipate the need at the very beginning but it will require some convoluted db hacks to allow O(1) simple move (i.e. make everything including project just an item and just have parent_id in each item, then have messages to be children of the milestone. but this will have other problems, performance and complexity etc. so I wouldn't go this way)

- Moving a to-do list whose to-do items have associated time tracking entries needs to move those time entries to the destination project too.

Not really. time tracking entries should not have project_id in it. just todo_id

- Moving a message or file needs to re-create its category in the destination project if it doesn't already exist.

Indeed.

- If a moved file is backed up on S3 we need to rename it there. If it's not, we need to make sure it doesn't get backed up with its old filename.

why the file path on S3 has project id in it. attachments.id would suffice to uniquely identify stuff. its not like you need to invent multilevel directories on S3 like you do on a file system.

- When someone posts a message to the wrong project, then moves it to the right place, we need to make sure that everyone who received an email notification from the original message can still reply to the message via email.

If the email link just has the message_id and not the project_id it solves itself. You might need to play with some ACL though.

- Similarly we need to make sure that when you follow a URL in an email notification for a moved message, comment or files, you are redirected to its new location.

Dont' include project_id in the url to begin with. Use 'flat' routes.

- Since moving one milestone could potentially result in hundreds of database operations, we need to perform the move asynchronously. This means storing information about the move, pushing it into a queue, and processing it with a pool of background workers.

Not really. Even moving a milestone should only require 2 transactions. move the milestone, and move all milestone's children. Thats might be a non-trivial UPDATE sql operation but its not hundreds of queries.

- We also have to build a new UI for displaying the progress of a move. It needs to poll the Basecamp servers periodically in the background to check to see if the move is done yet, and take you to the right place afterwards.

Probably right.

- We can't use database transactions because performing a big move would slow Basecamp down for everyone. So we have to log the process of each step of the move, and make it so any failure in the move can be rolled back gracefully. That means a move is actually a series of copies and deletions instead of just changing a field for each moved item.

If you eliminate most of the complications above then you can use transactions.

Groxx2• 16 years ago

Sounds like they're in need of some hefty normalization. This is the sort of thing that should be handled by changing a single foreign key on each todo / on a single todo list entry; everything else should be pointing to each individual entity, and need no modification.

Unless they're denormalized for optimization purposes? Threaded comments can result in some massively deep queries.

richardhenry• 16 years ago

> Unless they're denormalized for optimization purposes?

Bingo?

richcollins2• 16 years ago

This would be easy in a graph db. Just move the node and its children come along for the ride.

josephholsten2• 16 years ago

Not exactly. If you look a bit closer, you'll see that this isn't just a strict child, otherwise they might have managed by swapping a foreign key or something. In a graph db, you'd still have to reconnect dangling edges that no longer make sense, like for comments and users. Also, the transaction rollback issue. Which is to say: tanstaafl.

richcollins2• 16 years ago

The comments and user nodes are moved with the parent. It's like moving a folder with subfolders in the filesystem.

nawroth• 16 years ago

Doesn't need to be a strict child when using a graph (not to confuse with hierarchical) database. Not sure what you mean by dangling edges, at least the Neo4j graphdb guarantees that edges are always connected at both ends. It's also ACID compliant so rollbacks are supported. Sure, tanstaafl, but when dealing with data in a graph structure there's much better tools than RDBMS nowadays!

tomafro2• 16 years ago

The comments here remind me of an old Irish joke where a hopelessly lost tourist asks an old man by the side of the road "Can you tell me how to get to Dublin?". After a few minutes thinking, the man replies "Well, you don't want to start from here".

I doubt 37signals wanted to be in a place where an apparently simple change would involve so much work, but that's where they found themselves. They did what they had to do. There's no point snarking about their starting place without knowing how and why they got there.

bambax• 16 years ago

Is this an old Irish joke? I thought I knew for a fact that it was French!

- "Pouvez-vous me dire comment aller à Paris"? - "Ben si j'étais vous je ne partirais pas d'ici!"

Very French in its non-helpful but matter-of-fact way... but maybe the Irish are the same!

tomafro2• 16 years ago

Your French version feels authentic to me, but then so does the Irish one. I doubt one can say where jokes like these originate. All cultures and languages probably have versions of them.

mkramlich3• 16 years ago

Great post but note that the total complexity/LOE of adding that feature would be significantly reduced if (1) it was delivered earlier in the app's lifecycle (less users, simpler code, lower expectations, less/no customers, etc), and/or (2) quality/UX standards were relaxed somewhat.

I agree that even a seemingly simple feature can be hard to deliver, but on the flip side there are dials a developer can turn to adjust his LOE up or down as desired. Tradeoffs as always.

josephholsten2• 16 years ago

> quality/UX standards were relaxed somewhat

Yow.

mkramlich3• 16 years ago

why "yow"?

edanm• 16 years ago

In the same vein, I highly recommend the article "How many Microsoft employees does it take to change a lightbulb?". This was features in Joel Spolsky's "The Best Software Writing 1" (http://www.amazon.com/gp/product/1590595009?ie=UTF8&tag=...).

Link to article: http://blogs.msdn.com/b/ericlippert/archive/2003/10/28/53298...

nostromo• 16 years ago

Nice affiliate link. ;-)

mpk• 16 years ago

Though not directly related to this post, the sentence 'well ... how hard can it be?' is a signature quote from the lead Top Gear presenter.

This almost invariably leads into a challenge that seems easy in theory but the practical execution is always plagued with unforeseen hurdles, crap, unexpected mishaps and other random elements.

It's not that hard to draw parallels between software development and the crazy Top Gear challenges, which is probably a reason that many coders who've never been behind a wheel enjoy watching them.

mkramlich3• 16 years ago

if I were to sum up the counter-reaction to a lot of folks here who think that the claimed difficulty is too high:

You're right, under the right conditions. Namely: small site, early in lifecycle, and a perfectly normalized and non-sharded/non-partitioned database.

But I suspect it's more complicated because:

1. they have a ton of traffic

2. they want to maintain a seamlessly perfect UX everywhere

3. their database is denormalized, sharded and/or partitioned

plus possibly:

4. it's gotten at least a little crufty with age (shrug, it happens)

eterps• 16 years ago

http://www.w3.org/DesignIssues/Axioms#opaque

fleaflicker• 16 years ago

I frequently quote the way Joel phrased this:

If you've spent more than 20 minutes of your life writing code, you've probably discovered a good rule of thumb by now: nothing is as simple as it seems.

http://www.joelonsoftware.com/articles/NothingIsSimple.html

"How hard can it be to implement?"

Related Stories