Distributed Systems Are a UX Problem

http://bravenewgeek.com/distributed-systems-are-a-ux-problem/

by platz • 11 years ago

89 43 11 years ago

At trivial scale, sure. But consider a multicore CPU or GFS. Both these are distributed systems, but where UX has no solutions.

einarvollset3• 11 years ago

GFS: Google File System (sorry should have said)

Terr_• 11 years ago

That's only because you've drawn such tight boundaries around your definition of "distributed system" that "User Experience" is actually "API design".

The same principle applies: The system can't do everything magically, so it exposes some of that through the API. I mean, I doubt the GFS API is simply a read() and write() method that are always guaranteed to work instantly.

Instead you start posing the same kind of "Hey, I need you to make a decision here" tradeoffs that the author is talking about:

"Sometimes X will happen, if so you need to check Y and try to recover with Z, unless you didn't really care about W. If you don't want to recover and would rather discard, make sure you call Q, otherwise we'll eventually destroy it after T minutes."

einarvollset3• 11 years ago

I would have to disagree. One of the main benefits of distributed systems is that you don't need to know that in reality your file system is sharded on to 5 different disks at 2 different colocs. I understand that sometimes you need to manually recover, but OP sweepingly states that distributed systems is a UX problem, which I think is a ridiculous reduction and equivalent to saying "well this shit is hard so let's just spill the guts of the system internals up to the user"

tylertreat• 11 years ago

There seems to be a bit of confusion over just what exactly I'm trying to get at here. I'll be the first to admit, the article might not do it justice. This is a comment I posted on it which hopefully helps clarify:

The point is there are certain realities inherent in distributed systems which can’t be papered over, and those realities often manifest themselves at all levels in the stack. Sure, we can try to build abstractions which hide those problems, but they often leak or simply eschew the problems in dangerously subtle ways.

Frontend and UX folks need to understand what they are building on. They need to understand the potential pitfalls of these abstractions and why they are as such. Sometimes there is a good business case for apologizing, sometimes there’s not. These things are never black and white, but if everyone can understand how systems work, we can build better ones and make better choices.

AnEngineer2• 11 years ago

And my reply:

I agree that there are complexities (as you state, “certain realities”) inherent in distributed systems. If there is one thing I believe we could both agree upon is that distributed systems increase the surface area of complexity exponentially when compared to the relatively simpler stack of clients communicating with one system.

Failures in a distributed environment must be expected. Embraced even, since treating them as “an exception” is tantamount to the proverbial ostrich poking its head into the sand.

However, failure in a distributed system does not necessitate “Frontend and UX folks” having to address failures which are not directly a result of client communication with the next tier. For example, should the UX be prepared to handle when the primary provider of a service is down and fail over to a secondary system? If the service is the one which the client is directly interacting with, then likely that makes sense.

But what of the case when the primary system has issues interacting with its collaborators? Such as its persistent store. Or a strategic partner providing required functionality for the workflow. Is it your position that this type of failure be handled by the UX?

Why wouldn’t a server maintain separation of concerns and only communicate to a client when its options have been exhausted? IOW, why assume what “the other side of the connection” is going to do in order to satisfy a request placed on it?

In summary, my position is that when the collaborators in a distributed system have a well defined software contracts (protocol, expectations, etc.) and all parties adhere to them (without assumptions in implementation), then this allows the system(s) involved to provide resilience in depth. However, the fashionable trend of replicating a PowerBuilder environment does not lend itself to these separation of concerns.

Foisting complexity to either end of the spectrum is where truly nasty problems are bred.

huslage2• 11 years ago

In a distributed system, there is no "primary" system. All systems are distributed and decoupled. The UX sends a request and waits for a response that itself be composed of other requests and responses from multiple other systems. The failures have to be assumed and mitigated for at the UX level, not at some back-end level.

mrxd• 11 years ago

As a UX person, I'm intrigued by this idea, but I would probably avoid creating situations where it is necessary to apologize. If you confirmed a flight, hotel and rental car, but then came back with a "Sorry!" later, that would be a negative experience for a user. They may have passed the confirmation on to others or set other plans in motion, which may also have to be "rolled back". You would need to compensate customers for the inconvenience.

The way to avoid that is to show requests in a pending state, but this has other implications. Users will want to continue to shop and to cancel pending requests, which might mean lost sales.

Generally it's more important to paper over the realities of a complicated system when you have a customer-facing self-service tool. Exposing those realities might create too much complexity for self-service, and you may need sales and service people to manage it. Basically you've built a travel agency instead of Expedia, which, again, is more costly.

jamesblonde• 11 years ago

Please don't read this article. I wasted 2 mins of my life on it....

vog• 11 years ago

Would you care to elaborate on that?

cies• 11 years ago

Myself did not think it was that bad, but the article is not very concise. The first comment on the article (below it, by "An Engineer"), sums up my sentiments pretty well.

uuilly• 11 years ago

I really enjoyed the analog world analogies like the work order.

hinkley• 11 years ago

A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.

-- Leslie Lamport

panic2• 11 years ago

Running this argument in the other direction: if you want an uncompromised user experience, you should use or build a centralized system rather than a distributed one.

prewett2• 11 years ago

Reading your comment made me think of Apple and Microsoft. Microsoft makes an operating system that runs on a distributed hardware ecosystem and they have the unenviable task of making the user with crappy (but cheap) hardware actually work. Apple is completely vertically integrated, so they can make sure everything works smoothly in their vision, but you only get Apple's vision.

Then you have Linux, which is a distributed software ecosystem on top of a distributed hardware ecosystem. UX is terrible, but flexibility is through the roof. You can pretty much have exactly what you want, although you might have to do parts of it yourself.

ArkyBeagle• 11 years ago

UX doesn't have to be terrible. It's just a long twisty maze of interlocking decisions. Depends on the U, too - I feel better with command line tool output and filters than with fancy GUI furniture.

Garlef• 11 years ago

That's a nice idea; But it is a misleading one.

Even an app for your phone that communicates with a server is a distributed system.

Even a programm performing a simple read/write operation on the file_system_ is part of a distributed system. Handling exceptions is precisely a way to deal with the uncertainty of these two systems working together.

marknadal4• 11 years ago

No.

If your server is in Virginia and you have a customer on a phone in Jo'burg... they're gonna have one sucky comprised experience.

Latency kills, and if you are centralized then your latency goes exponentially up the further away you get from that point.

panic2• 11 years ago

You don't necessarily need to build a distributed system to solve that problem. You could have an entirely separate centralized system which serves that region.

rubiquity• 11 years ago

This is why the first rule of distributed systems is: Don't build a distributed system if you don't have to.

Unfortunately, a lot of problems eventually grow beyond centralized solutions and thus have to be distributed.

marknadal4• 11 years ago

I work on distributed systems, and I thought this was a nice post and echoes some of my own sentiments.

The hardest problems in distributed transactions (banks, inventory, etc.) are often easier solved with human psychology (UX) than algorithms.

Hi, I am an ATM. Yes you have money, yes I am offline and can't check, yes you can withdrawal so you as the customer are happy with high availability. BUT I know who you are, and if you cheat me I will punish you when I find out!

Hi, I am a shopping cart. Why yes we have one of those in stock, but I am offline so I can't check. I'll take your money now and have it 2 day delivery for you. Oops, I just found out we don't have it in stock but I already have your money, this will take a few weeks now but we'll give you $20 off your next purchase - or do you want a refund?

This is the better approach, changing your business model to prioritize customer satisfaction (UX). Trying to build a globally consistent system instead either has to break the laws of physics with the speed of light, or make your customer have to wait - and if they have to, they probably won't be your customer anymore. One of these options is possible, but incorrect for your business, therefore use a distributed system with good UX.

learnstats24• 11 years ago

Why yes we have one of those in stock, but I am offline so I can't check.

Assuming the "I am offline" part is redacted, I view this as deliberate lying - deliberate in the "we already have your money" sense that you identified.

If I had made a decision to order with you because you had claimed something was in stock, but it wasn't, I will withdraw my patronage from you and I will complain loudly and vigorously.

This type of behaviour/UX has permanently harmed my relationship with several large retailers.

marknadal4• 11 years ago

Personal curiosity: Do you still use Amazon? They do this behavior.

Clarification: It need not even be offline. Even if you are online, there is no way to tell if in the TimeToServer after you clicked the buy button that somebody else hasn't already clicked the buy button.

And you can't say this is trivially solved by a central master server that uses a "first come first serve" basis that then knows to report an error back to you after you clicked the button.

Why? Because the whole point of online shopping is that you might have customers on opposite sides of the nation or world. There is no guarantee they will hit the same server, or even if you sharded it so they will a single server can only physically scale up to so many requests.

So point being this IS NOT LYING, because suggesting I can communicate faster than light or predict the future is lying. As I believe the article mentioned, computers can only make their best guess. This isn't a lie, but yes they might make a mistake - and the best corrective action is to apologize and compensate.

It would be a far greater mistake to assume FTL knowledge, and a lie to think the machine can't sometimes fail and be wrong.

jrochkind1• 11 years ago

I think to call it lying is based on the assumption that there is some perfect answer. There usually isn't.

I work at a university library. Our inventory is fairly small on internet scale (~3 million items in 5 or 6 separate 'warehouses'; although we usually only have 1-3 count of each 'item' in inventory), so we don't really have these large scale/distributed problems. The biggest problem we have with the system saying an item is 'in stock' when it isn't is -- the item has been stolen or lost, and we haven't noticed yet and recorded it as such.

Is it "lying" if our system says it's on the shelf, when in fact it's been stolen or lost and we haven't noticed yet?

There are obviously ways we could improve our 'loss reduction'. But there will ALWAYS be cases where the system's knowledge is an imperfect representation of the real world, in any system.

"I've been offline for 10 minutes so the last information I have is as of 10 minutes ago" is just one more.

You can spend more money to try to make the information more accurate, but it will never reach 100% (even before you add in distributed computing, which adds some of it's own issues), so as with everything, it's cost-benefit, how much does the customer care, what can we afford to do, at what point is our information good enough to keep them happy -- and, like the OP says, how do we properly make the UX to keep them happy despite information that's not 100% accurate, which it NEVER will be.

huslage2• 11 years ago

This is _exactly_ how Amazon works. It has a best guess at inventory at a given moment, but it's a snapshot in time that is sometimes wrong. The inventory is verified. The money is taken out. The pieces are all decoupled and essentially offline. This isn't a lie, it's an optimization and works quite well. If any piece fails, the recovery is obvious and easy to deal with.

shockzzz• 11 years ago

If you like the article, you might like CockroachDB. They just got funded by Google Ventures.

https://github.com/cockroachdb/cockroach#design

https://twitter.com/GoogleVentures/status/606534505332154369

Distributed Systems Are a UX Problem

Related Stories