Humans vs. Microsoft Excel: The Quest For Smart Tools

http://futureofwork.glider.com/microsoft-excel-the-quest-for-smart-tools/

by turoczy • 13 years ago

41 55 13 years ago

futureofwork.glider.com

justinlilly2• 13 years ago

Typed fields sound neat!

samspenc• 13 years ago

At a previous job, I tried to convince them (and succeeded in some cases) to build a domain-specific web app instead of sharing and e-mailing ever-changing Excel docs.

I used CakePHP, and it was really easy to whip up a complete CRUD web-app with domain-specific information and DB tables, allow people to collaborate through a web interface, and be happy users.

Some apps are tougher than others, I'll admit.

cmwelsh• 13 years ago

You can turn Excel spreadsheets into web applications quite easily with Google Apps Script. We wrote a REST interface to a massive multi tab Excel file recently and it only took like an hour, as opposed to days of reverse engineering the calculations (the client did not know what the equations in the file were because the employee who created it left).

Basically I'm saying it will get much, much worse before it gets better.

roin• 13 years ago

I work with an army of very smart people who need to build incredibly complex models to drive our business. Like most companies, we turn to Excel. It's the only widely used application with a low enough barrier to entry that "non-technical" people can build legitimately complex (if not always robust) models.

But the problems with Excel and general spreadsheet modeling limitations offer such a great opportunity for improvement that would impact just about every business out there. I wish I saw more people tackling this, or I had a great idea to do it myself. The problems I see are around barrier to entry, in that it must be usable in just a couple minutes, simple enough for non-programmers to learn, and equally or more efficient to get a basic model functioning. Otherwise I just don't see wide adoption despite possible maintenance, accuracy, and reliability benefits.

James Kwak has written some smart stuff about this (http://baselinescenario.com/2013/04/18/more-bad-excel/), and I'm interested in what Data Nitro (datanitro.com) is doing.

peteforde3• 13 years ago

I think Data Nitro is really cool, and as a data geek the idea of having a real language available inside the Excel environment gets me very excited.

Unfortunately, by increasing the power and capability of what a careless user can do, while expanding the learning curve and moving Python code to a different place than any cell-level functions or global VBA functions are stored... all of this just makes the potential for issues of human failure far greater.

BrianEatWorld• 13 years ago

Is there a reason you don't use one of the current data analysis languages like R, Stata, SAS, etc?

elirubel• 13 years ago

Interesting and agreed. What if excel was very basic in functionality, and then as soon as you get into complex modeling or advanced functions you pull from a user generated (either private or public) template library. The trick would be in making the implementation of templating simple and intuitive.

The goal behind templating would be to prevent complex formulas from being entered in tiny cells, and to leverage a userbase that has "been there, done that." Do you think that would shorten new hire ramp up in using existing complex models etc?

gruseom3• 13 years ago

I worked in a similar environment and came to similar conclusions. In my view, your three requirements (usable without a big learning curve, intuitive to non-programmers, equally or more efficient than Excel) point directly back to the spreadsheet UI. Spreadsheet users get spreadsheets in a way they don't get anything else. It's their mental model and they love it and there's no way they're going to give it up. (In particular, there's no way they're going to migrate to even the simplest of general-purpose scripting languages.) Therefore the only thing that will get them off Excel is a new kind of spreadsheet. The question is, what would that look like? If it's too different from Excel, it won't meet your three requirements. But if it's too like Excel, there won't be any reason to use it, nor will it address Excel's limitations which are the subject of the OP. I've been thinking about and working on this ever since—it's a deeper problem than it appears to be!

Edit: by the way, you're right (in my opinion) to emphasize barrier-to-entry as a great danger in this area. If you look at the history of attempts to innovate in the spreadsheet space (most famously Lotus Improv), that is the reef on which they tend to run aground.

pyoung4• 13 years ago

I don't think scripting languages, of themselves, are the real barrier, after all, excel formulas are not too far off from what one would use in a language like R. I think the barrier is the data import/manipulation phase. Scripting languages loose out to the ease of copy/paste and data inspection that spreadsheets offer. It's not that using scripting languages for these tasks is very difficult, it's just that spreadsheets are, comparatively, much easier to use.

I worked in a place that used both, and the programmers were successful in preaching the benefits and converting some of the excel users, where it made sense. However, I imagine a large majority of analysts using excel in the corporate world do not have that exposure and don't realize there are other alternatives out there, or if they do know about the alternatives, they get quickly turned off from them when they try to import/manipulate the data. I don't think any tool, by itself, is going to push individuals to change. People need to be exposed to the benefits of approaching data analysis as a small software/programming project over the long term. It is short term thinking that pushes people to use excel in the first place.

ajscherer• 13 years ago

The author said it himself, these errors were caused by humans, not excel. I don't think making excel "smart" enough to prevent people from making dumb mistakes is a good idea. In fact I think that would transform excel into microsoft word, which is the only thing that could make life worse for your typical competent office worker.

Excel already has a lot of this crap. It already requires an act of god to prevent excel from changing your string of digits into a number or date. It already shows you little exclamation point icons when your formulas omit adjacent rows or are different from other formulas in the same row/column.

A tool will never make it possible for dumb people to solve hard problems easily. It's like trying to design a knife that makes it impossible to cut yourself. Nobody with any kind of a clue would want that knife.

A tool should be straightforward and intuitive, but it shouldn't aim to be smarter than it's user.

thiele2• 13 years ago

There is low hanging fruit in the problems to solve though. For example, finding and highlighting outliers in a column. It's a relatively simple feature but yields big benefits to users.

pyoung4• 13 years ago

Except it isn't that simple. Your definition of an outlier may be completely different from some one else's (95th percentile, median/regression method, or simple max/min, some of these are already possible in excel). Then, how is the information displayed? Highlighting cells? Well if you have +10k rows of data, than that won't be too useful. Creating a summary in another column/sheet will add to the already cluttered 'results' spreadsheet, etc...

This is why I recommend using programming based tools for data analysis. Any non-programming tool has to find a balance between the number of features offered and the complexity/ease-of-use of the tool. With programming tools, you merely have to find the right package (or build your own) which essentially results in getting the exact set of features that you need to solve your problem.

wikwocket• 13 years ago

Excel already does this to a degree. If a cell has a formula that is inconsistent with its neighbors, it will be flagged. If a formula omits cells adjacent to the range it is working with, it will be flagged.

Check out Options: Formulas: Error checking rules. It's almost as if Microsoft has been continually developing this software for years and years and they have seen many common mistakes. ;)

calinet64• 13 years ago

Making Excel "smart" is not exactly what he's talking about. He's playing with ideas. It's high level, but you can do high level sometimes. You need to.

Your analysis is too low-level and nitpicky for this high level idea playground.

Excel is still a dumb tool. He's absolutely right.

We have ways of making smart tools. It's called "programming" and "database engineering." That part's a little too hard right now, and part of why it's hard is because if you try to make it easy to mould a flexible program to try to define what your data means and how it should work, it becomes so generic as to be as unusable and amorphic as a cloud of smoke.

I've tried this, moving from a space where Excel was predominantly used, and trying to capture the process into an application. We tried to keep all the customizability and malleability as Excel: but I'm now convinced that was a huge mistake. It led us into genericland.

It is possible to build applications that work for these processes, using the tools of programming and good data design. But that's why we have thousands of different applications all trying to solve these different problems, all of which you could probably represent in Excel in some way. The fact that all those applications exist means that people want something smarter than Excel. It proves the case.

But we haven't yet bridged the gap between Excel's extreme flexibility and an Application's intelligence and process fit. That's because it's really difficult to make that work; to fit all the pieces together into something that makes sense in both realms. It's really hard!

The solution will be a user interface masterpiece. I'm convinced of that. And it will be layered, like an onion, allowing people to build any application they need just by telling the computer about their data and how it fits together and what it should allow them to do. Someday we'll have a system—some sort of super-Rails—that makes this so easy that anyone can do it. Someday Programming will be an ancient art, something that only your Grandfather did, like crafting your own tools or woodworking or making jams and jellies.

Someday that will be true. But the way we get there is by thinking about the difference between Dumb Flexible Excel and Smart Rigid Applications and how we bridge that gap, because it's difficult and it's possible. We have to think at this high level, way up in the clouds—and more people should, and there's no reason to shoot them down.

jacques_chester5• 13 years ago

You can have adaptation or adaptiveness, but not both. Either a system is well-fitted to a purpose and needs professional intervention to change, or it is permissive and undemanding and can be turned to any purpose and do it poorly.

Life is full of tradeoffs. Sometimes the general, permissive loosey-goosey system is what you want. Sometimes it isn't. It takes judgement to decide which is which and when to switch.

zvrba• 13 years ago

Hmm... Excel-like front-end to Haskell?

gruseom3• 13 years ago

I would love to see someone try this.

malandrew• 13 years ago

This would be awesome.

jacques_chester5• 13 years ago

For inspiration, consider SIAG:

http://siag.nu/siag/

coderguy1232• 13 years ago

Having worked in nyc financial world, I can say excel is here to stay - at least for foreseeable future. Heck, the only reason I have windows running in parallel in my MBA is excel. We use to have what we called "excel test" for all quants and analyst positions. We use to QA our excel sheets just like any other software application. I am surprised that many companies don't do that - it only makes sense.

pinaceae3• 13 years ago

why? excel runs perfectly fine in its OSX version. which differences have you encountered?

coderguy1232• 13 years ago

being very heavy excel keyboard shortcut user, assumed it wouldn't be the same in mac. never tried it.

kermatt• 13 years ago

VBA, which was removed in Excel 2008 and returned in Excel 2011, is not fully compatible with Windows versions, where most users author code.

Backend data connectivity options are virtually non-existant. Excel 2011 has what appears to be a 10+ year old version of MS Query.

Some 3rd party financial plugins are Windows dependent, and I believe some of the Microsoft analysis additions are as well.

peteforde3• 13 years ago

This is a problem I've spent a lot of time pondering. The author makes several suggestions; one in particular (a GitHub for datasets) is an idea I felt strongly enough about that I founded a start-up called BuzzData to address this concern. Unfortunately, we did not find our critical mass and the product has since successfully pivoted into an entirely different product called LookBookHQ.

I do have another idea, which is to bring the concept of testing to the Excel world.

I think that the combination of:

- a framework for building up test cases that can be run against the data as part of the saving process (think rSpec)

- an English-like, indented grammar for defining user stories that can potentially generate code for tests (think Cucumber)

- a strong push for a culture of testing in the Excel community that would have to include Microsoft

I dunno, honestly. Could be crazy. The culture might never accept testing.

But then again, one of the basic ways to sell something is to give people insurance against humiliating themselves. It's possible that this thinking would make a lot of sense to the twice shy, highly risk-averse board members of the major financial institutions and governments of the world.

vladsanchez• 13 years ago

Started following LookBookHQ. =D

Ever read "Specification By Example? That approach along with some of your suggestions could be used to prove your testable specification spreadsheet concept.

The market awaits for you.

jacques_chester5• 13 years ago

I thought of a similar idea, and so have others: http://www.sysmod.com/

pinaceae3• 13 years ago

disagree completely. MS Excel is a masterpiece in software engineering, definitely the best piece of software MS ever created. Large pieces of the economy run on Excel, finance, controlling, etc. are unthinkable without it.

the flexibility, ease of use, speed is astounding. applying colors to cells and then being able to sort by color still blows my mind - not the feature per se, but the fact that they included it and solved a common problem (how many times did a client mark something by color in a big spreadsheet...).

excel is mobile, works offline. replacing it with some webapp or too much logic just shows the complete misunderstanding of its appeal.

AdamFernandez• 13 years ago

This. It's important to remember that from the perspective of millions of corporate users (not the hacker crowd), this is is THE tool. When you think about it, Microsoft really is Excel. That is their core. Why is it difficult for corporations to switch from Windows? Office. Why Office? Excel. If anyone attempted to create something that not only matched Excel (with all the messy file compatibility issues and macros), but surpassed it, they would be striking at the heart of Microsoft.

thiele2• 13 years ago

Agreed. Although you could replace 'Excel' with 'Word' and still be just as correct.

trapezoid• 13 years ago

Excel is a great tool, but it has its shortcomings. The article is highlighting the main one: it's easy to build huge, brittle systems with Excel.

This immediately brings up questions: What tools exist for building robust spreadsheets? How can we encourage people to use them? Should we make better ones?

rbanffy• 13 years ago

MS Excel is a masterpiece in software engineering, definitely the best piece of software MS ever created.

Won't comment on the "masterpiece" thing, but I like SQL Server better. Only their Natural keyboard and mice are better than SQL Server.

Large pieces of the economy run on Excel, finance, controlling, etc. are unthinkable without it.

Now, that's scary.

calinet64• 13 years ago

You are thinking on a whole different level than this article is trying to think about. "Can't see the forest for the trees" if you will.

karamazov• 13 years ago

The fundamental problem with big spreadsheets is that they're complex pieces of software, and they're being designed by people who haven't studied software engineering.

Building a complex program is hard. It's untenable if you don't approach it properly. This is true whether you're using Excel, Python, LISP, or anything else. Better tools can go a long way to help, but the ultimate answer is educating people on how to properly engineer systems.

On that last point - at DataNitro[1], we let people script Excel with Python, which is a better tool than VBA or than doing things by hand. The result is that for a given level of complexity, you see fewer errors and more robust spreadsheets; but when you give people more power, they naturally build more sophisticated spreadsheets. All else being equal, this might result in more, not fewer errors.

The great thing about introducing a new tool, though, is that there's also a chance to introduce new ways of working. Suddenly, people start building spreadsheets that work with version control and run unit tests. This helps tremendously with robustness and reliability.

[1] https://www.datanitro.com

maxcan• 13 years ago

As a dynamically typed language, how does Python specifically yield fewer errors than VBA? Or even compared to C# or F#?

6d0debc071• 13 years ago

I dare say that the problem is less about Excel being dumb than that if you're interested in the underlying logic Excel hides all that from you the minute you click off a cell. There's black box programming and then there's just silliness.

As for data-entry, ideally, that's not something that humans should be doing for anything but the most trivial concern. If you have huge columns of financial data or test results or what have you, then having someone type it into a computer is silliness of the highest order.

bliker• 13 years ago

I got theory. There is a limit what you can do with WYSIWYG editors. Once you reach the limit, the software starts getting in your way, hiding your errors and making you work harder.

It applies for many software. From HTML to 3D modeling.

glaugh• 13 years ago

I think it's impossible to replace Excel en masse, so lots of folks are starting to pick off bits of it to replace. And hopefully in doing so they'll invest in making those bits a lot smarter than they are in Excel.

Examples from the S12 YC batch alone:

. VBA isn't great, so DataNitro is letting you use Python.

. Grid is (mobiley) replacing the basic list-making use case of Excel.

. (Stretch example:) Financial modeling of big personal finance decisions is a nice use case of Excel, but too hard for most people, so SmartAsset bakes those kinds of models into an on-site calculator.

. Statwing is replacing pivot tables, which are clunky, devoid of statistics, do no error checking, etc.; in a few clicks Statwing will automatically choose, run, and interpret a statistical analysis, all the while looking for and alerting users to outliers or (some) other data issues. [I'm a cofounder. Also: https://www.statwing.com/demo]

I'd expect that soon someone will come along soon and pick off the financial modeling use case in a way that's a bit more generalized than SmartAsset. Really tough UI challenge, but I think doable.

Estragon• 13 years ago

I suggest you change your statwing demo to something more positive.

SiVal• 13 years ago

If you are amused at what autocorrect does for writing, imagine the hilarity of economic decisions based on "smart" spreadsheets.

jacques_chester5• 13 years ago

There are entire organisations and conferences about the dangers of spreadsheets[1] and there are now programming standards for develop spreadsheets[2].

Basically, the normal rules apply. If you want safe spreadsheets, you have to plan and verify.

The reason this doesn't happen is because firstly, spreadsheets are very accessible. Every working large system evolves from a working small system, as John Gall observed. Excel et al make that evolution relatively easy.

The second is that spreadsheets enable non-programmers to work with concrete concepts instead of the abstract ones that programming languages encourage. It's easy to visualise locations on a sheet when you are looking at the sheet.

Finally, spreadsheets support calculation by propagation. They are an excellent declarative environment. Most programming languages do not support this model of programming (though we're groping towards it with concepts of binding).

[1] http://www.eusprig.org/

[2] http://www.fast-standard.org/document/FastStandard_01b.pdf

justinlilly2• 13 years ago

Excel will always be used BECAUSE its so free-form.

http://readwrite.com/2013/06/11/excel-is-an-art-form-these-b...

vinchuco• 13 years ago

We need to think beyond the limitations of tools no matter how pervasive their use. It reminded me of : http://vimeo.com/66085662 (you can skip to the demo)

Humans vs. Microsoft Excel: The Quest For Smart Tools

Related Stories