AI Assistants Have Poor Usability: Study of Alexa, Google Assistant, Siri (2018)

https://www.nngroup.com/articles/intelligent-assistant-usability/

by spideymans • 6 years ago

56 60 6 years ago

meheszjeno• 6 years ago

These solutions don't mean to be help/assist their users but to help their vendors to collect information about the users.

trevyn• 6 years ago

Correct; incentives are not aligned. Even for Apple.

briga• 6 years ago

Can't they do both? I personally find Google Assistant very helpful. I'm sure Google finds my personal data helpful. Seems like a fairly reasonable trade-off for me

drdaeman• 6 years ago

I disagree. It's not about collecting information - at least, not directly. Assistants don't seem to be designed to gather neither any substantial amounts of information, nor any unique information.

Due to their limited communication capabilities they're extremely poor at collecting meaningful data - there's simply no point in talking to them as their utility fails with anything but the simplest and most direct commands. There's not much data in "turn kitchen lights on", "set alarm at 10am", "play $album by $artist", "remind me to buy milk" and "convert 4.7 miles to kilometers" - at best, this exposes a few minor behavioral patterns that can be trivially determined even without a dedicated device to talk to.

And, hopefully, outright spying wasn't one of the design goals, even though they can be of dual use. Google, Apple and Amazon may be cooperative but they aren't NSA, after all.

My opinion, it's about use of particular services. Alexa will push you towards Amazon purchases, use of Amazon Music, etc. Similarly, Homepod requires investment into Apple ecosystem. And Google Assistant will push towards all things Google. In my opinion, voice assistants were made to be helpful within their particular ecosystems and thus add more ties to their vendors' services. (And, sure, those services collect data.)

IshKebab2• 6 years ago

Why does HN have so many conspiracy theorists?

They're clearly designed to make money by keeping you in the Amazon / Google / Apple ecosystem. I don't think any of them make much money from data collection because they don't collect that much data.

I have Google Homes and you can go and look at all the data it collects - for me it is 90% "hey Google play some music" or "hey Google what's the weather" or "hey Google set a timer for 10 minutes".

Useful for improving they speech recognition but not much else.

hyperman1• 6 years ago

There is 1 more important piece of information: Your voiceprint.

With it, google can identify you by voice, put a mic in a public infopanel, and gues which commercial has most impact.

If the NSA has access, they can classify public recordings by people at that place, so I assume they are interested.

HN has so many conspiracy theorists because we know what is possible with infotech on an industrial scale, the spooks have the money for it, and snowden basically provided the proof they did it.

TulliusCicero2• 6 years ago

Why does HN have so many conspiracy theorists?

Reflexive cynicism is easy, and makes you appear smart to a large number of commenters.

Of course, it doesn't help that often times, the automatic cynicism is entirely accurate, especially when contrasted against empty PR statements.

TulliusCicero2• 6 years ago

They're pretty obviously designed to do both.

sdenton4• 6 years ago

This is a bit inevitable. The UI of your usual screen based app has buttons for the things you can do. This has the effect of limiting your expectations to those features which have been implemented. A general voice query is open ended, though: there's no guidance around what exists, so it's very easy to invent a query that's 'off book' in one way or another... Things which haven't been implemented at all, or which have, but are requested in a way the machine does not expect.

chrisin2d• 6 years ago

Indeed, an open-ended interface like a VUI invite unimplemented interactions.

I worked on a conversational agent and now realise that there are fundamental usability barriers that are very difficult to overcome:

- Limited information bandwidth. Adults can visually read between 250–600 words per minute, plus they have peripheral vision that can help in scanning for shapes, colors, and images. Plus information in a GUI is persistent on a screen for easy reference. For voice, adults can only comfortably listen at 150–160 WPM and would need to hold information in their memory for reference. This makes voice ecommerce impractical for anything beyond familiar essentials.

- Lack of editing layer. It’s simple and straightforward to correct text box input, but it’s difficult to correct a voice command and doing so is ambiguous and adds extra chaotic information. A lot of people think aloud, so they frequently self-correct themselves mid-sentence or add on information in an ad-hoc manner: “I’d like a cappuci—er, make it a latte. (pause) And oh with soy milk.”

- High context switching cost. GUI’s keep contexts within frames, tabs, and windows — and the user can easily (low cost) switch between them. This is an Amazon shopping cart context, this is a Reminders context, and so on. A button press is unambiguously contained within its context. In voice, contexts are not quite parallel but sequential and have to be built up over time.

gundmc• 6 years ago

What's the point of posting this analysis that is over 2 years old in a young and rapidly iterating space?

outtatime• 6 years ago

Have human voices changed that much in two years?

leafboi4• 6 years ago

No But recognition and analysis has...

raobit4• 6 years ago

Who genuinely uses Assistants from their respective phone and for what tasks in general,just wanted to know is someone really making most use of it?

cik2e• 6 years ago

I use mine all the time to send texts, make phone calls, set alarms and timers, and add calendar events.

xapata• 6 years ago

Weather, news, music, and random questions while I'm not at my computer, or I don't have my hands on the keyboard.

endanke2• 6 years ago

I use them almost on a daily basis to control the lights in my apartment, it's surprisingly convenient and helps with multitasking when I'm in a hurry. The same thing goes for weather and reminders. The best use case when you're in the middle of something, eg. cooking and just want to make a simple note or set a timer.

gambiting• 6 years ago

Does it actually work for you though? I've quickly abandoned Google assistant because it just misunderstands what I say about 2/3s of the time. A friend of mine who has all his lights controllable by voice complains about the same issue - a lot of the time, Google just doesn't understand the command so by the time you get it "right" you could have walked up to the switch 3 times.

qntmfred• 6 years ago

I use google assistant multiple times a day. play pandora or youtube playlists on various speakers throughout my house, set reminders, add things to my shopping list or todo list, check the weather, set alarms, navigation if I'm driving, random google queries

DEADBEEFC0FFEE2• 6 years ago

I set alarms and reminders. I ask how to say thing in various languages, then ask those words in that language to see if Google say the right definition. I check facts.

I turn light on and off.

joshvm• 6 years ago

Fact retrieval performance varies a lot and is a good test of an assistant. Alexa is very good, because it can't rely on a screen (and it seems a lot of people work on making it good at knowledge type stuff). Google is also decent most of the time. Apple often defaults to "here's what I found" which is useless.

goldenManatee• 6 years ago

Announce reminders of various routine tasks, and calendar meetings, throughout the day.

mywacaday• 6 years ago

I use my Google home for timers while cooking, turn on/off lamps that have awkward to reach switch's, Christmas tree lights when the time arrives. My biggest usability issue hey/ok Google is a mouthful and does not roll off the tongue like Siri/Alexa.

dade_• 6 years ago

Bixby responds to almost anything, so it is useful to find my phone. It can answer a question about my calendar, am I available at 9 am tomorrow, but it useless for setting an appointment. ‘Create an appointment with Bill tomorrow at 9 am’, will create an all day appointment with a location 9 am.

Alexa doesn’t integrate with my calendar at all. (incompatible)

graeme• 6 years ago

Siri, for the following:

* Check weather

* Open a certain app on my watch without touching it

* Start a phone call or facetime

* Play some music —> this starts a personal radio station in apple music

* Add a reminder to my grocery list or another specific list. Or “remind me tomorrow”

* start a timer on my watch

* Set an alarm on my watch

* turn on/off smart home lights

* Some stuff with Siri Shortcuts. I use this mostly for time tracking timers, starting and stopping. But sky’s the limit here, you can start any multi step automation from siri if you have a use case that doesn’t require input

* Show me pictures from the web of X

* Search basic facts

kinghtown• 6 years ago

I’m a big fan of “.. remind me when I get home to ______.”

netsharc• 6 years ago

For me adding calendar events is faster through speech than using the GUI.

GUI: find Google Calendar icon, scroll the months to find the date, open date, scroll to find the time, open it, type the title.

Speech: "OK Google, add event on October 28 at 12pm, lunch with Grace".

Setting alarms is also very convenient.

lolinder2• 6 years ago

"Take me to..." as I'm buckling up in the car has been very helpful. Then I'll use it to send an "on my way" text.

hevelvarik• 6 years ago

I use Siri all the time for sending/reading text/WhatsApp messages and making calls and setting/stoping alarms and timers, and creating notes and taking pictures (Siri will open the app to where you are a click away from taking the picture. I also use Siri as a dictionary and weather forecaster.

I wish Siri was more powerful, to do things like create contacts and turn of the phone but Apple seems super conservative about what it will allow Siri to do

raobit4• 6 years ago

Sending & reading messages in public?,because apart from home automation and minor tasks,telling and hearing private infos through assistant is not that comfortable i feel.

inetknght• 6 years ago

I've found Siri is absolutely useless for anything except the following:

Hey Siri, set the lights to red.
Hey Siri, set the lights to fifteen percent.
Hey Siri, in Houston, what's the weather right now?

The weather question has to be worded absolutely correctly. If I instead ask:

Hey Siri, what's the weather right now in Houston?

Then Siri will stop listening at "now" and immediately answer saying that she needs location services turned on.

microtherion2• 6 years ago

[Disclaimer: I work for Siri; discount my enthusiasm accordingly]

For me, Siri when holding my phone is the least compelling use case. I kind of like using it for Alarm Clocks and Timers, because recognition in that domain is quite reliable, and one instruction saves multiple clicks.

Since touch navigation on a watch is less convenient than on a phone, some further use cases become more convenient with voice than with touch, e.g. asking "Is it going to rain today" before stepping out the door.

On Homepod, music is an obvious use case, which is unfortunately a rather difficult domain, because of the wide variety of media names. I use commands like "play some John Coltrane", "Shuffle play list Aggro", but also "who's playing piano on this track" (availability is a bit variable) or "what song is this". Home control is also convenient.

With Airpods, music is again the obvious use case, but I also like using them for walking directions (because you can walk without having to constantly glance on your phone). They also serve as a "poor man's CarPlay" in cars not equipped with a suitable media system (With the transparency mode on AirPod Pros, I feel that they are not an undue safety risk).

CarPlay is one of my favorite use cases, because I can navigate, listen to music, and listen and respond to messages without having to take my eyes off the road. When stuck in traffic, I also like asking what my ETA is.

raobit4• 6 years ago

Good to know. As you work for Siri(Apple) I want to learn and contribute in NLP fields as well.

1.Can you share your journey if possible?

2.Is Ph.D or masters necessary and how helpful the work related is in the company vs the research one does as part of Ph.D(because i have heard one doesn't get autonomy to researh one's own subjects under the supervisor but need to do what supervisor says)

3.Can you share any resource you lookup to learn and about Siri internal's working

4.How deadline works in research field ,currently working as a Front end developer deadline in my work as per ETA we can judge roughly,but how it works in research oriented field where one is unsure whether things are delivered as per requirement

5. Where do you see NLP future going?

Thanks

leafboi4• 6 years ago

I use it for ad hoc hands free navigation while driving. It works 60% of the time and I don't have to intervene with my hands.

60% is sort of like a gamble with no downside. Do I have to stop the car or can I just set the destination while driving? I would have to stop the car anyway were it not for voice activated commands.

29athrowaway• 6 years ago

These days, a useful command has been "Alexa, what is the air quality index?". It's a good way of knowing if I can open the window.

brokenmachine• 6 years ago

That's /r/aboringdystopia material right there.

mlang23• 6 years ago

My main criticism about Alexa at least boils down to bad usability. To me, it feels like learning a bunch of magic spells by heart. It feels so awkward. I am used to exact commands as a long-time CLI user. However, when it comes to speech, I somehow don't want to accept having to remember the other end of a pattern matcher. All the skills I have tried so far had a similar feel to it. I feel overwhelmed when I realize I have to remember how to launch the skill. I should probably give GA a try one day. But for now, I give the voice assistant thing 5 more years, hopefully it will be something I want to use then.

lolinder2• 6 years ago

I took a class in college on voice interface design, and the professor named this as the biggest hurdle that's yet to be overcome. Screens and keyboards are the computer's domain, and we're comfortable adapting to them. But speech is our domain, and we expect the computer to adapt to us. The problem is that the tech isn't quite there for real understanding of language.

Catsandkites• 6 years ago

Half on-topic: Is there an API on Android that allows you to write your own assistant with your own keyword set without sending everything over the wire?

An app with microphone permissions would obviously work-ish, but is it possible to listen in the background like the assistants do?

thelazydogsback• 6 years ago

We've had better technology in NLP and problem-solving since the mid-1970's -- and many methods that were deemed intractable are trivial now given that compute is on the order of 1 million times more powerful. Systems that use partial-order hierarchical planning, abductive logic programming, bi-directional search, constraint satisfaction, multi-modal interaction, multi-agent conversational modeling based on planning applicable speech-acts, etc., have been replaced with simple state-machines, and in "advanced" cases, simple slot-filling -- and more recently relatively shallow black-box stochastic methods based on deep learning models.

Having worked at two companies that offered such systems, I can say that this is due to both not understanding/appreciating what has been done in the past, but also because companies treat these systems like car companies treat features -- they know that an deliver rich features in the future, so the "logic" is to wait to do so, in the hope they can make more money in the long-term, and be able to dribble out new features.

To be fair, there is also the issue of reproducibility and predictability -- companies expect everybody's agent to respond the same way given the same input, rather than have variability in agents due to non-determinism or context and learning -- but we can't ask for for flexible "human-like" AI on one hand and also not expect the variation (and occasional misunderstandings) that humans would also suffer.

jbarrs• 6 years ago

Even though it goes against the basic premise of human-centered design, users have to train themselves to understand when an intelligent assistant will be useful and when it’s better to avoid using it.

I find this is one of the truest things about this article. It's very easy when you first get a new voice assistant to get carried away with the features it has available. Most are either very basic, struggling to take much additional context, or simply just gimmicks/novelties. The majority of tasks can be performed faster through a GUI when you account for the time it takes to brute force the right "natural language" expression to do the thing you need.

aaomidi• 6 years ago

Well the idea is that you get hands free control of the device.

I'm laying in bed and I suddenly realize I need an alarm for tomorrow morning. I don't want to turn my phone on cause that can mess with my sleep.

I just tell siri or Google to set an alarm and sleep.

Or I'm in the middle of cooking and I want to listen to music but don't want to have to wash my hands first.

That's really where the benefit of the assistants lays.

dafoex• 6 years ago

This would potentially destroy any hands free benefit a virtual assistant would have, back in the day of the secretary (the roll these talking Pringles tubes are trying to emulate) the bossman CEO or whatever would press a button on the office intercom to talk with his secretary. I think virtual assistants would benefit from a simple push-to-talk style input so they don't misinterpret a pause as a stop.

AI Assistants Have Poor Usability: Study of Alexa, Google Assistant, Siri (2018)

Related Stories