Hey Siri: An On-Device DNN-Powered Voice Trigger for Apple’s Personal Assistant

https://machinelearning.apple.com/2017/10/01/hey-siri.html

by gok • 9 years ago

134 81 9 years ago

machinelearning.apple.com

allenleein2• 9 years ago

2012: OK Google / 2014: Alexa / 2017: Hey Siri

growt• 9 years ago

Not sure what you're trying to say with that timeline. Siri is available (on iPhones) since 2011.

hiddencost2• 9 years ago

I think the point is that on-phone DNN wake-word detection has been around for many years. Google shipped it like ... 3? years ago. A lot of Apple's technical ML posts feel a bit lackluster for that reason. Targeted at people who aren't active in the field.

gordyf• 9 years ago

Hey Siri has been an iPhone feature since the 6s, which came out in 2015.

Viper007Bond2• 9 years ago

It's a feature on older iPhones as well (such as my regular 6) but requires being on a power source since those lack the specialized, low-power chip.

RKearney• 9 years ago

I own multiple iOS devices, a few Echos, and a Google Home. One of the things I noticed after getting an Echo was how much more fluid and simple it was invoking the "assistant". Simply asking, "Alexa, what's the weather today" just seemed so much more natural than having to prefix everything with "Hey" or "Okay".

Using Siri or Google Assistant for more than one question at a time quickly makes me feel like I'm going insane. "Hey Siri.. Hey Siri.. Hey... Hey..."

I'm hoping Google and Apple fix these subtle annoyances. Or maybe it's just me.

huebnerob• 9 years ago

I think the reason why the 'catchphrases' for Siri and Google Assistant are longer is because the cost for false positives is higher. Phones are subject to much more varied conditions than a cylinder that lives on your nightstand. Anecdotally, I have seen Alexa activate incorrectly much more than Siri, I just care a lot less.

hammock2• 9 years ago

Hey Google and Hey Siri are not longer than Alexa.. still three syllables.

danso2• 9 years ago

Isn't part of the point of the prefix to avoid collisions with sounds that are part of everyday speech not intended for the assistant? "Alexa" becomes problematic when Echo is used in a office/home in which someone is named Alexa -- among the top 100 most popular female baby names since 1995 [0] -- which is why the wakeup word can be changed to "Echo" or "Amazon" or "Computer".

But the sound of the wake word, whether it's just "Alexa" or "Hey Siri" vs "Siri, doesn't seem to deal with the main issue of your complaint, which to me is how limited "conversation" is with the assistant.

If you ask Alexa for the weather, you'll still have to say her name for any followup questions within that immediate context, i.e. "Alexa, what's the weather today? Alexa, what's the weather this weekend?".

Though there are a few functional exceptions in which Alexa will prompt you for additional information without needing to be re-awakened, e.g.

You: "Alexa, set my alarm for 6 'o clock"

Alexa: "Is that 6 'o clock in the morning, or in the evening?"

[0] https://www.ssa.gov/OACT/babynames/index.html

wvenable• 9 years ago

I have my Echo set to use "computer" as the wake word because it's just way more fun that way. But, of course, "computer" is a pretty common word these days. It hasn't yet been annoying enough for me to change it.

Darthy2• 9 years ago

I always felt that Siri was named with the intent that it could be used as a wake word (so you could say "Siri what's the weather today?") because Siri itself is a rare given name and [si ri] are sounds rarely said at the beginning of a sentence.

And then at some point Apple realized they had to make a longer wake word to cut down the number of false positives ("Siri" -> "Hey Siri", from 2 syllables to 3).

Google probably went through the same process ("Google" -> "Okay Google", from 2 syllables to 4).

Amazon probably deliberately chose a 3 syllable name with "Alexa" for the same reason.

I can imagine future improvements where we can have the originally imagined wake words "Siri", "Google" and "Alexa", and at that point I would be most happy with "Siri" because it would be short and not-corporate.

analogmemory• 9 years ago

You can change the prompt it listens for

vollmond• 9 years ago

Plus, "Ok, Google" is just a terrible sequence of consonants/syllables. Just doesn't work in my mouth.

timdorr• 9 years ago

"Hey Google" is slowly becoming a secondary option: http://www.androidpolice.com/2017/10/17/hey-google-hotword-n...

2bitencryption• 9 years ago

Glad I'm not the only one -- I just can't say that phrase without garbling it: "Okayggglglele"

hmage• 9 years ago

If you're doing more than one interaction, you can just hold the home button to activate Siri for followups. And you don't need to prefix 'hey siri' if you're activating Siri via home button or screen button.

I've seen so many people prefix 'Hey siri' when they don't really need to:

Holds home, Siri activates "Hey Siri, how's the weather?"

If could just be "How's the weather?" In that instance.

cjsawyer• 9 years ago

I shouldn’t need to click a button to effectively interact with my -voice- assistant :)

tambourine_man• 9 years ago

I wonder how that works on the iPhone X

UnoriginalGuy2• 9 years ago

you can just hold the home button to activate Siri for followups

Unhelpful while I am driving...

A voice assistant that requires physical controls massively reduces its utility and core purpose.

azr79• 9 years ago

what if someone in the room is named alexa

Gaelan2• 9 years ago

Then you change Alexa to one of the other options (Amazon, Echo, Computer)

heavymark• 9 years ago

Agreed. The constant hey siri is exhausting vs Alexa's approach. Though as others noted Siri was developed for a phone and devices on the go. Echo is a device that is in your home where you don't have to worry about someone saying Alexa unless a family or a friend is named that in which case you can change the word. HomePod will be the first chance for Apple to change that but pretty sure they are going to stick with Hey Siri for consistent and reliability.

lloeki2• 9 years ago

I think it comes down to the fact that for all the « conversational » publicity stunt, the current implementations insist on the voice assistants to be non-modal. When you think about it, every vocal interaction you have with people is explicitly (through voice) or implicitly (through body language) modal. How hard would it be for the mode to have to end with « thank you Siri » or a sensible time-out, or now with the iPhone X, loss of directed attention (and even activated again with return of directed attention within a reasonable timeframe)

Viper007Bond2• 9 years ago

Pretty cool of Apple to post such technical blog posts as this. I love it when companies do this.

ktamura• 9 years ago

Indeed. Quietly but surely, Apple is changing its corporate policies around developers. We used to never see Apple developers at conferences with an official affiliation, let alone having them onstage as a speaker. I once ran into an engineer at a Ruby conference in which the guy demurred who he worked for, and when he finally told me he worked for Apple, he had to remind me the whole "what I say does not reflect nor represent..." preamble. That was as late as 2014.

Great to see such corporate changes toward developer-friendliness.

ComputerGuru• 9 years ago

Just because (judging from the comments) non-iPhone owners may not be aware, this isn't a new feature in the iPhone at all, Siri has had voice-activated prompts since late 2015 [0]. What it looks like is the team finally got the OK to share technical details about how this works with the public, and that's what we're seeing here.

Apple doesn't usually do tech writeups; I imagine a pinhead at the corporate level decided it wasn't worth the risk of leaking any "secret sauce" until now.

0: https://www.cultofmac.com/390181/5-ways-hey-siri-will-change...

IBM• 9 years ago

That pinhead was Steve Jobs.

Isamu• 9 years ago

Pretty cool how they reduce the power consumption - when it first came out, "Hey Siri" required your device to be plugged in:

To avoid running the main processor all day just to listen for the trigger phrase, the iPhone’s Always On Processor (AOP) (a small, low-power auxiliary processor, that is, the embedded Motion Coprocessor) has access to the microphone signal (on 6S and later). We use a small proportion of the AOP’s limited processing power to run a detector with a small version of the acoustic model (DNN). When the score exceeds a threshold the motion coprocessor wakes up the main processor, which analyzes the signal using a larger DNN.
Apple Watch uses a single-pass “Hey Siri” detector with an acoustic model intermediate in size between those used for the first and second passes on other iOS devices. The “Hey Siri” detector runs only when the watch motion coprocessor detects a wrist raise gesture, which turns the screen on. At that point there is a lot for WatchOS to do—power up, prepare the screen, etc.—so the system allocates “Hey Siri” only a small proportion (~5%) of the rather limited compute budget. It is a challenge to start audio capture in time to catch the start of the trigger phrase, so we make allowances for possible truncation in the way that we initialize the detector.

Another interesting nugget:

There are thousands of sound classes used by the main recognizer, but only about twenty are needed to account for the target phrase (including an initial silence), and one large class class for everything else. The training process attempts to produce DNN outputs approaching 1 for frames that are labelled with the relevant states and phones, based only on the local sound pattern. The training process adjusts the weights using standard back-propagation and stochastic gradient descent. We have used a variety of neural network training software toolkits, including Theano, Tensorflow, and Kaldi.

kejaed2• 9 years ago

This is interesting and a feature that I didn't know about, and Hey Siri will often not trigger in the car for me while driving. Now I know to retry and I'll have a better chance at triggering Hey Siri.

"We compare the score with a threshold to decide whether to activate Siri. In fact the threshold is not a fixed value. We built in some flexibility to make it easier to activate Siri in difficult conditions while not significantly increasing the number of false activations. There is a primary, or normal threshold, and a lower threshold that does not normally trigger Siri. If the score exceeds the lower threshold but not the upper threshold, then it may be that we missed a genuine “Hey Siri” event. When the score is in this range, the system enters a more sensitive state for a few seconds, so that if the user repeats the phrase, even without making more effort, then Siri triggers. This second-chance mechanism improves the usability of the system significantly, without increasing the false alarm rate too much because it is only in this extra-sensitive state for a short time."

dannyw• 9 years ago

This is brilliant. Apple could also cut out the “hey” part to make it more natural.

“hey Siri... Siri”

thisacctforreal• 9 years ago

That seemingly simple repeat functionality would require a much less simple nn, from what I understand.

That would trade off both security and privacy for very limited gain.

crooked-v• 9 years ago

"Hey Siri" has always seemed like such an awkward phrase to me. "Alexa" flows so much better, besides the general awkwardness if someone in the room has that name.

spike0212• 9 years ago

Odd, Hey Siri always triggers for me in the car while driving.

kejaed2• 9 years ago

Maybe you've got a nicer (quieter car) =D.

It usually works for me too, but sometimes it doesn't, now I know to retry.

paul7986• 9 years ago

With the iPhone 8 Siri is finally as useful as my Google Home speaker in terms of reliability and utility. Like to find your iphone say "hey siri, where are you," or talking to Siri in the car for changing songs or asking her to play similar songs. She finally understands almost every command I send her.

eric_h• 9 years ago

I occasionally use the "hey Siri" feature while in bed to mess with alarms in the morning (i.e. cancel whatever I had set and set a later one in the morning ;) apparently my morning groggy voice always requires a second hey Siri.

As an aside, alarms are literally the only thing I've ever used Siri for.

rjeli• 9 years ago

same, I use Siri every day, and only ever for two reasons:

"Hey Siri, set an alarm for {seven,eight,nine} a.m." "Hey Siri, call {mom,dad,[friend]}"

I've created unambiguous nicknames for my friends so I don't get name collisions

georgehm• 9 years ago

Does anyone know of papers and/or example implementations of similar DNNs for accoustic modeling using tf or some other framework?

gok• 9 years ago

Google's Deep KWS paper [1] is kind of similar, although they don't use an HMM.

[1] https://static.googleusercontent.com/media/research.google.c...

singularity2001• 9 years ago

Your connection is not private

Attackers might be trying to steal your information from machinelearning.apple.com (for example, passwords, messages, or credit cards). Learn more NET::ERR_CERT_COMMON_NAME_INVALID

Access Denied

You don't have permission to access ".../machinelearning.apple.com/2017/10/01/hey-siri.html" on this server. Reference #...

ecesena• 9 years ago

Is there any open source projects to do this? I mean just the triggering part, not the command recognition later.

MrBuddyCasino• 9 years ago

There are Snowboy and uSpeech on Github. Haven‘t used them yet.

woodson2• 9 years ago

Snowboy isn’t open source, though.

ChuckMcM• 9 years ago

Great explanation, this technique has a lot of applications for extracting event triggers out of audio streams. Even though I've trained my iPad repeatedly though it still is a bit to eager to answer to others who talk to it (either on purpose or by accident).

At some point I expect Apple to design an audio neural network processor to put on their CPU chips which will allow them to do both phrase recognition and highly accurate speaker dependent text to speech on their devices. It will be yet another way that people who don't build silicon won't be able to compete.

Hey Siri: An On-Device DNN-Powered Voice Trigger for Apple’s Personal Assistant

Related Stories