How would you use an accurate Natural Language Understanding (NLU) API?

by apetresc • 14 years ago

51 38 14 years ago

rhizome• 14 years ago

How accurate? Accuracy is a value, not a state.

apetresc2• 14 years ago

Our technology was initially designed to be used with external ASR. For sentences within our 18 domains, where the ASR is correct, we see accuracy (defined by mapping into the current domain and action) around 90%. The reality of course though is ASR is sometimes wrong, which does decrease our accuracy in practice.

The best direct demonstration of practical accuracy is just try out the app ;)

hooande2• 14 years ago

I would use a good natural language understanding api to do basic research, ideally monitoring many different feeds for topics of interest. It could also be a good way to monitor trends or competitors.

I can't think of many uses cases for natural language as an interface to a traditional website. If there's something that's more convenient to access with text/speech, the user can just use set commands.

On the topic of building a personal assistant app, from what I've seen the real value lies in the act of interactive scheduling. You can't say "schedule a meeting with george" without checking with george's schedule to make sure he's available, and potentially re-scheduling the event at a time of mutual availability. Of course this goes far beyond the scope of understanding language, but I think this is a big part of what will differentiate a good personal assistant app from Siri.

EDIT: Someone on the thread brought up customer service, which is also a great opportunity. Interacting with a company's service representatives is a huge time sink for people everywhere, especially on the phone. Anything that lets people spend less time on hold is a huge win for mankind.

drewch• 14 years ago

That's a pretty sweet idea. I wonder how difficult it is to set up sharing of calendars, and the permissions that are involved in it.

icebraining• 14 years ago

Google Calendar lets you share only the Free/Busy times of your schedule, without revealing the details of each event.

Ideally, of course, your scheduler should be able to "propose" some times to George's, and let it decide and reply back with the chosen one.

juharris• 14 years ago

Actually, I've worked for a company that specializes in finding topics based on your interests. http://primal.com/

vhf2• 14 years ago

I cannot answer your question -- I have no idea. (edited : well, why not for "customer service", along with the traditional FAQ ? Or use it instead of the useless "solve this problem" useless multiple choice walkthrough in Windows or such ? )

But I upvoted because I really liked to discover this company, what it does, and more importantly how it does it. Thanks for the clear examples and output provided !

jpantony7• 14 years ago

Thank you for your up-vote :)

languagehacker3• 14 years ago

From what I read, I don't really see a robust NLU engine here. I see a probabilistic transformation engine (maybe bootstrapped with some linguistic data structures) that identifies 7 buckets and then rephrases the query or request into a format better suited to achieve the desired result in that engine.

This isn't too far from saying that you have a search engine that "knows" how to answer questions, because you have hard-coded it to omit question words and transform question phrases into something declarative in hopes of finding the best match (e.g. "What is the largest mountain in Asia?" --> "The largest mountain in Asia", and then list the top search result from Google).

There's this tremendous amount of pragmatism that goes into some of the example questions, too. If I ask someone if they can recommend a good sushi place, personally, I'm probably more interested in the best sushi hit they've ever had than whether then place I can walk to from work is decent. You're going to have to make a lot of assumptions that you cannot adequately contextualize to take a broad question like that and say "Well, here are two within two blocks of you." I might actually be ranking proximity lower than, say, quality, popularity, or memorability. By hard-coding such assumptions into your reasoning, you're not in the trade of understanding natural language; you're in the trade of doing exactly what Bing does now, at a smaller scale, with a few more buckets.

If you're really into building NLU, I have a few somewhat snarky suggestions for you: 1) Swing by a cryogenics facility and check back in with us in 20-30 years. Repeat until cognitive scientists have figured out a way to transform human thought into something that can be quantified and transformed into words.

2) Do some reading in cognitive science to understand the immensely difficult problem it is to quantify meaning, both within the context of a discourse as well as discourse-agnostic (archetypal meaning versus contextualized meaning). Look at some charts, and realize that word meaning is not only relative to other words in the same discourse, but other words in shared classes.

3) Read up on Grice, and understand that, despite his philosophical approach to linguistics, his points on implicature and the value of what goes unsaid in a conversation lend as much to context as what goes said. Then try to sit back and figure out how to program and analyze what's going unsaid in a search query.

What I'm trying to say is that I would do nothing with this API because I can't trust it to deliver on its very lofty promises. Natural language understanding is essentially a pipe dream in the current state of computer science and bioinformatics. Word meaning is so idiosyncratic that it is a miracle that humans can generally agree on what a word represents; most of these are extremely fuzzy, and cannot be divorced from the contexts in which they are uttered. It's far too difficult to contextualize even a sentence outside of a pure information retrieval standpoint.

I appreciate what you're aiming at here, but I doubt you'll ever be able to succeed at it. That being said, if you rephrased it into something like a decision engine, I probably wouldn't be so irritated by it, and might even give it a shot.

hooande2• 14 years ago

They didn't say that they invented general intelligence. This is clearly providing a natural language interface to an existing api. It doesn't take degrees in neuroscience and philosophy to do that.

They've clearly spent a lot of time to build a working product. You reacted to only a few words from the page title. It's easy to read books on the theory of intelligence, much harder to make a contribution by trying to solve a real problem.

ChuckMcM• 14 years ago

I am sure they are earnest in their efforts, I chuckled when I read "Some of our engineers have been working on this problem for two years, ..." because I know engineers who have been working on that problem for 20 years and they still haven't got it right. One of them (at Intel in the '80s) had a poster that said "Time flies like an Arrow, Fruit Flies like a Banana." which captured the challenge for him. One of the founders of Android, Rich Miner, was an Amiga guy that I knew peripherally through the developers network. He did a company called WildFire which was an assistant that would be on the phone with you that you could ask things of or make requests too. Also felt the problem space was huge, then there were no less than three Phd's at Google who gave a tech talk on the Goog-411 project where they were listening to people ask for phone numbers and trying to figure out what number they were looking for. (one domain of questions so almost like cheating).

Perhaps they just write optimistically, I am excited to seem them working on the problem (I'd suggest they search blekko since we can add slashtags and do better domain search but that is just me). Having seen the folks who have run at this problem and been knocked on their backsides I have tremendous respect for taking on a really really hard problem. Can't wait for the beta.

joe_the_user• 14 years ago

Actually, saying you have "an accurate Natural Language Understanding" api pretty much is saying you've invented general intelligence.

jpantony7• 14 years ago

I feel that you have misunderstood what we have here. I agree that "true" understanding requires is almost an impossible problem to solve due to the fact that it requires context that is only available to the individual. However that level of understanding is not really necessary for applications today. What we are aiming to provide is a system that will allow developers to have natural language queries as their input and have all the pieces of information required for their application extracted. Secondly, the name "natural language understanding" is given to a large area of computer science ranging from sentiment analysis to information extraction. Many of the areas do not even involve deep understanding of contextual meaning in the way you describe. That however does not diminish the importance of any of these fields nor does it eliminate them from the the umbrella of natural language understanding.

languagehacker3• 14 years ago

You'd be a whole lot better of saying you've written a natural-language interface to a number of search domains. You're promising a lot by claiming to have an API that not only accomplishes understanding (which, contrary to your claim, is reserved for much deeper and more structured inferences than what you're providing here -- are you transforming sentences into a formal language before feeding it to epicurious, or are you just matching terms against a list of known food items?), but that it is accurate in its output. I still think that the claims of NLU are nothing short of hubris, and that the product would be more compelling if it didn't advertise something so unattainable.

JabavuAdams• 14 years ago

Serious question: why do we expect machines to do the right thing with a single isolated query, rather than taking part in a conversation?

"Ok, you want Sushi. What places have you been to? Is there anything in particular you're looking for tonight?"

fatbat3• 14 years ago

I am interested in the calendar + reminder domain of it. More so if it works offline (ie- Soundhound/Shazam app).

Can you elaborate on the calendar domain?

pdat• 14 years ago

hi,

Well currently we support 4 actions within calendar, "CREATE_EVENT", "SEARCH_CALENDAR", "MODIFY_EVENT" and "DELETE_EVENT". There are a number of different entities including "dateRange, lengthOfTime, timeRange, date, time, meetingTopic, contactName, friendOrRelative, location, originalTime, originalDate, originalLocation, replacementTime, replacementDate" and a few others I'm sure I've missed.

For reminder we support "CREATE_REMINDER" and "SEARCH_REMINDER", we also support the entities "message", "time" and "arrivalLocation" and "leaveLocation". Does this answer your question or did you want a more specific answer? If you want more information feel free to email us directly :) .

fatbat3• 14 years ago

Hi! Thanks for the reply! One more quick one. :) Are the "time" entities relative or absolute time or either?

ckrailo• 14 years ago

I wish there was something I could hook up to my electronic life (calendars, search engines, knowledge stores, bug tracking, etc) and send IMs to from all devices. Like Siri, but more useful and more connected. A general, all around AI instead of a glorified Google search with speech.

Example commands I wish I could do (some would require me hosting a server to communicate with my home, but I'm okay with this):

"Lock my doors." "I'm going home." (Notify people, set AC, bring up a map, etc.) "What's the weather tonight?" "Write 'happy birthday!' on my friends' Facebook walls each time it's their birthday." "Ask my girlfriend what's for dinner when she gets on Skype." ...and so on...

That said, I'm super excited about this and would love to see Maluuba's work accessible on all devices so I can just IM the service instead of using a phone when I'm at home.

jpantony7• 14 years ago

I'm excited to see someone talking about a single unified nexus for everything. That is part of the future vision of our company, and smart homes are an example of where we want to go in the next few years :)

StavrosK4• 14 years ago

I had this idea not too long ago. An assistant as a service, that would be accessible on IM and would do increasingly more things, starting with handling the scheduling of appointments between two people completely automatically (it would email them and ask them if they're available, of course).

I made a small prototype, but it was way too limited, as NLP is a huge problem. Even a simple TODO list/alerts/meeting scheduling app with calendar integration was very, very complicated.

I'm pretty sure lots of people would pay for it, though.

caublestone• 14 years ago

https://lizibot.com/

It's pretty cool. Try it out.

flipstewart• 14 years ago

Siri satisfies all my query related NLU needs, BUT...

There are other applications for NLU that I'd use daily were they available. A prime example is taking text and producing a tl;dr version for me.

skrebbel• 14 years ago

I wouldn't even consider it when "all" it understands is English. The world is a big place!

chezmo• 14 years ago

As written in the article, the technology is meant to understand short queries, in other words short commands.

I guess everybody agrees, nowadays entering commands to any kind of machine is still much more accurate and quicker if you use your hands for doing so.

This said, i think the technologies is in particular useful in situations in which you are not able to use your hands. This could be for example because you are busy working or you are handicapped. So one idea which comes to my mind would be an assistant for handicapped people, e.g. a device which can be used to control an electric wheelchair.

How would you use an accurate Natural Language Understanding (NLU) API?

Related Stories