Hawking looking for technical help to maintain and improve his speech system

http://www.theverge.com/2011/12/29/2668408/stephen-hawking-technical-assistant-speech-system

by shawndumas • 15 years ago

67 22 15 years ago

theverge.com

shawndumas• 15 years ago

http://www.hawking.org.uk/index.php/component/content/articl...

MPSimmons• 15 years ago

There is no amount of money that should be considered "too much" to keep this man as productive as he can be.

Seriously. If it costs 10 million pounds a year to run that chair, it's worth it, and the rest of humanity needs to chip in to help.

dextorious• 15 years ago

But why? You really believe he is making important scientific contributions NOW? I think it's more of sustained hype from his earlier contributions, plus a guy in a chair makes for a compelling media story.

That said, I'm all for social health and providing for the needing and handicapped, but that's regardless of if they are "worth it" or not.

AndyJPartridge• 15 years ago

Speaking purely about his brain, and nothing else, I'd wager money on him producing something useful over many other brains.

noonespecial• 15 years ago

He's well know and respected for science. And I mean Michael Jordan well known. Science needs every hero it can get right now. As a spokesman alone he's worth every penny. As a role model...

chime6• 15 years ago

I've been working on http://ktype.net for 18 months now. KType is an iPad app to improve communication for individuals with speech & motor disabilities. When I talk to people, I actually mention Hawking as the most famous use-case and that while he can have the best engineers and computer scientists work on his device, very few devices are available for my late grandpa who had stroke or cousin who is paralyzed from an accident and can't talk. I'm making KType for them. Kinda funny to see Hawking looking for assistance with improving his systems. Wonder how I could show him a demo of KType :)

My end goal is to turn KType into an integrated box like Hawking probably has but make it easy enough that anyone can set it up. I've seen old specs of his system but that was long ago. I wonder how good the word completion/prediction is. I built my own standalone prediction algo based on data-mining 10m tweets and can search through 1m words and phrases on the iPad in real-time as the user types (you can see that in the demo video). Here's how I mined the Twitter data: http://ktype.net/wiki/research:articles:progress_20110209

If anyone's interested, I could write how my algo works. I'm quite certain that it is significantly better than typical smartphone auto-suggests and it was quite an engineering challenge to make it work on the iPad (bit-counting and bloom filters and tons of optimizations). I've thought about making it a standard C-library that you call from your code - pass it a 1 to 3 word phrase and it will return 3-7 potential word matches.

z0r• 15 years ago

I'm interested in this kind of algorithm on principle, and analysis of successfully hurdled engineering problems are probably of interest to most readers of hacker news. Please elaborate!

chime6• 15 years ago

I think I posted a few of these ( http://ktype.net/wiki/ ) on HN as I wrote them but barely got any votes so I wasn't sure if HN was the best avenue for it. When I get some time, I will do a write up on how I built the suggestion engine. Here's the gist:

* Built a list of 1m words and phrases (1-4 ngrams) and gave each a weight based on how many different people mentioned it and how often: http://ktype.net/wiki/research:articles:progress_20110209

* Sort this list in the following order: first-letter ASC, word length DESC, weight DESC. About 12MB in size.

* For each word/phrase in the sorted list, compute a 4-byte (32 bits) value as follows: First 26 bits = 0s and 1s to indicate which letter is present in the word/phrase. So CAT = 101...1000000 (A = 1, C = 1, T = 1, everything else = 0). Then normalize the weight and store it in the last 6 bits. So that's 1m x 32bits = 4MB of precomputed data.

* Build an index containing start-end position of all words of same length that begin with the same letter. Words starting with 'A' of length 20: position 1-200. Words starting with 'A' of length 19: position 201:543. And so on. About 32KB in size.

* Upon app init, load the 12MB word list, 4MB precomputed data, and 32KB of index. Takes 10s on iPad1, 5s on iPad2.

* When user types 'CAR' (3 characters), find the range of all words of length 3-9 that begin with C. Start position P1 = Start position of 'C' of length 9. End position P2 = End position of 'C' of length 3. Everything in between P1 and P2 is fair game for suggestion. Sometimes this can be as small as 100 words (e.g. user typed XP) or as many as 150000 (user typed ENT).

* Now, create a 26bit value from what the user typed, like we did with all the words in our 1m long list. So CAR = 101......100000000. Here's the fun part: For each precomputed 26bit value, AND it with the CAR-bits and count the number of 1s in the result. This tells us how many letters that the user typed, are present in each of the 3-9 length word/phrases that begins with a C. Since this is integer math, even looping through all 1m entries is near instant on the iPad. If least 2 out of 3 characters that the user typed are present in a word/phrase, go to next step.

* So user typed 'CAR' and we now have a list of words like 'CATARACTS', 'CRIMSON', 'CARBON, 'CRAYON', 'CHARM', 'CARE', 'CAR', 'CRY' etc. How do we know which one the user is trying to type? Well, depends on which word fits best and how high the weight is. By 'fit', I mean 'CAR' => 'CAMERA' is a better fit than 'CAR' => 'CRAYON' because the letters 'CAR' are in the correct order in the word 'CAMERA' but 'A' and 'R' are flipped in 'CRAYON'.

* If we do a distance calculation between the words at this point, it will just match the shortest words that have similar letters. So CAR will match CAR, CRY, CARE and not match CATARACT at all. We want it to match longer words also because those are much harder to spell out. E.g. PMGN should match POMEGRANATE and PYGMALION better than PONG or PIGEON. Spell-checks work by measuring distance between words - http://norvig.com/spell-correct.html but we're not making spell-check. We're making a word-predictor. So we need to find what words fit CAR and PMGN and how well.

* Turns out it's easy. To find out the fit, simply remove all letters in the word/phrase that the user did not type. So CAMERA => 'CARA'. 'CRAYON' => 'CRA'. If user typed PMGN, then POMEGRANATE will become PMGN, PIGEON will become PGN. Now take the http://en.wikipedia.org/wiki/Levenshtein_distance of what the user typed e.g. 'CAR' and each resulting phrase: CARA, CRA, CAAR, CR, CA etc. Do some funky math with the resulting distance x word length x word weight x exact-match-ness and give it a score. If score > min-fuzzy-value, return it. Else word does not match. The min-fuzzy-value is calculated based on how many words begin with the letter, have the same length etc. If user typed 'ENT', we want min-fuzzy-value to be quite high, if they typed 'QLX', then we want a really low min-fuzzy-value or else there won't be any positive matches.

* Once you have at most 100 (or 1000 depending on processor speed) matches that have a high enough score, sort descending by score and return top 3-7 as configured by user.

This is just the algo for single word match. The multi-word match is similar but has a lot more conditions to speed things up. But it works significantly better than single word match because it really narrows down the potential matches. Typing 'T' can mean any of the 20,000 words of 2-5 in length. But typing 'T' after 'EIFFEL' really narrows it down.

While my approach to word suggestion may seem pretty obvious, what I didn't mention were the 50+ approaches that did not work because of performance, quality, or stability issues. I'd love to hear any questions about it and I can provide my ObjC code (after I clean it up) if someone wants to build on it.

Please note I wrote all of the above off the top of my head and not as a detailed research paper. I left out lots of minor optimizations and kludges. Also I'm pretty sure I can describe the indexing and bit-counting better given some free time.

LearnYouALisp• 15 years ago

Tweets? At first that seemed like a little biased prediction set, but on the other hand people who have to make such an effort to communicate will want write few words. However, writing "Tweets" takes almost no effort at all, so it is interesting to consider.

chime6• 15 years ago

I used tweets because they're informal and mention a lot of "what to do" and "how I feel" topics, which I think is very important for basic communication, especially for individuals with multiple disabilities. A bonus is that I get a lot of pop-cultural lingo (e.g. "Harry Potter", "Hulk SMASH") that makes my suggestions more fitting for informal communication than plain old dictionary lookups. And you are right about the tacitness of tweets - helps in building a list of common phrases like "on my way to" and "by the way."

fl3tch• 15 years ago

Thank you for doing something valuable for the world. I see too many people building trivial things and calling themselves innovators.

_rs• 15 years ago

I'd be interested. Also worth noting that the 9 mil tweets link seems to 404, and looking through their site it appears to no longer be available. Any chance you could suggest other resources for this type of data? Asking that you host the file wouldn't be fair, as I assume the bandwidth would kill you easily.

chime6• 15 years ago

I think there was a licensing issue with the original data so they probably pulled it.

joe_the_user2• 15 years ago

As far as I can tell, the Hawking example shows that even with pretty much the best possible resources available, just about every special-use system for the disabled is more or less doomed to become something like a stovepipe system; a system that's a wonder when its created but by its very nature of having limit audience, subject to limited maintenance after a certain period - maintenance that usually can't prioritize upgrading for the next version of whatever OS it runs on. Steve Hawking, the world's best known quadriplegic, is still running WINDOWS XP.

Given this, the choice of iOS as your OS seems kind of unfortunate. Why choose an OS where the hardware makes a point of being a black box and where the support for, say, an old version ten years from now seems rather doubtful.

And thing with Hawking is that his system work. It's a virtual certainty he has no interest in your system 'cause getting used a system is lifetime affair. But someone will be used to your system. Will you be happy explaining to them they have to upgrade or even abandon their system because backward compatibility is not a high priority for Apple?

chime6• 15 years ago

> Given this, the choice of iOS as your OS seems kind of unfortunate. Why choose an OS where the hardware makes a point of being a black box and where the support for, say, an old version ten years from now seems rather doubtful.

KType is almost entirely PhoneGap (HTML5/CSS3/CoffeeScript) except for the prediction and TTS libraries. I could probably make an Android version in a week and plan on doing so in the near future. The reason I chose iPad is the ubiquity and stability. The best part about using tablets with PhoneGap is that the hardware/OS completely gets out of the way of the user. KType is the only interface that the end-user sees and interacts with. And as long as KType does not change drastically, users will not be inconvenienced.

angstrom• 15 years ago

Probably too limited, but a former colleague works for the company that's developing this:

http://www.think-a-move.com/tongue.html

gexla• 15 years ago

Oh, the fun I could have with that job. First prank, Stephen Hawking sounding like Scooby Doo. Or how about a plugin which would translate his speech to jive? Or maybe Darth Vader for the Star War fans?

frabcus• 15 years ago

Fifteen years ago, I had an interview for this job. This was before the Internet was popular, and it was advertised in The Guardian.

Worth applying just for the interview, to have tea and a personal chat with Stephen Hawking.

One of the questions was "You're in the US, and the Professor's screen shatters. It's Thanksgiving Day, and no shops are open. What do you do?"

We quickly realised that I wasn't good enough at electronics to do the job!

I found it hard talking to Hawking, as I didn't know what to do while he is composing his sentences. It seems rude to look at his screen, or out the window, or to stare at him.

Hawking looking for technical help to maintain and improve his speech system

Related Stories