Using A Kalman Filter To Make Sense Of Noisy Data

http://seatgeek.com/blog/dev/using-a-kalman-filter-to-predict-ticket-prices

by jack7890 • 14 years ago

89 18 14 years ago

jack78902• 14 years ago

If there are other folks interested in using this approach, drop us a line. We're happy to try to help other startups make sense of their data.

jianshen• 14 years ago

I was working with Kalman filters for a hardware project and had serious trouble wrapping my head around the topic.

I ended up confusing a Kalman Filter with a plain old Low Pass Filter at first (and you can reduce a Kalman filter to that if you don't have enough inputs) but it really is quite a powerful tool.

It's neat to see it applied to a different problem that might make it easier for novices (like myself) to understand. Thanks for posting!

steverit• 14 years ago

Thanks for your kind words. As somewhat of a novice myself, I found the lack of documented real-world use cases frustrating, so I'm super happy to see this kind of feedback! I hope this can help you down the road should you make a second attempt at Kalman filtering.

psykotic• 14 years ago

Kalman filters are a simple idea when viewed from the right angle, at least for someone with a good grasp of linear algebra.

You have a linear dynamical system with normal-distributed dynamical noise. You then take linear measurements of that system over time, and again those measurements are subject to noise that is normally distributed. The question is then, given the series of measurements, what is the best estimate of the state of the dynamical system? When written out like this, it's just a least squares problem. What makes it efficiently solvable is that the matrix structure is block tridiagonal. If you apply the block version of Gaussian elimination to that block tridiagonal system, you get the Kalman filtering equations. That's all there is to it.

pork• 14 years ago

Any thoughts on writing up similar articles about more advanced nonlinear filters, like particle filters?

derekja• 14 years ago

Very nice post, thanks!

dredmorbius2• 14 years ago

Yet another blog which, if visited without JS enabled, displays a rather disconcertingly unrelated page (ticket search).

Noscript -> seatgeek.com -> Temp

(For anyone else confused by the apparent topic/content discrepancy).

And for those interested in a more readable technical explanation, there's the Wikipedia article (linked from the seatgeek blog): https://duckduckgo.com/lite

jack78902• 14 years ago

Good point, we will absolutely fix this.

If you don't mind me asking, why are you visiting websites without JS?

dredmorbius2• 14 years ago

Short version: principle of least privilege.

I crank down perms and open them as little as possible, for as brief a time as possible.

If your site's annoying enough to visit, I won't.

SpiderX• 14 years ago

Too many slow javascripts on too many pages, the web is ridiculous these days.

bhrgunatha• 14 years ago

There are a variety of reasons:

* Safety - not everyone protects their website from XSS attacks

* Speed - Downloading 30 or more script files from all over the planet slows down browsing

* Tracking/privacy - it doesn't make your browsing private, it cuts down the more obvious things - https://www.youtube.com/watch?v=f_f5wNw-2c0

* Principle - browsers render HTML - nobody should be forced to require javascript ONLY to read the content or just to have access to the correct page. If you have fancy effects or some additional functionality or if it's required to provide core functionality (e.g. maps) then fine, but JUST rendering text?

thespin• 14 years ago

Costs >> Benefits

sytelus• 14 years ago

If you want to understand Kalman filter in simplest term, check out the article at http://credentiality2.blogspot.com/2010/08/simple-kalman-fil...

jnazario• 14 years ago

great post, and thanks for sharing it. i came across the kalman filter after developing an algorithm to detect worm propagation. it applies equally well in that scenario, basically predictions.

essayist• 14 years ago

Useful.

I'd love to find FAAS - Kalman (and other) filtering as a service.

For instance: I run daily backups on various databases. I expect the backup size to increase roughly linearly, but I'm just going to look in on the backups at random, likely ignoring them for months at a time.

It'd be great to be able to run the backup size series through a filter that would alert me when something unexpected happened, e.g. slope changes significantly or some unusual step change.

Using A Kalman Filter To Make Sense Of Noisy Data

Related Stories