Information Theory, Inference, and Learning Algorithms (2003)

https://inference.org.uk/itila/book.html

by malshe • 3 years ago

237 30 3 years ago

inference.org.uk

ml_basics4• 3 years ago

Great book by a great teacher. I read this during my masters and was my route into machine learning.

David did many great things but sadly passed away too young in 2016, surely would have continued to have a great positive impact in the world had he lived longer. His other projects included the sustainable energy book https://www.withouthotair.com/ and information theory inspired keyboard software for people with severe disabilities https://www.bltt.org/software/dasher/

I had the honour of being a PhD student in Cambridge during his last year of life, and saw him give talks a few times in our lab. It's truly a remarkable experience to see someone who is a passionate explainer in action. Like how I imagine it must have been to see Feynman in the flesh.

This is a great short video to get a feel for his style of explaining: https://youtu.be/UR8wRSp2IXs

(Writing this post feels wrong because he's had such a positive impact on my life and those of many others, and this post definitely can't do it justice)

azalemeth2• 3 years ago

His books are fantastic and his writing is captivating. It's such a shame he passed -- cancer is a bitch, but the blog post about the radiators in the hospice that he died in are one of the most heart rending pieces of writing I've come across from a mathematician.

ml_basics4• 3 years ago

Is this the post you mean? http://itila.blogspot.com/2016/04/appendix-two-open-letter-t...

malshe• 3 years ago

Thanks for sharing your experience. The video link is fantastic!

sideshowb• 3 years ago

As a follow on from Sustaianble Energy he also drove the original version of the my2050 game, which communicates the tradeoffs we need to make in decarbonizing to the general public very well. More recent update here https://my2050.beis.gov.uk/?levers=433233211331111

I've taken this as inspiration for my own transport models, though I have a model with some sliders to drag around I don't have a flashy game yet!

jll29• 3 years ago

A great communicator and a kind man, indeed.

It's been a privilege to see him lecturing at the Cavendish in 2002.

Here's his last video interview: https://www.youtube.com/watch?v=sCyidsxIDtQ

diego898• 3 years ago

I absolutely adore this book and its approach to information theory.

I will also note the author made available the (non compilable) latex source [1] of the text for easy searching!

[1]http://www.inference.org.uk/mackay/itprnn/booktext.txt

dang2• 3 years ago

Information Theory, Inference, and Learning Algorithms (2003) [pdf] - https://news.ycombinator.com/item?id=11289488 - March 2016 (9 comments)

Information Theory, Inference, and Learning Algorithms (free ebook edition) - https://news.ycombinator.com/item?id=852562 - Sept 2009 (21 comments)

p1esk• 3 years ago

I remember him (David Mackay) leaving comments here on HN, but I don’t remember the username. I wonder if you can find them somehow.

ml_basics4• 3 years ago

If anyone can find this I'd be very grateful, would be cool to read his comments

dang2• 3 years ago

I did some searches but couldn't find anything. If someone else does and the thread is closed for replying, let me know at hn@ycombinator.com and I'll reopen it.

ckrapu• 3 years ago

This book shows up every year or so on HN and I will never stop upvoting it. My favorite textbook of all time, along with my favorite quote:

“Marginalization is a macho activity, appropriate only for those fluent in definite integration”

agileAlligator• 3 years ago

"Exact marginalization over continuous parameters is a macho activity enjoyed by those who are fluent in definite integration."

http://www.inference.org.uk/itprnn/book.pdf#page=331

a-dub• 3 years ago

one of my favorites (even though the pdf is free i keep a copy of the book). very clear introductory chapters in probability leading into a really nice blend of applied math and signal processing, finishing out with neural networks.

his father published was something of a figure in the fields of perception and cognition as well.

mindcrime2• 3 years ago

There's also a nice set of recorded lectures available by Professor MacKay covering the core material from the book, available on videolectures.net.

http://videolectures.net/course_information_theory_pattern_r...

i_like_apis• 3 years ago

Fantastic. Thank you for sharing this!

noud• 3 years ago

"Your browser does not support playback of available video formats. Please install Adobe Flash player or upgrade to a more modern browser."

...Okay?

I found the videos on youtube as well: https://www.youtube.com/watch?v=BCiZc0n6COY&list=PLruBu5BI5n...

scythmic_waves• 3 years ago

From lecture 1 [1]:

Ok, so, we have an answer of 1000 plus or minus 30 thanks to "Mr. Binomial", the inventor of the distribution that bears his name.

Lol, this is great.

[1] https://www.youtube.com/watch?v=BCiZc0n6COY&t=971s

mindcrime2• 3 years ago

Huh. Weird. I'm using Firefox 109.0 on Linux and the videos on videolectures.net play fine for me.

ml_basics4• 3 years ago

Yes these are so good! Highly recommend.

smath2• 3 years ago

I attended an amazing talk by David circa 2006 at Princeton where he demoed his probabilistic text entry system called Dasher [1,3]. I think it was made into an iOS app later.

I also fondly recall his videos on Gaussian processes - best explanation on GPs I ever came across. [2]

[1] https://www.inference.org.uk/dasher/

[2] https://www.youtube.com/watch?v=NegVuuHwa8Q

[3] https://dasher.acecentre.net/about/

EDIT: Fixed the youtube link in [2], added [3].

zzz95• 3 years ago

Would love to see the GP video, but [2] is points to an unrelated video.

smath2• 3 years ago

Fixed

Tomte• 3 years ago

The book compares quite well to Harry Potter, unless you need the Latin translation: https://www.inference.org.uk/mackay/itila/Potter.html

Schiphol• 3 years ago

Cybernetics is alive the same way punk is alive: people say it's dead, but then there are punk bands and punk records and just like a truckload of punks. This book, like those written by his dad, is cybernetic theory in everything but name. I absolutely adore it.

sriram_malhar• 3 years ago

His books have taught me soooo much; in addition to information theory, I have learnt many tricks of LaTeX typesetting (the info theory book's LaTeX source is open) and how to estimate (from his other book on sustainable energy)

David was a fantastic ultimate frisbee player. He taught me how to properly throw forehand and overhand.

wiz21c• 3 years ago

From the preface:

"My work has depended on the generosity of free software authors. I wrote the book in LATEX 2ε. Three cheers for Donald Knuth and Leslie Lamport! Our computers run the GNU/Linux operating system. I use emacs, perl, and gnuplot every day. Thank you Richard Stallman, thank you Linus Torvalds, thank you everyone"

optimalsolver• 3 years ago

Was the David MacKay on Stargate: Atlantis named after the author?

Information Theory, Inference, and Learning Algorithms (2003)

Related Stories