A Concrete Introduction to Probability (2018)

https://github.com/norvig/pytudes/blob/master/ipynb/Probability.ipynb

by tosh • 5 years ago

529 75 5 years ago

This is a cute introduction to probability. However, I would've loved to see some mention of dependent events and continuos probability.

scribu• 5 years ago

It does have a section on continuous probabilities at the end.

senthil_rajasek2• 5 years ago

Yes, In the appendix thanks.

cesarosum2• 5 years ago

I do think that there's some merit in sticking with probability on discrete spaces for a while. Once you start dealing with continuous spaces, soon you're talking measure theory and you can wade deep into the technical details and miss some understanding of what's going on. I go back and forth on this as I think it's largely down to the reader to figure out what works for them, but I think probability is one of those fields where developing intuition early on is a must if you want to go further.

beforeolives4• 5 years ago

The actual requirement for measure theory is overblown. As long as you've taken single and multivariable calculus, you can study continuous probability without any problems and without even knowing what measure theory is.

screye• 5 years ago

Shoutout for another great online MOOC : https://www.edx.org/course/probability-the-science-of-uncert... (It is the same as MIT OCW's 6.0.41)

I did it as preparation for my Masters and it was genuinely helpful. Would recommend it to everyone looking to do a prob 201 before taking advanced-ish courses.

beforeolives4• 5 years ago

I would recommend Harvard Stat 110 over MIT's probability courses - https://www.edx.org/course/introduction-to-probability (you can find full lectures on youtube and the book online)

someguy101010• 5 years ago

How come?

jackallis• 5 years ago

do you the link to youtube. Cant' find it; that why i gave up on the class.

SeaWhales1000• 5 years ago

I found the EdX course too rushed (assuming you don't pay for the verification to get lifetime access). I like Bliztstein, so I instead used his Youtube playlist [0]. It has his full lectures. Also, his book is free to view [1].

[0] https://youtube.com/playlist?list=PL2SOU6wwxB0uwwH80KTQ6ht66...

[1] https://drive.google.com/file/d/1VmkAAGOYCTORq1wxSQqy255qLJj...

xxdd8378yjk• 5 years ago

thanks

javier10e6• 5 years ago

I read portability and rush to the link waiting to hear about pip and all other mayhems and I landed on a math page. Boy I need my coffee.

sillysaurusx2• 5 years ago

Similarly, I thought "from fractions import Fraction" required a `pip install fractions`. But nope, turns out fractions is a built-in module I've never heard of. Neat!

beforeolives4• 5 years ago

Similarly, one of the most suprising things for me was that complex numbers are built into the language (no imports at all).

sillysaurusx2• 5 years ago

Be sure not to miss the other two probability notebooks:

- Probability, Paradox, and the Reasonable Person Principle https://github.com/norvig/pytudes/blob/master/ipynb/Probabil...

- Estimating Probabilities with Simulations https://github.com/norvig/pytudes/blob/master/ipynb/Probabil...

There are dozens of other notebooks on a variety of topics in the 'ipynb' folder: https://github.com/norvig/pytudes/tree/master/ipynb

cousin_it• 5 years ago

Nice! I had fun solving this problem in my head:

I have two children. At least one of them is a boy born on Tuesday. What is the probability that both children are boys?

An interesting thing about this problem is the unspoken assumption of what happens in other counterfactual worlds. If the person always answers the question "is one of your kids a boy born on Tuesday?" then the problem is solvable. But if a different family history would've caused the person to answer a different question ("born on a Monday" instead of Tuesday), then the answer would depend on the person's algorithm. Eliezer gave a dramatized explanation here: https://www.lesswrong.com/posts/Ti3Z7eZtud32LhGZT/my-bayesia...

Further on this path, there are seemingly basic questions that cause disagreement among actual statisticians. For example, see the voltmeter story in https://en.wikipedia.org/wiki/Likelihood_principle:

An engineer draws a random sample of electron tubes and measures their voltages. The measurements range from 75 to 99 Volts. A statistician computes the sample mean and a confidence interval for the true mean. Later the statistician discovers that the voltmeter reads only as far as 100 Volts, so technically, the population appears to be "censored". If the statistician is orthodox this necessitates a new analysis. However, the engineer says he has another meter reading to 1000 Volts, which he would have used if any voltage had been over 100. This is a relief to the statistician, because it means the population was effectively uncensored after all. But later, the statistician ascertains that the second meter was not working at the time of the measurements. The engineer informs the statistician that he would not have held up the original measurements until the second meter was fixed, and the statistician informs him that new measurements are required. The engineer is astounded: "Next you'll be asking about my oscilloscope!"

benlivengood• 5 years ago

Maybe I'm missing something about the voltmeter example. My assumption is that the 100-volt-maximum meter can distinguish between 100 volts and more than 100 volts, in which case there's no problem. If the voltmeter doesn't accurately indicate whether or not a measurement is outside of its range then the statistician is correct that everything should be re-measured.

Do some people think that the possibility of not being able to take accurate measurements is the same as not having taken accurate measurements?

EDIT: Maybe the ambiguity is in what the engineer would have recorded if finding a voltage >100 volts while the other meter was broken? It's like undefined behavior in programming; if you know your software will have undefined behavior when encountering certain data then you can't trust whether the output is valid unless there's independent confirmation that the data won't cause undefined behavior. If the statistician doesn't have certainty that the engineer will have defined behavior (e.g. say "I couldn't complete the measurements" vs. undefined behavior like writing down "99" or exploding) then they of course want to re-measure.

vlovich1232• 5 years ago

In the correct version of this story, the mathematician says "I have two children", and you ask, "Is at least one a boy?", and she answers "Yes". Then the probability is 1/3 that they are both boys.

I don’t understand this reasoning. If at least one is a boy, the only configurations I can think of is 1 boy 1 girl or 2 boys. Where does the 1/3 come from?

at_a_remove3• 5 years ago

I got reasonably far in stats -- up to a single grad level course.

My biggest problem with the teaching of math (and I can weigh in on this a bit because I spent over a decade as a private tutor of math) is that math is often not introduced in a concrete manner, as well as in a way such that its utility is obvious. These are two sides of the same coin.

You can see this problem in most Wikipedia math pages. I took more math than most physics majors and I find quite a lot of what I stumble across either baffling or of unknown utility. Set theory is a great instance of this. I rarely see multiple examples listed as it is explained, nor does anyone bother to tell me what it is for, other than to do more set theory.

Stats at least has the benefit of having to stick closer to applicability, I think.

nightski2• 5 years ago

The problem is sticking to applicability in probability/stats only gets you so far. I'm in that rut having a firm grasp of how it is applied but not being able to actually apply it in the real world because the real world scenarios are all different.

It's easy to make trivial mistakes without a firm grasp of the theoretical side in my opinion. It's important to understand the implications of the choices being made and the theoretical limitations of the models being created.

at_a_remove3• 5 years ago

I am not suggesting ditching all but applicability, I am suggesting that applicability be present. It's not an "either or."

pstuart• 5 years ago

math is often not introduced in a concrete manner, as well as in a way such that its utility is obvious

This cannot be emphasized enough. Most people won't be doing any higher order math in their lifetime -- we should be teaching math literacy and "real world" math for the masses rather then pretending that every student is going into physics.

NovemberWhiskey• 5 years ago

And we should be clear here that the bar for improvement is verrrrry low. Take a look at the questions asked in the National Financial Capability Study, and the results:

https://www.usfinancialcapability.org

The survey is composed of five really-quite-basic questions about interest rates, inflation and risk assessment.

As of 2018, 66% of survey participants get 3 or fewer of the questions right.

segmondy2• 5 years ago

It's difficult tho, the idea is the student should want to learn that they learn no matter how boring because the teacher/institution says so. I remember when I got introduced to matrix in high school, it felt pointless and stupid. Why would I want to transform matrix? I hated the entire thing, sure I could solve it but for what purpose? Then I learned I could use it for 3D engine, and I could load my object in a matrix and transform to move things around, that got my attention. However, a student that has no interest in 3D engines won't care. How then do you make it exciting? It's not enough to make the problem concrete, but the student will also need to be interested in the concrete application of the subject. Teaching is hard!

at_a_remove3• 5 years ago

Absolutely. As a private tutor, finding applicability and motivation for students was quite a challenge. I spent a lot of time reformulating problems in terms of things they might care about.

laichzeit0• 5 years ago

Maybe it was taught in the wrong order. Matrix multiplication makes perfect sense if you see (a) that linear transformations can be represented as matrices, I.e A = f, B = g and (b) you define matrix multiplication so that the composition of linear transformations I.e. f(g(x)) = ABx gives the same result. That’s a pretty cool idea, that you can represent a function by a matrix and that you can compose those functions and the composition is the same as multiplying the matrices together. So let’s say f(x) computes the derivative of a polynomial, and you have a matrix representation for that, say A. Now what if I want the second derivative? That would be f(f(x)) or just AAx.

I suppose I would tell a student that asks “why would I want to remember the stupid rule for multiplying matrices together in that way?” to try and figure out a rule for multiplying two matrices together such that you could represent functional composition by it, and they would self-discover the matrix multiplication rule and see why it would be useful to do it that way.

_carbyau_• 5 years ago

A good teacher can make a great difference but as you say, the pupil needs some motivation - some reason to want to learn it.

I took exactly one class of 2nd year physics and couldn't see the point of it. I walked out, quit physics altogether and came back the following year to do CS.

agumonkey• 5 years ago

AFAIK logo (the old turtle language) was born as a way to address this. Papert and his buddies wanted to turn thinking into semi tangible forms. More interactions, more inputs to feed your brain to think more.

Also, maybe it's only me, but mathematicians often forgot their own culture. Or maybe it's due to the iconic status of the field during early education, making people never explain why they do the things they do. When you read about history of mathematics you see that problems were 1) very practical ones 2) first tricks were very natural. It is not a thing from the gods.. at least it didn't start like this.. it condensed into a diamond over centuries of refinements. But if you don't show that to the crowd, you lose 90% of the audience. It's a pity.

anthony_r3• 5 years ago

This is a super common issue that people are raising over and over again, but there's already a solution for this: split teaching into technical and non-technical. In some countries universities are split along this axis.

Take statistics for example - most people that know something about statistics don't exactly know what a statistical space is (and it is a very precisely defined concept, embedded in the set theory). And that's fine, that's for the "pure" nerds. How to use it can be taught without defining the roots and proving theorems from the ground up. It is also how most of software development is done, few people out there that write code understand how CPUs fetch and execute instructions, talk to other perhipherials, how does malloc()_or sin() work, what is a page fault, or how to balance a red-black tree. Just use std::map or dict() or something, it just works :)

anon_tor_123453• 5 years ago

how CPUs fetch and execute instructions, talk to other perhipherials, how does malloc()_or sin() work, what is a page fault, or how to balance a red-black tree

literally every single one of these things is taught in an undergrad class and understanding of which is deemed important by the community - curricula get lots of input from industry partners. so you're not making a great case for why people don't need to know what a sigma algebra is...

Just use std::map or dict() or something, it just works :)

wouldn't it be swell if this is how we practiced medicine too? patient has early stages of atherosclerosis just do a triple-bypass or something, it just works.

thaumasiotes• 5 years ago

most people that know something about statistics don't exactly know what a statistical space is (and it is a very precisely defined concept, embedded in the set theory). And that's fine, that's for the "pure" nerds. How to use it can be taught without defining the roots and proving theorems from the ground up.

It's a popular theory, but then one day you read every recent biology paper and notice that only 2% of them are able to do statistics in a way that isn't total nonsense.

The only way for "learn how to use it, but not how it works" to work is if you get feedback whenever you make a mistake. That is not true when you use statistics.

madhadron• 5 years ago

The problem is that professional mathematicians have a common language that they all have learned which serves them well...and they're the ones writing the Wikipedia articles. Imagine if all the programming documentation you worked with was at the level of a 1980's first book on BASIC.

Set theory is the concrete starting point after that training, and the first step of concrete problems is to translate them into an abstract form of maps on sets. It's a decoupling. Instead of translating n techniques into m domains (n*m bits of work), you develop n techniques in terms of set theory, and translate m domains into set theory (n+m bits of work).

Physics majors largely use the same pieces of math on the same domains, so this decoupling doesn't make sense for them. Similarly, most domains carve off some piece of math and statistics and specialize it. But if you're writing a reference, whose specialty do you choose?

anon_tor_123453• 5 years ago

Imagine if all the programming documentation you worked with was at the level of a 1980's first book on BASIC.

mathematical maturity is the same as "code sense". when i started writing code a couple of years ago i would get cross-eyed reading large blocks of code i.e. i would get lost in the syntax and the abstractions and the idioms. at the same time i had pretty decent "mathematical maturity" i.e. i could read papers and textbooks pretty handily. comparing it seems obvious that formal mathematics, with its idioms, abstractions, and syntax is basically the same thing (without pushing the curry-howard isomorphism too far).

aaron-santos• 5 years ago

Most computer science students get a set theory introduction and construction of natural numbers, rationals, and reals. Those with an interest in statistics get a foundation laid in measure theory. For other domains, what are good places to look for set-theoretic approaches? A set-theoretic approach to calculus has me intrigued.

state_less• 5 years ago

I've found it helpful to use probabilities throughout my life. When picking a career, it's helpful to know the graduation rate and the likely salary if you do graduate to calculate the expected value of pursuing any particular career. I started playing poker and in order to be profitable, I learned the odds of various situations.

Sometimes folks will be afraid of things, even though the odds are they'll be okay. Knowing the odds can give you courage where others find it difficult to go beyond fear.

I've been a fan of Peter's AI book(s) since my university coursework. Glad he's introduce so many of us to these helpful ideas. Basic ideas can go a long way if you learn where they apply to your life.

Topgamer7• 5 years ago

I graduated with a computer science major, and I've taking courses on linear algebra. It didn't really hit me about practical applications of linear algebra until I watched a video on youtube by Stuff Made Here[1].

Which is funny because I totally recognize now how it can be applied, but moving from system of equations on paper to real world applications never really clicked in usefulness until it was explained in that video.

1: https://www.youtube.com/watch?v=myO8fxhDRW0.

mjreacher• 5 years ago

I believe this is due to historical reasons, in particular due to the Bourbaki and the French school of mathematics where abstraction was heavily prized. On the contrary mathematics in the Soviet Union had more of a focus on intuition and geometric grounding for mathematics, however history played its course and mathematics is taught more towards the Bourbaki style these days, however it did bring out gems like [1] which give an opposite, albeit extreme, view of how mathematics should be taught.

[1] https://www.uni-muenster.de/Physik.TP/~munsteg/arnold.html

bigdict• 5 years ago

On the contrary mathematics in the Soviet Union had more of a focus on intuition and geometric grounding for mathematics

A lot of it comes from mathematics always being taught in conjunction with physics. Not sure what the roots of that are.

nanidin• 5 years ago

The beauty of Wikipedia is that you have the power to improve and make changes as you see fit.

I know it’s a meme that edits to Wikipedia are fraught with editor politics and reversion of good faith changes made by users, but that hasn’t been the case for me lately. If you have something to add or improve, don’t let anything hold you back!

skipants• 5 years ago

I suppose it's tough to balance detail with complexity when it comes to a general wiki like Wikipedia. Especially with math and its domain language.

I believe that the issue you're outlining was the precursor to simple.wikipedia.org. Sorry... I couldn't come up with a math example on the spot but here's a good CompSci example:

https://simple.wikipedia.org/wiki/Dijkstra's_algorithm

joppy• 5 years ago

I agree with you on the incomprehensibility of mathematics Wikipedia pages, even as a working mathematician I can really only read the pages about content I already know in-depth. But Wikipedia (and many other reference-type sources) are not the place I would go to learn a new topic in maths, I would always prefer to read a short book aimed at the correct level of knowledge, or a set of course notes (many of which are available online). I don’t think Wikipedia is representative of the teaching of maths at all.

Many of the courses on very abstract content I’ve taken (from other people, in person) has been very concretely introduced, or at least the course has been run with a 50/50 split on the abstract (general theory) and the concrete (solving problems).

Tainnor2• 5 years ago

You can see this problem in most Wikipedia math pages.

This gets mentioned so often that I feel it hints at a deep misunderstanding at what Wikipedia is or is supposed to be.

Wikipedia is an encyclopedia. It's not a teaching resource, and it shouldn't be. (Nor, for that matter, is it a primary source, which makes it useless as a reference if you're writing a paper, unless you're specifically writing about Wikipedia.)

Mathematics is such a ubiquitous language. Different people will require different kinds of mathematics, for vastly different purposes and in various levels of depth and formality, there's no way this can be unified into a single-size-fits-all resource. If you want to learn mathematics, now more than ever there is such an amazing breadth of text books, blog posts, lecture videos, online communities and so on that you can use depending on your very specific needs. I don't know why people go and try reading up on mathematics from Wikipedia, out of all places.

jfengel• 5 years ago

Wikipedia is an encyclopedia. It's not a teaching resource

Isn't an encyclopedia a teaching resource? Etymology isn't destiny, but the "pedia" part is from Greek παιδεύω, meaning "child rearing". I'd interpret that as meaning "instructional".

The "encyclo" part means "circle", which in this case alludes to "comprehensive". As in, it's not just one instructional resource, but a guide to everything.

The word will grow and change over time, but I think most encyclopedias are still used as teaching resources. That doesn't mean Wikipedia has to be, but if it really wants to be an encyclopedia, I think it means the opposite of "shouldn't be a teaching resource".

coliveira• 5 years ago

The role of university is exactly to teach you the language of math, or at least enough that you can get more by yourself. If your professors spent all the time talking only about concrete examples, you would be completely at a loss about why math is like it is, from the point of view of the untrained person it would be just cargo cult. So, math is hard, but the university has the responsibility to teach it (at least the introductory material) to as many people as possible.

contravariant• 5 years ago

If you define the probability function you'll need set theory within about 5 minutes.

If you want to define the probability measure you'll need to be pretty comfortable with set theory.

analog31• 5 years ago

I'm the odd counterexample, where proofs were what made math come alive for me. I suppose you could say that the utility of math is in furnishing tools for proofs, but that's probably not what most people mean. ;-)

Utility was for physics, which I also majored in.

jypepin• 5 years ago

Everything I see from Peter Norvig is just always so incredibly well written and coded. I'd love to know what it's like to work for him.

Every year looking at his solutions for advent of code [0] brings just so much learnings. Strongly recommend.

[0] https://github.com/norvig/pytudes/blob/master/ipynb/Advent%2...

mikevin• 5 years ago

Everything I see from Peter Norvig is just always so incredibly well written and coded. ....

I feel his skill of dividing a problem into small pieces and expressing them in code in a natural way is unparalleled. Most books/blogs/articles I see often focus on one of two patterns.

The most frequent one is pulling in some dependencies and using a high level API, essentially skipping any real problem solving. Great when you just need a problem solved and are familiar with some framework/library but not that great for learning to program or problem solving

The other one is a deep dive into a data structure, algorithm or performance tuning. This is great when studying theory or optimizing. These articles are more interesting but I haven't encountered many people who are in a position where this is relevant to day to day work.

The missing pattern is one where Peters' work shines. The parts in between. All the libraries that are used in the first example I described are the result of someone taking the building blocks that result from the second example and applying them to a real world problem. Peter Norvig is my go to recommendation when someone is interested in becoming better at solving day to day problems because of this.

the_decider• 5 years ago

The following data science book also does a great job of balancing problem solving with underlying theory. https://www.manning.com/books/data-science-bookcamp And it starts with sample-space probability problems in Python, much like Peter’s tutorial.

abecedarius• 5 years ago

The times I had a peek behind the curtain, he didn't expect to stop with the first version, even though it'd be good.

(Maybe those Advent of Code solutions are the first working draft, I don't know.)

khazhoux• 5 years ago

I'm always amazed at how Peter Norvig continues to create the kind of content that a top grad student would do, even as his career and rank in the industry has skyrocketed.

Back in university and grad school I would write tutorials and post them online (and still get thanks from random people many many years later). I would explore random interesting subjects and dive deep. I would constantly publish code and demos, etc. As my career grew, one of one these started to fall off. I look at my peers and it's the same story: they were all vibrant hot-shots in their early-mid 20s, and now are just weighed down by the teams and projects they manage.

Peter is an inspiration. I will ponder this...

ericol• 5 years ago

"one of one" or "one by one"?

da39a3ee2• 5 years ago

This is some extremely stylish expository python (as most of the other comments are saying).

spoonjim2• 5 years ago

I've never seen Peter Norvig choose anything but the most elegant and perfect data model for the problem at hand. I wonder what it's like to be in his brain.

867-5309• 5 years ago

any recommendations for probability as a foundation for cryptography?

A Concrete Introduction to Probability (2018)

Related Stories