ivan_ah

Or with and .assign:

    (
        purchases.loc[
            lambda x: x["amount"] < 10 * x.groupby("country")["amount"].transform("median")
        ]
        .assign(total=lambda df: df["amount"] - df["discount"])
        .groupby("country")["total"]
        .sum()
        .reset_index()  # to produce a DataFrame result
    )

Hi Matt! Big fan. The first that I thought when I read this article was "How would Matt use his .assign-chain-kung-fu on this one" and I tried

   import pandas as pd
   purchases = pd.read_csv("https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-from-r/purchases.csv")
   (purchases
     .assign(country_median=lambda df:
         df.groupby("country")["amount"].transform("median")
     )
     .query("amount <= country_median * 10")
     .groupby("country")
     .assign(total=lambda df: (df["amount"] - df["discount"]).sum())
   )

Hahahah... Yes, contributing to the frequency tables. At least we're not including the symbols in question in our comments, so as not add to the imbalance!

do you still intend to publish a guide to statistics?

Yes! Very much so. I'm in the final stages of cooking it right now, hoping to make it available for sale in the coming months.

Statistics is by far the hardest subject I've had to write about, as I had to (re)learn pretty much everything. I did several rewrites (working on tis book for the past seven years!), but now it's finally ready to ship. You can check out the notebooks from the book here: https://nobsstats.com/ and a detailed outline here https://docs.google.com/document/d/1fwep23-95U-w1QMPU31nOvUn...

ETA July for digital download, then one round of copy-editing, and September for first print version.

Here are the concept maps from the books that might be interesting for anyone learning these topics:

- https://minireference.com/static/conceptmaps/math_concepts.p...

- https://minireference.com/static/conceptmaps/math_and_physic...

- https://minireference.com/static/conceptmaps/linear_algebra_...

Nice to see adults (re)learning math. It's one of the rare subjects that are guaranteed to provide "knowledge buzz" and also widely applicable.

<shameless plug>For other adults interested in learning math, check out my (nonfree) book here: https://noBSmath.com/ PDF preview and sample chapter: https://minireference.com/static/excerpts/noBSmath_v5_previe... I also have a longer book that includes calculus and physics (cf. links in profile). </shameless plug>

Non-paywall link: https://archive.is/IJcOI

Very interesting.

I was thinking of scaling this up by adding Sunday too, but offline mornings might be even better... I'll look into adding a schedule in the router. I think it might be helpful to program the rule to start in the desired bedtime in the evening to make the pixel devices unexciting and incentivize reading old school books (the most efficient technology for getting me to sleep).

I've heard the advice don't touch the phone/email first thing in the morning so many times, but I can never stick to it, but if my "habit" is enforced at the network layer, I think it is doable ;)

UPDATE: Apparently iOS has a built in automation tool Shortcuts app, which allowed me to schedule this, see https://imgur.com/a/TsQJak3 I added similar rulesets to the router. Watch out world, Ivan's productivity is going to skyrocket!

I'm living on my own now so I don't have experience with the group version of this.

I try to be proactive if I know I'm meeting someone to remind them to call/text and not expect Signal/Telegram comms. The only thing that has been a problem are RCS text messages since my iPhone assumes if RCS was used once to communicate with someone, it should always be used. I can manually use "send as SMS" for outgoing, but I don't get the reply.

One thing that has been working really well for me this year is the "internet Shabbat" rule: 24 hours of no TCP/IP every week. Friday night I pull the plug on the modem + turn off data on my phone, then enjoy all of Saturday in a disconnected mode. I can still be on the computer, just no internet.

It's amazing how much things I get done... it literally feels like I have two days instead of one. If I put a item like "outdoor exercise for 30 mins" in my TODO list, then I actually do it! Close friends and family can still reach me by phone/text, but all the chat groups and other messaging app distractions are silent.

I can still end up zoning out watching a movie in the evening, but it's more intentional: I had to pick the movie in advance and download it so I can view it offline.

The contrast with Sunday when I turn the internet back on is very notable. Suddenly the next day is disorganized and end up wasting hours on youtube recommendations, and following rabit holes, and hacker news of course.

My conclusion is that it's not the apps or the internet, it's the feeds (when information is being pushed down to you) that are the problem. It's just too convenient and we end up staying in it.

Highly recommended.

Wow thanks for the plug!

Here is also the concept map from the book for people to see what topics are covered in it: https://minireference.com/static/conceptmaps/math_and_physic...

Since you mentioned DSP, you might want to also check out this excerpt on Fourier transformations from the No Bullshit Guide to Linear Algebra: https://minireference.com/static/excerpts/fourier_transforma...

If anyone is interested in trying this, here is some Python starter code you might find useful: https://github.com/minireference/lulu-api-client?tab=readme-...

This worked four years ago when the API was still launched, but there might have been changes since, so no guarantees.

Most ArXiv PDFs are probably lulu-printable out of the box, but to make a general solution, one would probably need to do some pre-processing with ghostscript (gs), e.g. embed all fonts and flatten images (no transparency).

Which JASP analyses would you recommend?

I think the Bayesian linear regression is cool, but I'm not a big fan of the Bayes Factors tests, since they are very sensitive to the choice of priors.

If we just switch from p-values (with 0.05 cutoff) to Bayes Factors (with hard cutoffs), then we'll have many of the same problems stats currently has...

Thanks for the plug!

If anyone is interested in the "No Bullshit Guide to Math & Physics," you can check out the extended PDF preview, which includes a free chapter: https://minireference.com/static/excerpts/noBSmathphys_v5_pr...

The concept map from the book might also be of interest independently of the book, since it shows the connections between high school math topics, physics 101 (mechanics), and calculus: https://minireference.com/static/conceptmaps/math_and_physic...

Non-paywall link: https://archive.is/tYZLX

A couple of years ago I discovered the MacOS accessibility feature "Speak selection" under "Accessibility > Spoken Content." I've set up a keyboard shortcut for it, which allows me to take any ebook in PDF format, select a bunch of text (usually one chapter) press the key and let the computer read it to me. Basically, it turn any text into an audio book, then listen while exercising, doing the dishes, cleaning the house, or simply when you like down and need a break.

I've never was a literary person (despite everyone else in my family reading a lot), but ever since this discovery I've been catching up on a lot of fiction, philosophy, psychology book, and pretty much anything that doesn't have code or equations. Highly recommended.

See here for screenshots: https://docs.google.com/document/d/1mApa60zJA8rgEm6T6GF0yIem...

Yesterday, I saw a thought provoking talk about the future of of "math jobs" assuming automated theory proving becomes more prevalent in the future.

[ (Re)imagining mathematics in a world of reasoning machines by Akshay Venkatesh]

https://www.youtube.com/watch?v=vYCT7cw0ycw [54min]

Abstract: In the coming decades, developments in automated reasoning will likely transform the way that research mathematics is conceptualized and carried out. I will discuss some ways we might think about this. The talk will not be about current or potential abilities of computers to do mathematics—rather I will look at topics such as the history of automation and mathematics, and related philosophical questions.

See discussion at https://news.ycombinator.com/item?id=42465907

Nice. I see you cover hierarchical (multilevel) linear models, which is already a big step up from the techniques normally covered in STATS101.

The general advice about measuring/comparing models also seems useful.

I've been working on a introductory STATS book for the past couple of years and I totally understand where the OP is coming from. There are so many books out there that focus on technique (the HOW), but don't explain the reasoning (the WHY).

I guess it wouldn't be a problem if the techniques being taught in STATS101 were actually usable in the real world. A bit like driving a car: you don't need to know how internal combustion engines work, you just need to press the pedals (and not endanger others on the road). The problem is z-tests, t-tests, ANOVA, have very limited use cases. Most real-world data analysis will require more advanced models, so the STATS education is doubly-problematic: does not teach you useful skills OR teach you general principles.

I spent a lot of time researching and thinking about STATS curriculum and choosing which topics are actually worth covering. I wrote a blog post about this[1]. In the end I settled on a computation-heavy approach, which allows me to do lots of hands simulations and demonstrations of concepts, something that will be helpful for tech-literate readers, but I think also for the non-tech people, since it will be easier to learn Python+STATS than to try to learn STATS alone. Here is a detailed argument about how Python is useful for learning statistics[2].

If you're interested in seeing the book outline, you can check this google doc[3]. Comments welcome. I'm currently writing the last chapter, so hopefully will be done with it by January. I have a mailing list[4] for ppl who want to be notified when the book is ready.

[1] https://minireference.com/blog/fixing-the-statistics-curricu...

[2] https://minireference.com/blog/python-for-stats/

[3] https://docs.google.com/document/d/1fwep23-95U-w1QMPU31nOvUn...

[4] https://confirmsubscription.com/h/t/A17516BF2FCB41B2

I recently learned[1] that he didn't write this book. He was telling stories to Ralph Leighton, who then wrote the books, and there is reason to believe many of the stories were embellished/enhanced. I'm kind of disappointed now that he didn't write the book himself, but I remember enjoying it when I read it as an undergrad.

[1] https://www.youtube.com/watch?v=TwKpj2ISQAc

I like the idea of math as "self help." People don't realize it, but the more math you learn, the easier/simpler life becomes... In other words, people learn math not because they like complexity, but because they are lazy and don't want complexity in their life.

For a specific example, consider some complicated arithmetic expression involving a dozen numbers and repeated operations +/-/*/÷. A person who knows high school algebra, could introduce some structure in the expression (e.g. by defining variables), then use the rules of algebra to simplify the expression, and end up doing much arithmetic overall to compute the answer.

The more (as in abstraction and modelling) math you know, the less math (as in arithmetic) you'll have to do!

What is probabilistic programming actually useful for?

You can think of a probabilistic programming language as a set of building blocks for building statistical models. In the olden days, people used very simple frequentist models based on standard reference distributions like the normal, Student's t, chi2, etc. The models were simple because the computational capabilities were limited.

In modern days, thanks to widespread compute and the inference algorithms, you can "fit" a much wider class of models, so researchers now tend to build bespoke models adapted for each particular application they are interested in. Probabilistic programming language are used to build those "custom" models.

Nice!

The key step of the derivation is counting the "number of ways" to get the histogram with bar heights L1, L2, ... Ln for a total of L observations.

I had to think a bit why the provided formula is true:

   choose(L,L1) * choose(L-L1,L2) * ... * choose(Ln,Ln)

I recently watched a talk[1] on this exact use case: a RAG system that runs on localhost with a simple web UI, and based on very powerful text processing, and a simple backend (PHP and sqlite3 with FTS and vector search extensions).

You can see the project page here: https://textualization.com/ragged/

src and scripts here: https://github.com/Textualization/the-ragged-edge-box

[1] video presentation about the project https://www.youtube.com/watch?v=_fJFuL2pLvw

Not the OP, but I think I know what wongarsu is referring to.

In order to make an object turn, it needs to experience an centripetal acceleration (towards the centre of rotation). This is the force causing objects to change trajectory.

If there is another object inside the turning object (like clothes inside a washer, or a person inside a car) they will "feel" like they are being flug out as if a centrifugal force existed, but actually that is just the effect of Newton's first law: the natural tendency of every moving body is to continue to move in a straight line, so when the containing object is changing direction (due to the centripetal force), Newton's first law tends to push you outwards.

All of the above is from the static (world) frame of reference.

It is also possible to put a coordinate system on the rotating object, in which case something like a centrifugal force will exit, but we kind of created it by choosing an accelerating reference frame, so it's not real. Sometimes called a pseudoforce.

demo video: https://www.youtube.com/watch?v=MOdSeUp8YcE

paper: https://arxiv.org/abs/2405.18614

From the abstract "The guidelines here cover both the LaTeX source as well as the output, so this PDF is intended to be read alongside its own source code," which I found here:

https://github.com/michaelchanwahyan/latex_templates/blob/ma...

That would push things towards the mean... not necessarily a bad thing, but presumably later steps of the analysis will be pooling/averaging data together so not that useful.

A more interesting approach, let's call it OPTION2, would be to sample from the predictive distribution of a regression (regression mean + noise), which would result in more variability in the imputations, although random so might not what you want.

The multiple imputation approach seems to be a resampling methods of obtaining OPTION2, w/o need to assume linear regression model.

Direct link to Part 2: https://profmattstrassler.com/2024/10/03/why-is-the-speed-of...

Me too. I do a lot of text editing on a daily basis and I find I gain at least +10 IQ points when I print out the text and review on paper.

My colleague swears by her iPad, but I always go the paper route.