mmmmpancakes

So this is just showing a bit of your ignorance of stats.

The general notion of compound risk is not specific to MSE loss. You can formulate it for any loss function, including L1 loss which you seem to prefer.

Steins paradox and James Stein estimator is just a special case for normal random variables and MSE loss of the more general theory of compound estimation, which is trying to find an estimator which can leverage all the data to reduce overall error.

This idea, compound estimation and James-Stein, is by now out-dated. Later came the invention of empirical Bayes estimation and the more modern bayesian hierarchical modelling eventually once we had compute for that.

One thing you can recover from EB is the James-Stein estimator, as a special case, in fact, you can design much better families of estimators that are optimal with respect to Bayes risk in compound estimation settings.

This is broadly useful in pretty much any situation where you have a large scale experiment where many small samples are drawn and similar stats are computed in parallel, or when the data has a natural hierarchical structure. For examples, biostats, but also various internet data applications.

so yeah, suggest to be a bit more open to ideas you dont know anything about. @zeroonetwothree is not agreeing with you here, they're pointing out that you cooked up an irrelevant "example" and then claim the technique doesnt make sense there. Of course, it doesnt, but thats not because the idea of JS isnt broadly useful.

----

Another thing is that JS estimator can be viewed as an example of improving overall bias-variance by regularization, although the connection to regularization as most people in ML use it is maybe less obvious. If you think regularization isn't broadly applicable and very important... i've got some news for you.

No it is not bogus, you just don't know much stats apparently.

parquet is perfectly fine

There is Parquet. It is very efficient with it’s columnar storage and compression. But it is binary, so can’t be viewed or edited with standard tools, which is a pain.

I can open parquet in excel

To me, Polars feels like almost exactly how I would want to redesign Pandas interfaces for small - medium sized data processing, given my previous experience with Pandas and PySpark. Throw out all the custom multi index nonsense, throw out numpy and handle types properly, memory map arrow, focus on method chaining interface, do standard stuff like groupby and window functions in the standard way, and implement all the query optimizations under the hood that we know make stuff way faster.

To be fair, Polars has the benefit of hindsight and designing their interfaces and syntax from scratch. The poor choices in Pandas were made long ago, and its adoption and evolution into the most popular dataframe library for python feels like mostly about timing the market than having the best software product.

Suggest the following pattern:

1. load and process / aggregate in polars to get the smaller dataset that goes into your plot. 2. df.to_pandas() 3. apply your favourite vis library that works with pandas.

There's no use case i can think of where building a data viz interface more specific to polars than this is beneficial or necessary.

i mean, if you are reading and writing csv then yeah, you've already fucked up.

Scientific fraud makes me sad and angry.

Disagree. The purpose of a textbook and a lecture is very different. A good textbook can be a helpful resource for teaching and lecturing, but it is not sufficient to guarantee high quality math education. Conversely, a good educator who deeply understands the material can deliver fantastic education without a good textbook. Claiming that profs writing bad textbooks is the cause of poor quality in class math instruction is absurd.

The ability to communicate math this way is honestly rare. It comes from a combination of deep understanding, long experience in communicating math, and a certain level of "culturing" that is specific to the academic experience.

Among the best, Feynman was singular in his ability to communicate math and physics.

In other words, don't be so hard on the teachers who were disappointing in comparison to the stellar examples you see from top mathematical communicators. What you're reading is quite rare and, while education quality could certainly improve, its not fair to expect this of a 5th grade teacher who covers 5 topics in a day. Even for the best, developing this type of material takes time and thought that a school teacher probably does not have.

Because with real world data like in production in tech there are so many factors to account for. Brittle methods are more susceptible to unexpected changes in the data or unexpected ways in which complex assumptions abut the data fail.

This seems like a natural feature of any sensitive method, not sure why this is something to complain about.

I am exactly complaining it is sensitive. If theres robust alternatives why would i put this in prod?

Yeah, the only common theme I see in causal inference research is that every method and analysis eventually succumbs to a more thorough analysis that uncovers serious issues in the assumptions.

Take for instance the running example of catholic schoolings effect on test scores used by the boook Counterfactuals and Causal Inference. Subsequent chapter re-treat this example with increasingly sophisticated techniques and more complex assumptions about causal mechanisms, and each time they uncover a flaw in the analysis using techniques from previous chapters.

My lesson from this: outcomes causal inference is very dependent on assumptions and methodologies, of which the options are many. This is a great setting for publishing new research, but its the opposite of what you want in an industry setting where the bias is/should be towards methods that are relatively quick to test and validate and put in production.

I see researchers in large tech companies pushing for causal methodologies, but I'm not convinced they're doing anything particularly useful since I have yet to see convincing validation on production data of their methods that show they're better than simpler alternatives which will tend to be more robust.

From my experience propensity scores + ipw really doesn't get you far in practice. Propensity scoring models rarely balance all the covariates well (more often, one or two are marginally better and some may be worse than before). On top of that, IPW either assumes you don't have any cases of extreme imbalance, or, if you do you end up trimming weights to avoid adding additional variance, but in some cases you do even with trimmed weights..

It may be ideal to have all these items checked off. I think a productive way to look at this is "how many of these items does the roadmap check"?

If the answer is "very few" then that might be an early warning sign you're working for a product or org that is headed for serious problems. As a technical IC who can't hope to solve those large scale management problems, this can be useful a red flag. Don't wait for shit to hit the fan to leave.

I have seen a large scale failure of this kind and the items listed here line up very well with some of the root causes I observed.

- Is the roadmap flexible or iterative? The roadmap was hard, aggressive business targets.

- Are the roadmap initiatives scoped and prioritized based on evidence? The roadmap initiatives were derived by working backwards from business targets and then evidence was found after the fact.

- Does the roadmap identify major dependencies or risks? Many risks were identified much later because techincal teams were not part of input to initial planning.

- Does the roadmap feel aggressive but achievable? Aggressive but not physically possible.

- Does the roadmap take on appropriate risk? No, there was multiple possible independent points of failure.

Anyways, if you ever see this kind of product culture I suggest running for the hills unless you like having your time wasted. And if you are a technical manager I hope you push back like hell when presented with this situation.

thanks

can you please spell out what MDL is an acronym for?

I mean, of course? Concurrency != parallelism, so that makes sense.

Where multiprocessing shines is when you have an algorithm that can be fully parallelized and represented in a baby map-reduce framework, where the data being sent to each process isn't too big. The idiom I often reuse is literally the first example in the python docs:

  from multiprocessing import Pool

  def f(x):
    return x\*x

  if __name__ == '__main__':
    with Pool(5) as p:
      print(p.map(f, [1, 2, 3]))
      
      # to give more of a map reduce flavor:
      print(sum(p.map(f, [1, 2, 3])))

I mean, hopefully why you might need multiprocessing in python is clear?

If you have a python task that is highly parallelizable on a single machine with multiple cores, then multiprocessing is probably the right tool to quickly see if you can dramatically speed up your code with parallelism with basically no code overhead or investment in distributed solutions (there are edge cases where it is not, but it takes very little time to test if you are an edge case).

I encounter this situation in my data science workflow routinely. It is an easy way to impress product / managers and say "hey, I made this batch algorithm 50x faster, so now it runs in 10 minutes instead of 500."

how do you work 4 years on a project before having a tracked goal. absurd. sounds like management's fault.

fancy name for what seems like a fairly standard "intro to proofs" textbook, at least based on the ToC.

The fact that Microsoft is baking GPT into all of their products guarantees explosive growth.

Why?

I still think Elon is a smart guy.

I don't think he realizes that the techniques that work for cars and rockets don't work in social media.

Hmmm...

Amazon has roughly 350 MM products if you account for third party sellers, so yeah, the scale is massively different.

Not only this but Amazon is a marketplace for buyers AND sellers whereas it appears McMaster-Carr is the seller. Sellers on Amazon are also Amazon customers who want their own pages, copy, etc.

The problem with your comparison to Amazon is that the scale of products offered on Amazon is completely different.

A catalogue system like this is only viable if the number of products is relatively small, as appears to be the case of mcmaster-carr.

Confidently incorrect.

It goes without saying that quality of research is very important.

However, getting a TT position is much more about connections and fashion than people think, especially in a field like pure math where it's much more difficult to argue concrete applicability and outcomes of your theoretical work. And this is especially true when there are many people doing top quality research (which there are).

A top advisor will be able to sell you to departments elsewhere for a top postdoc, or straight into TT if you are impressive enough. They will also be able to culture you appropriately so that all your application materials look precisely right.

Finally, a top advisor will have a significant breadth and depth of knowledge across research fields (this is rarer than you might think in pure math) and be able to guide you to work in areas that have better job markets.

The reality now of TT applications at desirable universities is that the pile is hundreds of applications deep. Each application is a hefty bundle (cover letter 1-2 pages, research statement 2-10 pages (probably on the longer side for math), teaching statement 1-2 pages, cv 3-6 pages depending on formatting, 4+ letters of recommendation each several pages, and other documents like EDI statements which can run 1-2 pages). How the hiring committee winnows this list down to a short list of candidates depends on the institution, but you better believe that if famous prof X from Harvard, who was best friends with commmittee member Y in grad school at Harvard, makes a personal recommendation for candidate Z to committee member Y, then candidate Z has a significantly higher chance of making that short list than anyone else in the pile.

Another reason for this is that the work is so esoteric and fields are so siloed now that it's very hard for committee members to make a fair and true evaluation of research quality for themselves. I guarantee you they are not taking time to read and evaluate quality of research in research papers in most cases. The input of their trusted colleagues at other institutions holds weight as does metrics like publication counts and where the papers are accepted. By the way if you have a top advisor it may be much easier for you to figure out how to get your results published in a top journal.

You see?

Also I was in differential geometry.

I think that's a very strange thing to say. I am not sure which priorities you are referring to, but I think I have a good guess, and I think they're ridiculous things to expect that aren't / shouldn't be correlated with research outcomes.

What I have experienced over my time in academia is that priorities change over time.

When I was young and in graduate school I was very happy to live a spartan life and devote the bulk of my time to research. I thought very little about finances.

Now I have a partner and a child. Postdoc salaries are simply not sufficient to support them financially. The need to move every few years for a new position makes it difficult for my partner to have her own steady career.

I'm quite confident I could still make a good researcher. In fact, I'm being paid for precisely that at my new job. I just don't happen to be in academia where the pay and WLB sucks and everyone expects you to sacrifice quality of life on the altar of their virtuous research.

Incentives and priorities in academia (especially pure math) are by now almost totally disconnected with reality. My former colleagues have no idea what I am doing at my current job and don't even know what sort of questions to ask about what I do (beyond idiotic questions like "what math do you use?"). There's no need for them to know because they're coddled in an archaic system that doesn't foster true innovation (it shields them from the market forces that would otherwise compel them to innovate).

Beyond the better living conditions, I am happy I left academia because IMO it is not a good setting to do good research, it is exactly the opposite.

Same here but for math. I did a survey of who was getting the TT jobs I wanted in the cities I wanted to live in and the trend was that they all went to Harvard / Princeton with a few exceptions. Seeing how strongly those profile elements, which I don't have, correlated with success getting into TT, and how few positions there are, it was easy to decide to leave.

Other factors include pay for TT professors in high CoL cities not keeping up well enough with inflation over the last 10 years. Those salaries look much worse now than they did when I entered grad school. For the level of education it is underpay for overwork.

Doing science in industry pays substantially better, has better WL balance (really anything will compared to academia), has more job openings in more cities, and there are plenty of challenging problems to work on. Moreover, research skills that STEM phds and academics have are highly valued, at least by some companies.

In the end things seem to have worked out for me. I was warned about all this back when I entered grad school but didn't listen because I really wanted to do math. Following that passion was a good instinct after all even if I wasn't able to achieve the original goals exactly as I planned. I'd 100% do it again.

It means a small change in x.

So here is the issue. "A small change in x" is a concept that is relevant and meaningful for Riemann integration: it represents that the integral is defined as a limit of Riemann sums as the change in x becomes infinitely small.

But this is not relevant for Lebesgue integration! We are not taking a limit of Riemann sums and there is not a limit of change in x getting arbitrarily small. What matters is the measure (and the definition of the integral in terms of simple functions is something entirely different).

So you see using dx this way in the context of Lebesgue integration seems like a potential source of confusion, not simplification.

This is why i ask what dx means. Either it represents the standard Lebesgue measure on R (this is valid and a special case of the notation d\mu(x) since \mu is the identity function), or it is nonsense that potentially confuses concepts from other integration theories.

This is why mathematicians don't like your idea. They encounter many students who are confused about this distinction and don't find that it's a useful simplification.