noahgolmant

There is some literature on student-teacher training with Jacobian matching: https://arxiv.org/abs/1803.00443 this might be in the direction you're looking for. I believe in the cross entropy case matching the Jacobian should imply a matching Fisher information matrix.

Back in 2018 I published pytorch-hessian-eigenthings, a niche open source package for GPU-accelerated curvature analysis of PyTorch models. Loss landscape curvature metrics like the eigenvalues of the Hessian have been implicated in many generalization properties of neural networks (like flat-minima hypotheses, low-rank Hessian claims, etc.). But the full Hessian costs memory quadratic in the parameter count, which is usually infeasible. This library uses Hessian-vector products + iterative methods (Lanczos, power iteration) to get the eigendecomposition in linear memory instead. I stepped away from the project for years, but it ended up being used by other researchers doing curvature analysis work. I noticed the original implementation had aged so I thought I'd revisit it. I also have more professional engineering experience under my belt to inform the design.

I just shipped a v1.0 rewrite. The new version adds new curvature operators (Generalized Gauss-Newton, empirical Fisher), and new algorithms (Hutchinson + Hutch++ trace estimation, spectral density via Stochastic Lanczos Quadrature). It also has a fused Triton/torch.compile cross-entropy Hessian-vector kernel for foundation-model-scale vocabularies (where standard implementations blow up). More importantly it adds a lot of numerical analysis validating the operators: closed-form correctness on linear/logistic regression where the Hessian is known analytically, and cross-library tests against curvlinops to catch any regressions.

https://github.com/noahgolmant/pytorch-hessian-eigenthings

I'm hoping to use it for some follow-up analysis. For example right now I'm looking at inter-agreement between various optimizer updates (Muon, K-FAC, Natural Gradient Descent) on Pythia checkpoints.

Very open to suggestions or requests from anyone who's been working in this space. I've been out of the field for a while, so pointers to recent work I should be aware of are very welcome.

The human has unique context. They may work in a niche domain or they talked to people and observed an unsolved problem. Then they express a potential solution via OSS. It's like product sense. Then they share that with others who find it interesting. The code is a great way to encapsulate the idea. It is usually the result of research and back and forth not a single prompt. It would be way harder to think through or build a solution without AI even if they had context.

I don't really know. Certainly we need a higher bar. The Kafka example in the post may be hyperbolic but I agree it pollutes the space. But we also can't swing the other way and rely completely on out of date proxies. If you ban AI code there will be very little code to see in a year. It'll take time but we'll arrive at new norms. We built semi successful ways to filter content farms in the earlier internet days. The signal has to shift to "did they think hard about this problem" which has some observable properties. Like how they articulate the problem, or why it became important to them.

There has to be room for an AI-driven project that expresses a unique idea, even if there's no community around it yet. Someone has to express it, and from now on that idea will largely be implemented with AI.

A good use of AI is when it enables people to do something they couldn’t do before, to contribute to a community when they couldn’t before.

I agree 100% with the novel contribution aspect. But there's some nuance there.

For example a project might have no active contributors. It might not be something you can drop directly into your codebase. Neither of those is inherently bad.

As AI becomes more responsible for higher-level planning decisions, the value of an OSS project becomes less tied to visible community activity like PRs and issues.

I notice this in my own work a lot. I might not use that project's code directly. But I think about a problem differently as a result. I often point my agent to existing OSS projects as inspiration on how to solve a problem. The project provides indirect value by supporting architectural decisions, deployment approaches etc. Unfortunately OSS activity doesn't capture this.

I would not extrapolate California results to the rest of the country. Other regions like Northern Virginia or Dallas-Forth Worth dominate current and future data center load. They have significantly different water and land use trends. Of course agricultural water usage is higher in California- it's the top agricultural state in the country. And energy prices are much higher, resulting in a smaller data center presence. State regulations also determine water usage efficiency requirements. So it might differ in less regulated states like Texas.

You may want to look into the PMTiles format and tippecanoe. It efficiently produces pyramidal XYZ tile overviews of vector data. Sometimes this is also done server side via the PostGIS asMVT ffunction, or Martin.

For client side rendering, deck.gl is quite good, also a newer library called lonboard from DevelopmentSeed.

A tool to build labeled object detection training datasets from large-scale satellite/aerial imagery collections: https://github.com/noahgolmant/label-tiles

Web maps usually join together lots of small images called tiles (this is why you see square patches as google earth/map loads). They do this by querying a "tile server" API. It turns out this standard can also be leveraged to label and fine-tune models on map imagery. In my day job we built infra to efficiently serve imagery through tile servers for map visualization. So I wanted to test out ML applications of that infra.