nirinor

Answering myself: I've tried pi and oh-my-pi (omp.sh) in the last couple of days, and the latter feels solid and more effective than CC/opencode/pi. I have not reviewed the code, and 2 days is not a lot, so take with a large grain of salt, obviously.

I use CC at a client because that's their stack. I use (a particular version of) OpenCode for my own work, because it is so much better. Reading this post makes me sad.

Because everything matches and explains oddities I've seen and forgiven. So despite the hyperbole and the places where the authors' general sentiment are not mine, he's basically right, and I need to find a different harness.

If the post resonates for you too and you've found something better, happy for recommendations. pi seems to get the most mentions here, anyone feel it (or another option) is specifically better on these architectural aspects?

So, are computing gradients details of backpropagation that it is failing to abstract over, or are gradients the goal that backpropagation achieves? It isn't both, its just the latter.

This is like complaining about long division not behaving nicely when dividing by 0. The algorithm isn't the problem, and blaming the wrong part does not help understanding.

It distracts from what is actually helping which is using different functions with nicer behaving gradients, e.g., the Huber loss instead of quadratic.

Nah, Karpathy's title is "Yes you should understand backprop", and his first highlight is "The problem with Backpropagation is that it is a leaky abstraction." This is his choice as a communicator, not the poster to HN.

And his _examples_ are about gradients, but nowhere does he distinguish between backpropagation, a (part of) an algorithm for automatic differentiation and the gradients themselves. None of the issues are due to BP returning incorrect gradients (it totally could, for example, lose too much precision, but it doesn't).

Its a nit pick, but backpropagation is getting a bad rep here. These examples are about gradients+gradient descent variants being a leaky abstraction for optimization [1].

Backpropagation is a specific algorithm for computing gradients of composite functions, but even the failures that do come from composition (multiple sequential sigmoids cause exponential gradient decay) are not backpropagation specific: that's just how the gradients behave for that function, whatever algorithm you use. The remedy, of having people calculate their own backwards pass, is useful because people are _calculating their own derivatives_ for the functions, and get a chance to notice the exponents creeping in. Ask me how I know ;)

[1] Gradients being zero would not be a problem with a global optimization algorithm (which we don't use because they are impractical in high dimensions). Gradients getting very small might be dealt with by with tools like line search (if they are small in all directions) or approximate newton methods (if small in some directions but not others). Not saying those are better solutions in this context, just that optimization(+modeling) are the actually hard parts, not the way gradients are calculated.

I am not hiring, but might be able to help with some other parts. DM me if you want to talk.

I use python extensively. I've used bash (+awk+xargs+sed...) extensively.

Nushell is already a great improvement over bash _as a shell_. It is even better when using it to compose _preexisting text based programs_. I would say it is better in every way I can think of, except for: - not (yet) coming pre-installed, and - stability of interfaces and language.

Its already better enough to be my default shell on my daily driver, though I keep bash around because some things really assume it. I very much look forward to one day having a userspace with no traditional shells at all.

nushell is not yet a strict improvement over python, but it might one day be, and it is already better at: - munging text, json, dates and tables - quickly creating nice CLIs callable from the shell (even if that shell isn't nu!) - fun of programming in it

Nushell is trying to blend two domains, shells and programming languages, which I see distinct advantages in keeping seperate.

Interesting, though, how many PL features the most popular shells tend to have...

I do not want the world to be built on the back of shell scripts, regardless of how good you make the type system.

If I had read that before knowing nushell, I would strongly agree. Yet, it turns out you can make a shell so good I wouldn't mind if... not the world, but _a lot more_ was built on it.

Very nice formalization.

One area for refinement: it considers two stacks either identical or unrelated. Consider that stack A;B is actually very close to A;B;C, the difference might be due to a sample time occurring just before or just after the call to C. OP considers them just as different as A;B and Z;W, therefore amplifying a measurement noise.

This suggests using a refined metric between stacks (e.g., an edit distance counting pushes and pops), and then we can use it in defining the metric between flamegraphs (e.g., an optimal transport metric [1], instead of the proposed L1).

Avoiding that noise amplification reduces the background noise level, therefore the cost of effective measurements. From another perspective, the current OP scheme creates an avoidable curse of dimensionality in the form of the Hotelling test's requirement that each sample has more measurements have more samples than distinct stack frames. So the same code split into more functions is harder to measure, and too-small samples are useless. I think neither of those is necessary if we take stack similarity into account.

[1] https://en.wikipedia.org/wiki/Wasserstein_metric

I've needed these capabilities often while using awk for converting messy logs/error outputs into tables/commands.

Nowadays I like the nushell approach to the composition:

    echo 'quux=123 foo=123 bar=123' | str replace '.*quux=([0-9]+).*foo=([0-9]+).*' $"$2,$1" | from csv -n | each {|r| $r.column1 + $r.column2}

Mislabeled? sounds like you're SEEKING WORK, SEEKING FREELANCER is for the other side.

SEEKING WORK | NYC area | Remote

Hi, I am Daniel Vainsencher, ML PhD and practitioner. You want to use AI to solve a business or technical problem, have data and a team, but no senior research staff.

I can, on a consulting/contracting basis:

1. Translate your problem and context into well defined prediction and decision problems, connecting performance to the bottom line.

2. Plan out, chunk, and help implement solutions to prediction and decision problems via existing (and if needed, new) ML and optimization software.

3. Support your engineer/research staff in applying ML and connecting it to business goals, via ideas, coaching and troubleshooting.

Have years of experience building software systems, leading dev teams, developing algorithms, publishing state of the art ML/optimization research, and applying them to challenging problems. I've worked/done applied research in domains of entertainment, algorithmic trading, video processing, face recognition, signal processing, demand forecasting and more.

Tech: Python, Julia, Rust, git and others.

https://www.linkedin.com/in/danielvainsencher/

https://www.semanticscholar.org/author/D.-Vainsencher/273458...

Email: danielv at nirinor.ai

I've joined the Tribe community a few months ago, done one project with them, so far very happy with my experience.

- Pleasant, helpful, knowledgeable people on the slack.

- Brief and effective process matching me with a client.

- They negotiated a rate I was happy with, taking a cut I found reasonable.

- Good working relationships with others in the project, both from Tribe and the client, everyone happy.

- Payment was simple and timely, when I encountered a technical issue with their provider during setup, Tribe were very helpful resolving it.

Feel free to ask questions here or DM me

Some applications depend on approximately solving optimization problems that are hard even for small problems. The poster child here is combinatorial optimization (more or less equivalently, np-complete problems), concrete examples are SMT solvers and their applications to software verification [1]. Non convex problems are sometimes similarly bad.

Non smooth and badly conditioned optimization problems scale much better with size, but getting high precision solutions is hard. These are important for simulations mentioned elsewhere, but not just for architecture and games, also for automating design, inspections etc [2].

[1] https://ocamlpro.github.io/verification_for_dummies/

[2] https://www.youtube.com/watch?v=1ALvgx-smFI&t=14s

We see so many examples of regulations having directionally wrong effects...

That regulation is adversarial (largest regulated entities are savvy and often propose the regulations), has higher order and long term effects, this all makes it a hard domain, but also is well known in advance.

What's the current best practice for validating regulation have their intended effects in advance?

SEEKING WORK | US East coast | Remote

Hi, I am Daniel Vainsencher, ML PhD and practitioner.

I can, on a consulting/contracting basis:

1. Convert your business problems into well defined prediction and decision problems, connecting performance to the bottom line.

2. Solve prediction and decision problems via ML and optimization.

3. Support your engineer/research staff in applying ML, with ideas, coaching and troubleshooting.

Have years of experience building software systems, leading dev teams, developing algorithms, publishing state of the art ML/optimization research, and applying them to challenging problems. I do one time consultation, mid term projects and retainer.

Willing to share risk/reward of the right company, but no equity-only at this time.

Tech: Python, Julia, Rust, git and others.

https://www.linkedin.com/in/danielvainsencher/ https://www.semanticscholar.org/author/D.-Vainsencher/273458...

Email: danielv at nirinor.ai

SEEKING WORK | US East coast | Remote

Machine Learning/optimization expert, I can:

1. Design ML systems tailored to business needs,

2. Coach your engineer/research staff on ML/Optimization,

3. PoC/customize/troubleshoot algorithm implementations, on a consulting/contracting basis.

https://www.linkedin.com/in/danielvainsencher/

https://scholar.google.com/citations?user=FnM8S4kAAAAJ&hl=en

Now this sounds like a proper fun problem! :)