HN user
[ my public key: https://keybase.io/wdmn; my proof: https://keybase.io/wdmn/sigs/KaH_BqIpR_cc-JNsLgWnzWCnKy_ZnbQfoKTC0ms3-1o ]
similar to RWKV7’s new (sub quadratic) attention mechanism which models key values as v≈kS’ and does an in-context descent on ||v - kS’||^2/2 (where the state matrix S is one attentional head) , explained more by the author here https://raw.githubusercontent.com/BlinkDL/RWKV-LM/main/RWKV-...
and i tried to unpack it a bit here https://wdmn.fr/rank-1-take-on-rwkv7s-in-context-learning/
ha, yeah, thanks.
no, wrote it by hand for use with my own Heun implementation, since it’s for use within stochastic delayed systems.
jax is fun but as effective as i’d like for CPU
looks like a nice overview. i’ve implemented neural ODEs in Jax for low dimensional problems and it works well, but I keep looking for a good, fast, CPU-first implementation that is good for models that fit in cache and don’t require a GPU or big Torch/TF machinery.
i like how the contraction it’s and abbreviation T’is are anagrams
I didn’t look it up, but at first glance, it reminded me of discussions like this one
that’s not what the story says. in any case, the point is to explain, in terms of dualistic if-then logic, that the if (you practice now) and then (you will wake up) are a single non-dual thing. but to communicate in in terms which make sense to the dual, if-then mind, one needs to use dualistic language.
do you have a link for your game? how was the release process?
how far from your previous experience is the game work?
i wonder if there are any semi automated approaches to finding outliers or “things worth investigating” in these traces, or is it just eyeballs all the way down?
https://www.biorxiv.org/content/10.1101/2024.10.25.620245v1....
covers some of the recent perspectives on this modeling approach if you’re interested.
having worked on whole brain modeling the last 15 years and european infra for supporting this kinda research, this is a terrible buzzword salad. the pdf is on par with a typical masters project.
HH is kinda the opposite of LIF on the abstraction spectrum.
The pigeon experiment is a great one to learn from not just about programming or software, but about life in general. Where are you getting your next dopamine hit? Is it random? Maybe that’s where our idiosyncrasies come from.
CPUs have the memory capacity advantage
perhaps also more precisely they also have quite an advantage on anything that needs and plays nicely with caches? when I sliced my problem to maximize cache usage, I saw pretty clear scalability with cores: L1/L2 cache bandwidth is ~30GB/s, so e.g. a 32 core system starts to compete with the big consumer GPUs.
My approach to this is to write a bunch of tiny “kernels” which are obvious to SIMD and then inline them all, and it does a pretty good job on x86 and arm
People won't even pay for most of these pocket distractions
If someone is paying, the transaction, by construction reinforces the psychological boundaries that obliteration eliminates. So I think not paying is part of it, just like addicts ignore the (perhaps partially non monetary) price of their behavior.
I think “a cloud never dies” is more apt for this sentiment
How would you want to host or update the markdown file? Sending an email for instance? Or run your own host?
Unlike most prior efforts, we release not only model weights and inference code, but also the reproducible training data, complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols for open scientific research.
Regardless of the specific performance of this model versus another model, I think it’s good to keep in mind that everyone benefits from this kind of work
Just brainstorming here, but as a parent and someone who subscribed to a weekly CSA, it’d be a big boost to be able to take a picture of what I’ve got and get some ideas about recipes, especially if I could say, I’ve only got about 15 minutes to make dinner for kids. I guess that doesn’t really fit the Tinder model..
At the risk of taking some heat, I’d wager a preprint is recognized rightly by the Chinese as a flag planting, we’re first formality, where in the faults may even serve to validate it was written by human and not an LLM.
Whereas the Western academic may want to make the preprint as close to print as possible.
The core intent - communicating an idea - is still upheld.
I tend to think of science as a distributed consensus process, and that peer review is analogous to a proof of work, and publishers are gatekeeping the distribution of proof of work. I think this is a useful analogy because one can subtract (in theory) the gatekeeping entirely: distributing proof of work is required for the distributed consensus to update.
However, and, crucially, journals differ in their effect on the consensus, e.g. IEEE or PNAS have much higher impact factors, and the competition both among researchers and institutions creates a market opportunity for gatekeeping, that naturally sorts those same researchers and institutions for the next ground of grants.
Again, I think it's hard to understand what a fix would look like, if we don't first recognize how distributed consensus should work for science. Algos like Paxos require a leader, and editorial boards for journals are effectively leaders.
Was your path towards the decision point a sudden one or a slow burn?
Good question, I only looked briefly at unscented, I just thought ensemble was easy to understand.
This is really nice approach. We are doing some nonlinear system id, and faced with this kinda problem (not irregular spacing but low sample rate and noisy). Definitely will check it out.
What’s your opinion on ensemble KF? We’d like to use that for parameter estimations. I saw unscented in your bayesfilter, but not ensemble, so I’m curious. Thanks!
Ha, I was benchmarking arm really, not writing from scratch.
All of the other cores from ARM or Qualcomm aren't as fast as the top Intel and AMD x86
I don’t think top cpu perf is relevant. I was working on some C code for science stuff inside Termux on a Pixel 7a, and would’ve been perfectly ok having that perf on a standard format laptop. I even noticed some branch prediction was better than x86. It’s more an issue that no one is making a decent arm in laptop format with nvme, enough ram etc.
lol as if teachers have money to buy this to make their lessons plans.
Hi! Thanks for your reply. First time hearing about Brski, I’ll have a look
I guess your takeaway is that despite spotty support, the benefits of rust are net positive? I guess learning to call C from Rust is gonna be part of it too right ?