tatef

Hi HN!

Coming to you today with Autolab, a framework I built that combines Ralph loops and Karpathy's autoresearch to design and run tests and make novel research discoveries. I hope one day a similar framework can be used with hardware, but for now, I've been using this tool in my workflows and find it incredibly useful. Wanted to share.

Hi HN,

Wanted to share a project I've been working on aimed at helping solve the context problem called Memoryport. It works across LLM providers/apps, stores conversations locally, is fully OSS, and enables anyone to keep track of their memories over time. I tested mine with 500M tokens of context space (note: not the same as a context window) and only added ~300ms of latency to the inference session. I also built out an open spec called AMP that standardizes the communication protocol for how memory systems should interact with LLMs (see repo).

More than happy to answer any questions and hope you all find this as useful as I have.

Thanks for sharing this! If you'd be interested in running the benchmark yourself with Hypura I'd happily merge into our stats. Otherwise will add to my todo list :)

Yes definitely. I use a M1 Max with 32gb of RAM daily and it's about on par from a performance standpoint with the new base M5 Pro 24gb. You can check the benchmarks in the repo if you're interested in seeing specific performance metrics, but investing in Apple hardware with as much memory as possible will generally get you furthest in this game.

Noted, thanks. I had LLM help positioning this message but I did the initial draft along with edits. Will keep in mind for the future.

Hypura reads tensor weights from the GGUF file on NVMe into RAM/GPU memory pools, then compute happens entirely in RAM/GPU.

There is no writing to SSDs on inference with this architecture.

Yes, exactly this.

I'm referencing it as being possible, however I didn't share benchmarks because candidly the performance would be so slow it would only be useful for very specific tasks over long time horizons. The more practical use cases are less flashy but capable of achieving multiple tokens/sec (ie smaller MoE models where not all experts need to be loaded in memory simultaneously)

Yes, definitely agree. It's more of a POC than a functional use case. However, for many smaller MoE models this method can actually be useful and capable of achieving multiple tokens/sec.

We aren't a package manager. We're a registry and CDN (of sorts). Blockchain is actually a huge solution to this problem for three very notable reasons. The first is that Deno module imports are url based, and we don't want code going off the internet, as this would break the code dependent on it. Blockchain solves this because transactions (module code) are unable to be modified or deleted. This means that import links will never break, thanks to blockchain! In addition, it's unbelievably cheap to permanently store data. For reference, we've stored 17,297 files on the blockchain. For proof, you can see our wallet address and transaction history here: https://viewblock.io/arweave/address/tySYSW93nDky1sbCO56PmyE... This permanent and decentralized data storage has cost us right around 5 cents USD. Thirdly, thanks to the blockchain, the module data is completely decentralized across over 340 nodes and counting around the world. You can see the exact statistic here: https://viewblock.io/arweave Again, thanks for bringing these things up. These are great points for us to address publicly.

Actually, this raises a very good point. I'm Tate, a co-founder. Our publishing system works in a way that users will be able to publish malicious modules, yes, but our registry is not decentralized up to a certain point; let me elaborate on this. If a user finds that a module is malicious and wants to report it, we can remove it from the registry completely because the registry is centralized. Though this data will still be accessible from the blockchain and the import url will be functional, we're building a system to warn the user whenever the url is imported from a Deno-specific response header. Now, after a certain amount of time has passed and a module isn't reported as malicious, we're building a system to automatically publish the entire registry to the blockchain as well, so that the registry AND the module are immutable. This is called Fossil, our "archiver." You can see its code here: https://github.com/nestdotland/fossil Again, thanks for bringing this up. I hope this explanation helped. Our goal certainly is not to promote or enable malicious code!

Indeed, this is a massive issue with any url based imports. Because Trex supports nest.land, this is not an issue. nest.land is actually a first-of-its-kind blockchain module registry and CDN. Because we use the blockchain for storing modules, they can never be deleted or altered in any way. This also means that they are permanently and indefinitely resolvable from the web. Because of this, module vendoring is no longer an issue!

Trex is not a company; Trex is a product under the crewdevio organization on GitHub.

In my opinion, Trex is actually working against what npm introduced to Node. Though I don't exactly know what "main problem" you're referring to, I can say that:

1) Trex is supporting multiple module registries, not just one. (Thus not necessarily centralized) 2) Trex is not associated with one entity in particular. This means that they have no company dedicated to hosting modules in house (another reason that they aren't centralized) 3) Node's package.json included many other things than just imports. Using Trex is completely optional, and if you do use it, an import map does not hinder one's development workflow. In my opinion, it makes dependencies easier to manage.