ramesh1994

Sorry I don't know how hackernews notifications work, didn't see this reply. But if you see this here's another reminder for you to clean this up into a public demo :)

This is really cool! Appreciate sharing this work and the explanation.

You mentioned that massaging the data into shape as one of the problems, which I think is possibly one of the best applications of LLMs in my opinion. Creating a pipeline of a data feed (still hard) into an LLM which outputs JSON with the fields of interest would be amazing.

Hey this sounds pretty cool! If this is public would you mind sharing a link?

I just found about this project from this comment, absolutely excited to try this out.

As someone who's never used any of the infrastructure tools, I'm thinking of pyinfra as a way to run shell commands + install dependencies on hosts (declaratively?) on a bunch of hosts via ssh.

Inventory is to sort of take a self-defined inventory on a bunch of hosts?

One final question on usage, would it be possible to sync or reference files from the machine running pyinfra on the remote hosts? Or would that have to be done indirectly via running shell commands to sync?

I think distillation in the original sense isn't being done anymore but finetuning on outputs from larger models like GPT-4 is a form of distillation (top-1 logit vs all logits and a curated synthetic data instead of the original dataset)

On quantization though its still weird how just the weights are quantized in methods like gptq / int8 while there are other methods which quantize the activations as well. There's also the matter of KV cache still being in original 16bit precision regardless which is also unsolved here. Do you have any thoughts or insights into this?

Head over to the elueuther.ai discord and discuss with some of the folks there. Tons of small experiments with LLMs can use the $10k in compute

It prohibits anything that competes with OpenAI services i.e as long as you're not literally providing an LLM API commercially you should be fine

This means it is way cheaper to look something up in a vector store than to ask an LLM to generate it. E.g. “What is the capital of Delaware?” when looked up in an neural information retrieval system costs about 5x4 less than if you asked GPT-3.5-Turbo. The cost difference compared to GPT-4 is a whopping 250x!

In a narrow use-case of a strict look-up. This seems to exaggerate the cost difference while having completely different trade-offs.

I think parts of the write-up are great.

There are some unique assumptions being made in parts of the gist

10: Cost Ratio of OpenAI embedding to Self-Hosted embedding

1: Cost Ratio of Self-Hosted base vs fine-tuned model queries

I don't know how useful these numbers are if you take away the assumptions that self-hosted will work as well as API.

10x: Throughput improvement from batching LLM requests

I see that the write up mentions memory being a caveat to this, but it also depends on the card specs as well. Memory Bandwidth / TFLOPs offered by say 4090 is superior while having the same amount of VRAM as 3090. The caveat mentioned with token length in the gist itself makes the 10x claim not a useful rule of thumb.

I've been looking for a course like this! Especially great given how much of the recent progress in training large models is made possible with the aid of flash attention and fused kernels

Was it for seeding/hosting torrents or from just downloading them? How long did the whole thing take to play out?

I've always assumed that consuming torrents has been low stakes to the point where its not worth any enforcement

It is a pretty fun game https://wiki-race.com/

The term "chinchilla" predates llama/alpaca. It doesn't directly map to a specific model, rather a family of compute-optimal models.

That fact that OpenAI decided to not even reveal the parameter counts / training data composition in their "technical" report is surely a sign that their moat isn't as big as it would seem in fear of competition. This paired with the rise of LLaMa and other alternatives makes the monopoly even less likely.

Its just a matter of time till every company has an internal ChatGPT equivalent.

I would also highly recommend the blog post series [1] from Cliqz talking about the tech behind the search.

[1] https://0x65.dev/

I think it is definitely a hard problem to solve on a large scale to address latency, quality and size of the index they plan to address. It definitely isn't as easy as spinning up an elastic search cluster.

I agree that getting something "mostly" good or a domain specific search engine isn't as hard with the newest advances in this space with vector similarity indices.

I've taken Georgia Tech's OMSCS program. What really sets this apart is the active feedback on assignments / course interaction from Professors and TAs in Piazza.

Current online courses have really watered down content and zero/terrible ways of evaluating the material.

Things that I like too see in an online course:

- Keep assignments on a regular basis (on par with your campus schedule). Quizzes are great to keep them engaged but do not use them to replace regular assignments.

- Design them to be auto-graded as much as possible. Bonus points if there was some way the students can evaluate their solution locally.

- Something like Piazza/Forum setup to answer student questions and have an appropriate number of TAs.

- Post all the resources used in the videos to be accessible for later reference (slides and links).

If you're recording content for future offerings, ensure that the content is consistently updated to stay relevant.