activatedgeek

Congratulations on the strong reception of min-p. Very clever!

We may be talking about two orthogonal things here. And also to be clear, I don't care about theoretical guarantees either.

Now, min-p is solving for the inadequacies of standard sampling techniques. It is almost like a clever adaptive search which other sampling methods fail at (despite truncations like top-k/top-p).

However, one thing that I noticed in the min-p results was that lower temperatures were almost always better in the final performance (and quite expectedly the inverse for creating writing). This observation makes me think that the underlying model is generally fairly good at ranking the best tokens. What sampling allows us is a margin-for-error in cases where the model ranked a relevant next token not at the top, but slightly lower.

Therefore, my takeaway from min-p is that it solves for deficiencies of current samplers but its success is not in contradiction to the fact that logprobs are bad proxies for semantics. Sampling is the simplest form of search, and I agree with you that better sampling methods are a solid ingredient to extract information from logprobs.

That has been my understanding too. More generally, a verifier at the end certainly helps.

In our paper [1], we find that asking a follow up question like "Is the answer correct?" and taking the normalized probability of "Yes" or "No" token (or more generally any such token trained for) seems to be best bet so far to get well-calibrated probabilities out of the model.

In general, the log-probability of tokens is not a good indicator of anything other than satisfying the pre-training loss function of predicting the "next token." (it likely is very well-calibrated on that task though) Semantics of language are a much less tamable object, especially when we don't quite have a good way to estimate a normalizing constant because every answer can be paraphrased in many ways and still be correct. The volume of correct answers in the generation space of language model is just too small.

There is work that shows one way to approximate the normalizing constant via SMC [2], but I believe we are more likely to benefit from having a verifier at train-time than any other approach.

And there are stop-gap solutions to make log probabilities more reliable by only computing them on "relevant" tokens, e.g. only final numerical answer tokens for a math problem [3]. But this approach kind of side-steps the problem of actually trying to find relevant tokens. Perhaps something more in the spirit of System 2 attention which selects meaningful tokens for the generated output would be more promising [4].

[1] https://arxiv.org/abs/2406.08391 [2]: https://arxiv.org/abs/2404.17546 [3]: https://arxiv.org/abs/2402.10200 [4]: https://arxiv.org/abs/2311.11829

Reasoning tokens are indeed billed as output tokens.

While reasoning tokens are not visible via the API, they still occupy space in the model's context window and are billed as output tokens.

From here: https://platform.openai.com/docs/guides/reasoning

This effect is very interesting.

Veritasium covered this effect in a video [1] for the interested.

[1] https://www.youtube.com/watch?v=aIx2N-viNwY (2016)

I use Astro + Cloudflare Pages for my website [1]. I document the key bits of my stack here [2] for completeness.

I've been very happy with Astro because it is a good example of low floor and high ceiling software. I can start with plain HTML, make it more flexible with Astro language (still very close to HTML), make authoring easier with Markdown (+ lifestyle extensions from Remark/Rehype), and extend to frameworks like React on a need basis (which I use for some pages where I use maps).

[1] https://sanyamkapoor.com [2]: https://sanyamkapoor.com/kb/the-stack

The best thing that one can do for themselves to develop the creative "muscle" is to _own_ their time.

Unfortunately, I am yet to feel even close to such a breakthrough. I think very few are fortunate to afford such kind of luxury (as the author alludes to as well). There is always something to deliver for, a deadline to meet (although many would argue deadlines are a forcing constraint); a life waiting to happen. With a tiny bit of envy, I feel very happy and inspired when someone does achieve the "flow" state.

On the subject of "tools" to spur creativity, I have always been skeptical. It feels similar to believing that there is a productivity app right around the corner that will unleash your potential. For me, the only true indicator of my productivity has been actually putting in the _time_, making any kind of progress along a chosen direction and then re-evaluating.

What are fellow readers here doing to _own_ their time?

The fact that Pyinfra does not currently support a feature which can be implemented using Pyinfra philosophy does not make it different than Ansible. I believe that was what the parent comment was about.

Any kind of provisioning doesn't seem too far a step though. It is just another "operation" with its own state management logic.

I current use Ansible to setup both local and remote hosts. I've been very happy with it, and love that Pyinfra intends to support the Ansible connector.

My main gripe with Ansible is the YAML specification. Ansible chooses to separate the task specification and task execution. Pyinfra chooses to directly expose the Python layer, instead of using slightly ugly magic functions/variables. I like this approach more since it allows standard Pythonic control flow instead of using a new (arguably ugly and more hassle to maintain) grammar.

Excited for Pyinfra!

If you use HuggingFace models, then a few simpler decoding algorithms are already implemented for `generate` method of all supported models.

Here is a blog post that describes it: https://huggingface.co/blog/how-to-generate.

I will warn you though that beam search is typically what you do NOT want. Beam search approximately optimizes for the "highest likely sequence at the token level." This is rarely what you need in practice with open-ended generations (e.g. a question-answering chat bot). In practice, you need "highest likely semantic sequence," which is much harder problem.

Of course, various approximations for semantic alignment are currently in the literature, but still a wide open problem.

Makes sense. Thank you for a the QS reference!

I hesitate to the use description as "think," just biasing correlations for subsequent generations.

In any case, there is at least one work that shows that CoT may not be necessary and biasing the decoding path via logit probabilities is also promising. [1]

One could argue it still doesn't contradict the benefits of CoT, but I suspect there is nothing fundamental about CoT, except that we happened to have been pre-training on sequences that use certain prompts that were easy to conceive from a human's perspective.

[1] https://arxiv.org/abs/2402.10200

I see. Following up on this, for the sake of being explicit: was the bottleneck here getting all the data sources in place (perhaps for instance access permissions, legal, etc.), writing the SQL query, both, or something else?

I want to point out a tweet [1] that is very relevant to the miracle of CoT, and probably a simpler explanation.

  > Let's think "step by step"!

  > Another tidbit I like about data and prompts that miraculously work.
  > Searching for this phrase resulted in this website (among others),  
  > http://geteasysolution.com, containing many math step-by-step solutions. 
  > How common are they? Quite.

  > Makes you think.

In AI/ML research, text to SQL always sounded to me of merely academic interest, in the sense that the outputs are easily verifiable and make for a good proof of concept of a language model's (or a translation model's) capabilities.

But looks like there are plenty of products coming out in this area, and it has me wondering: what is the actual big picture for enterprises here?

I would assume enterprises employ enough people to write yet another query for whatever use case.

- Is the expectation that in the future, we can bring the flexibility of SQL-like languages to people unfamiliar with SQL?

- Perhaps a salesperson unfamiliar with SQL would like to conduct an analysis. Is the volume and variety of such queries so high that optimizing for the turnaround time from an SQL query designed by data analyst to the salesperson to consume the results is so worthwhile?

Perhaps I am underestimating the scale of the problem but would love some insider perspective here.

I looked at Ollama before, but couldn't quite figure something out from the docs [1]

It looks like a lot of the tooling is heavily engineered for a set of modern popular LLM-esque models. And looks like llama.cpp also supports LoRA models, so I'd assume there is a way to engineer a pipeline from LoRA to llama.cpp deployments, which probably covers quite a broad set of possibilities.

Beyond llama.cpp, can someone point me to what the broader community uses for general PyTorch model deployments?

I haven't quite ever self-hosted models, and am really keen to do one. Ideally, I am looking for something that stays close to the PyTorch core, and therefore allows me the flexibility to take any nn.Module to production.

[1] https://github.com/jmorganca/ollama/blob/main/docs/import.md.

LibraryThing [1] also has a local book search on each book's page.

[1] https://www.librarything.com/home

Thanks for the reference, Lakshya. Looks very cool!

(Just thinking out loud next)

If you allow me to be a little imprecise, guided-generation is prompting "just-in-time" unlike the other kind of prompting where you provide all reference tokens "ahead-of-time". Now there's work [1] out there that shows that smaller models rely much more on prompting than larger models do, i.e. smaller models are more faithful to the tokens in the prompt than the larger models which just do whatever they were going to do anyways.

Your results seem very much in line with this kind of a qualitative result --- you show that CodeGen-350M outperforms CodeGen-6B, and CodeGen-6B outperforms text-davinci-003 using MGD. Smaller models perhaps respond more strongly to certain kinds of prompting strategies than larger models do.

[1] https://arxiv.org/pdf/2307.13702.pdf

I'm not sure of why you would want to use raw llama-2

Sure. My concern was not specific to llama-2, and was only using it as a placeholder example of a decent pre-trained base model. Replace it with your favorite base model, which you want to use for guided generation. My question is more fundamental - how does post-hoc guided generation interfere with the potential benefits of instruction-tuning?

About your second point, the goal is that the model can only generate JSON (for example), which can 100% be done by constraining which output token can and cannot be used.

Mechanistically, yes. I am not arguing that. The whole point is to generate JSON that is "useful".

I think what I am saying is tangential to TDD. I am not really even concerned about the ability of LLM to function as desired, and its verification.

I was rather concerned about a broader fundamental question - how does post-hoc guided generation interfere with the potential benefits of instruction-tuning?

Mechanistically, I think this library takes the simple idea of masking part of the vocabulary space and steps in time efficiently. Great!

I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get things to work.

And even then, it seems very counter-intuitive to me that given an instruction-tuned model, post-hoc masking of the state-space during generation then amounts to just changing the generation distribution, and potentially detrimental to instruction-tuning?

Another excellent book that I recently read was "Information Theory: A Tutorial Introduction" by James V. Stone [1]. This book is a fairly light read and builds up everything from scratch which was very nice.

Shannon's original paper [2] is often touted to be very parseable, but it is always enlightening to read the same basics many times from multiple sources, especially something as foundational as source coding (and broader information theory) in a more modern language.

I also finally discovered the meaning of "differential" in differential entropy [2], the continuous counterpart of discrete probability masses, something that I always swept under the rug.

[1] https://www.librarything.com/work/16165301/book/242295611 [2]: https://ieeexplore.ieee.org/document/6773024 [3]: https://sanyamkapoor.com/kb/differential-entropy

Amazing!

Oddly, two tests fail for me with Brave (Version 1.51.118 / Chromium: 113.0.5672.126 (arm64)) on macOS Ventura 13.3.1

- pow([0], [0]) gradient, with "Expected «-Infinity» to be close to «0» (diff: < 0.0000005)"

- xlogy([0], [0.30000001192092896]) gradient with "Expected «0» to be close to «-1.2039728164672852»"

Looks like r/sweatystartup, A hub for entrepreneurs of regular old fashioned businesses, is more active.

What are some examples of efficient packing results that led to or inspired practical solutions?

We are starting to see this kind of work already, e.g. https://code-as-policies.github.io

Opened the comments to put in a recommendation for NextDNS. The free plan allows 300k DNS queries and is more than enough for me across three devices (Mac/iPhone/iPad).

I have not needed an ad-blocker for the past 2 years and hardly see any websites breaking. Often syndicate links break, which is not a big deal for me.

Thank you!

The typography/layouts come from Tailwind Typography [1]. I like their defaults.

[1] https://tailwindcss.com/docs/typography-plugin

https://sanyamkapoor.com

With my most recent iteration, I have started using my website more as a mental decluttering tool than a show-tell medium. The wesbite is a bunch of markdown files spiced up with Next.js and my first attempt at structured navigation. I keep a similar private one.

I've now been rethinking of reorganizing into a flat heirarchy + what I like to call "theme portals" - essentially big picture entrypoints into details. This avoids a long list of scattered blog posts. The orrganization style of clean overarching themes like tech/math etc. isn't working for me.

Very ambitious! Good luck!

A small bug in the call-to-action at the bottom of the page - If I upvote only "Stripe" in the "Connect your Shops" section, the call-to-action reads:

Want to know when Stripe, Stripe, and more integrations are ready?

Voting for greater than one shop resolves this repetition error.