FlyingLawnmower

Very nicely written guide!

If the authors or readers are interested in some of the more technical details of how we optimized guidance & llguidance, we wrote up a little paper about it here: https://guidance-ai.github.io/llguidance/llg-go-brrr

Good point re: documentation...

We have support for Huggingface Transformers, llama.cpp, vLLM, SGLang, and TensorRT-LLM, along with some smaller providers (e.g. mistral.rs). Using any of these libraries as an inference host means you can use an OSS model with the guidance backend for full support. Most open source models will run on at least one of these backends (with vLLM probably being the most popular hosted solution, and transformers/llama.cpp being the most popular local model solutions)

We're also the backend used by OpenAI/Azure OpenAI for structured outputs on the closed source model side.

guidance can handle many context-free grammars. We use an Earley parser under the hood (https://en.wikipedia.org/wiki/Earley_parser) which gives us significant flexibility boosts over alternative approaches that use weaker parsers (and went through lots of effort to make Earley parsing fast enough to not slow down LM inference). However, XML is not perfectly context-free, though with some basic assumptions you can make it CF.

The annoying bit with grammars is that they are unfortunately a bit complex to write properly. Fortunately language models are getting better at this, so hopefully to get an XML grammar, you can get most of the way there with just a GPT-5 prompt. Suppose it would be a good idea to have a better pre-built set of popular grammars (like a modified XML) in guidance so that we cut this headache out for users...!

We did quite a thorough benchmarking of various structured decoding providers in one of our papers: https://arxiv.org/abs/2501.10868v3 , measuring structured outputs providers on performance, constraint flexibility, downstream task accuracy, etc.

Happy to chat more about the benchmark. Note that these are a bit out of date though, I'm sure many of the providers we tested have made improvements (and some have switched to wholesale using llguidance as a backend)

Thanks :)

Great question re: adoption...it's definitely dominated by JSON. Most API providers have standardized on JSON outputs, so application teams have started building shims that map other formats to JSON and back. Similarly, with models heavily being post-trained to generate "good" JSON, I think there's a better model-constraint alignment story with JSON than most arbitrary grammars.

That said, internally, we experiment quite a lot with custom grammars all across the stack. It's more complicated to write a grammar than a JSON schema (though LMs are very good at grammar writing now) and more error prone to debug, but it can help significantly in certain cases (e.g. having models write custom DSLs not commonly found on the internet, at various parts of a model training pipeline, etc. etc.). I'm hoping that with the right tooling around it, the broader community will start nudging beyond JSON.

To that end, the python guidance library is really an attempt to make writing grammars more friendly to a python programmer. More to be done here of course!

If you can screen tokens against your grammar fast enough, you can build a bitmask over the entire token vocabulary and apply it right before sampling. As vocabulary sizes grow, this gets more complex to do in real time, but we (and other libraries) have found several optimizations to do this extremely quickly (eg for guidance, we detail some optimizations here https://github.com/guidance-ai/llguidance/blob/main/docs/opt...).

Other libraries work by essentially pre-computing all the masks for all possible generations, but of course you're restricted to working with simple grammars in this case (like a subset of regular expressions)

If your masking is fast enough, you can make it easily work with spec dec too :). We manage to keep this on CPU. Some details here: https://github.com/guidance-ai/llguidance/blob/main/docs/opt...

OpenAI has started to (at least for tool calls): https://platform.openai.com/docs/guides/function-calling#con...

We do enable forcing these sequences of tokens in guidance, and find that it significantly speeds up structured generation. There are tricky alignment issues to make sure you pick the right sequence of tokens, but you can often proxy this well by using the model's native tokenizer. Some details here in an old blog: https://guidance.readthedocs.io/en/latest/example_notebooks/...

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works:

User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/

Core Rust library written for high performance mask computation (written mostly by my collaborator @mmoskal): http://github.com/guidance-ai/llguidance

Sidenote, but the scholarship on distillation always makes me a bit sad. The Original work, cited in the abstract of the Hinton, Vinyals, and Dean paper that is cited everywhere, was the model compression work from Caruana, Buciluǎ, and Niculescu-Mizil.

The distillation paper added minor parameter tweaks and had a fancier name, but the essence of the method came from Caruana et. al's model compression paper: https://dl.acm.org/doi/abs/10.1145/1150402.1150464

Yes. They are about entirely different patient reports.

Artificial intelligence holds great promise for expanding access to expert medical knowledge and reasoning. However, most evaluations of language models rely on static vignettes and multiple-choice questions that fail to reflect the complexity and nuance of evidence-based medicine in real-world settings. In clinical practice, physicians iteratively formulate and revise diagnostic hypotheses, adapting each subsequent question and test to what they've just learned, and weigh the evolving evidence before committing to a final diagnosis. To emulate this iterative process, we introduce the Sequential Diagnosis Benchmark, which transforms 304 diagnostically challenging New England Journal of Medicine clinicopathological conference (NEJM-CPC) cases into stepwise diagnostic encounters. A physician or AI begins with a short case abstract and must iteratively request additional details from a gatekeeper model that reveals findings only when explicitly queried. Performance is assessed not just by diagnostic accuracy but also by the cost of physician visits and tests performed. We also present the MAI Diagnostic Orchestrator (MAI-DxO), a model-agnostic orchestrator that simulates a panel of physicians, proposes likely differential diagnoses and strategically selects high-value, cost-effective tests. When paired with OpenAI's o3 model, MAI-DxO achieves 80% diagnostic accuracy--four times higher than the 20% average of generalist physicians. MAI-DxO also reduces diagnostic costs by 20% compared to physicians, and 70% compared to off-the-shelf o3. When configured for maximum accuracy, MAI-DxO achieves 85.5% accuracy. These performance gains with MAI-DxO generalize across models from the OpenAI, Gemini, Claude, Grok, DeepSeek, and Llama families. We highlight how AI systems, when guided to think iteratively and act judiciously, can advance diagnostic precision and cost-effectiveness in clinical care.

https://chatgpt.com/share/6760a122-0ec4-8008-8b72-3e950f0288...

My first try with o1. Seems right to me…what does this teach us about LLMs :)?

Interesting paper, but their reason for dismissing constrained decoding methods seems to be that they want to academically study the in-context setting.

For practitioners, using a framework like Guidance which forces the models to write valid JSON as they generate text solves this trivially (https://github.com/guidance-ai/guidance)

For json in particular these frameworks have functions that take in json schemas or pydantic schemas https://guidance.readthedocs.io/en/latest/generated/guidance...

Last I checked, physician pay represents a fairly small (single digit) fraction of healthcare expenditure. If I recall right, administrative overhead and insurance is significantly more of a contributor to pricing.

Late to the thread here, but the paper announcing Med-PaLM (https://arxiv.org/abs/2212.13138) does not report many benchmark results on Med-PaLM and is instead mostly about Flan-PaLM 540B (which is compared against in this paper). I am curious if any other Med-PaLM benchmarks have been published but I don't believe it is currently possible to do any further comparison against Med-PaLM given that the model is not public and no other open benchmark results are reported in the original Med-PaLM paper.

"We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks...results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct."

Do you have some sources where I can read up more on this? I have always believed that the cost of front line worker salaries (Doctors/Nurses) was a relatively small % of the total cost of healthcare.

Studies like this [1] led me to believe that massive rises in administration costs and inefficiencies due to insurance structure are bigger culprits than doctor/nurse salaries. But I am admittedly not very well informed in this area.

[1] https://pubmed.ncbi.nlm.nih.gov/31905376/

Have you thought about using central/global differential privacy (which tends to have much less noise) on the "high level aggregates" or "aggregated datasets" that persist after the research study ends?

E.g. from the FAQ: "We do intend to release aggregated data sets in the public good to foster an open web. When we do this, we will remove your personal information and try to disclose it in a way that minimizes the risk of you being re-identified."

It's a little worrying to think that this disclosure process might be done with no formal privacy protection. See the Netflix competition, AOL search dataset, Public Transportation in Victoria, etc. case studies of how non-formal attempts at anonymization can fail users.

Check out this work from Rich Caruana & collaborators on model compression: http://www.cs.cornell.edu/~caruana/compression.kdd06.pdf

which was a precursor to the model distallation work from Geoff Hinton: https://arxiv.org/abs/1503.02531

Hats off to people like Carlos Guestrin & John Giannandrea, who seem to have pushed a cultural shift through Apple. I didn't think I'd see a site like this a few years ago, based on Apple's historic reputation of strongly discouraging employees from publishing research.

I don't understand how Facebook got a $5B fine, yet Equifax gets a ~$650m fine. The data breached in the Equifax case seems to cause far more direct harm, and affected many more Americans. It feels like the 10x difference should go the other way.

Can someone more educated in how these fines work teach me about how these numbers are calculated?

Can I ask what your business model or pathway to monetization is?

Thanks for making it run locally, I can finally give it a real try now.

I like the sentiment, but unfortunately predefining a set of categories doesn't prevent the issues. Even without access to a specific attribute like "wealth" or "race", the learning algorithms used here can still be used to target poor people, minorities, etc. by leveraging correlations within the dataset. Without these features, the bias just gets dispersed through other correlated variables, and makes it harder to detect that an algorithm is disproportionately targeting poor people.

Additionally, even a modest list of attributes can be sufficient to uniquely identify someone.

I think limiting collection is a good thing in general, but to meet your stated goals, we unfortunately have to do more.

Thanks for the site! Do you have thoughts on removing all the print/console.logs calls on every line? It feels like there's a lot of visual noise there, that can still be met with the comments showing state/output.

Thank you for sharing your experience and journey with us. It's disheartening to see that your courage in speaking up and sharing is rewarded with such a negative attitude. I appreciate your attitude.

I don't have much experience using HE libraries, but I know another (fairly popular) one is SEAL from Microsoft Research: https://www.microsoft.com/en-us/research/project/simple-encr... Perhaps it has the support you need?

Hi Jon,

Just want to say I absolutely love plotly. It's made data science and visualization fun again. Thank you for all your hard work!

I really like the idea of libraries that make it easy to learn a data structure/algorithm/general concept. Thank you for publishing this! It will certainly make CS concepts easier to teach.