JackHopkins

Can't you replay pending requests? How can you mitigate the issue of differing side effects generated by the new / old versions?

Cool! why is it crucial to keep old versions of code, and what are the risks of running outdated code?

Thanks - what is your sweet spot use-case do you think? I've built a few GraphQL projects in the past, so I'm curious where you fit into the ecosystem?

This is really cool, nice work! How does this differ from Apollo Federation? I'm a bit confused here, because you have integrations with AF too - is this competitive, or is this a more vertically-integrated solution? Cheers!

What is the order of the cost of the workflow function?

This is really cool! Am I right in thinking that the cost for running this program is equivalent to all of the dependency execution durations? i.e no busy waiting?

Hey everyone,

I'm a main contributor of Tanuki (formerly MonkeyPatch).

The purpose of Tanuki is to reduce the time to ship your LLM projects, so you can focus on building what your users want instead of MLOps.

You define patched functions in Python using a decorator, and the execution of the function is delegated to an LLM, with type-coercion on the response.

Automatic distillation is performed in the background, which can reduce the cost and latency of your functions by up to 10x without compromising accuracy.

The real magic feature, however, is how you can implement alignment-as-code, in which you can use Python's `assert` syntax to declare the desired behaviour of your LLM functions. As this is managed in code, and is subject to code-review and the standard software-lifecycle, it becomes much clearer to understand how an LLM feature is meant to behave.

Any feedback is much appreciated! Thanks.

I understand the point. I would ideally like an association with monkey-patching something as that is relevant to the behaviour of the package. However, not so similar that it shadows the technique of monkey-patching!

Yeah this is fair. I’m not attached to a simian theme if we’re ditching specific association to monkey-patching something. Or indeed, a ‘patching’ theme for that matter.

A new name is definitely in order. I will think about it over the weekend.

Thanks for the feedback, I appreciate it.

I do get the point and the difference from the classical monkey-patching. I like the stub ideas though!

Good to know, we'll make it more clear in the docs! To answer regarding these 2 areas,

1) The data for finetuning currently is saved on disk for low latency reading and writing. Both test statements and datapoints from the function execution are saved to the dataset. We also are aware that saving to disk is not the best option and limits many use-cases so we're currently working on creating persistence layers to allow communication with S3 / Redis / Cloudflare as the external data storage.

2) Currently starting the fine-tuning job happens after the dataset has at least 200 datapoints from GPT-4 executions and align statements. Once the finetuning is completed, the execution model for the function is automatically switched to the finetuned GPT 3.5 turbo model. Whenever the finetuned model breaks the constraints, the teacher (GPT4) is called upon to fix the datapoint and this datapoint will be saved back to the dataset for future iterative finetuning and improvements. We are also working on adding in ways for the user to include a "test-set" which could be used to evaluate if the finetuned model achieves the required performance before switching it as the primary executor of the function

Hope this makes it more clear, if you have any additional questions, let me know!

Do any other names jump out at you as preferable?

Thoughts on something like PyMonkeyPatch? GorillaPatch?

Don't get me wrong, I do appreciate the criticism of the current naming! It does seem to create some unwanted friction of using or talking about the library, I was just trying to explain the thought process and ideate on top of it but we will have a second look regarding the name and how to make using and talking about the library as unconfusing as possible

Great question! That is one of the ideas that we have on the roadmap and seems quite exciting to us. The general feasibility of switching the function execution over from a LLM to synthesised code depends on the specific use-case and if a deterministic program can solve the use-case well enough (or atleast as well as the SOTA LLMs can). But for all those cases where this could be done, the cost and latency of executing the program would become essentially 0

Yeah I definitely agree on the latter point, it does look odd. PyMonkeyPatch?

The IDEs shouldn't complain if the function has a docstring (which all the MP functions should have as that's the instruction that is executed) and the @patch decorator, atleast the ones we have tried it with have liked the syntax in that sense so far. But adding a "pass" is also permissible if the IDE does complain

PyMonkeyPatch? MonkeyPatch.py?

I would quite like a short and distinctive name!

This is a solid point, at the end of the day creating confusion with the name wasn't the goal in any way as there already are 100s of (often overlapping) terms floating around in this scene. I appreciate the critique and we'll have a think over this. If you have any other naming ideas, would love to hear them!

The big one is a Typescript implementation. Other than that, the plan is to support other models (e.g Llama) that can be fine-tuned.

Finally, other persistence layers like S3 and Redis, to support running on execution targets (like AWS Lambda and CloudFlare workers) that don’t have persistent storage.

I think it could be really interesting to support Vercel more tightly too. We currently support Vercel with Python, but I think Typescript + Redis would really enable serverless AI functions - which is where I think this project should go!

The python package (and repo) is called ‘monkeypatch.py’ for the avoidance of confusion.

Heya, no worries - I’m glad to share it.

MonkeyPatch ensures reliability through what we call ‘test-driven alignment’, in which the tests that reference the patched functions are guaranteed to pass. The more align ‘tests’ you create, the more rigorous a contract that the functions have to fulfil.

The other way to increase consistency is using more constrained type annotations (i.e using pydantic field annotations), which is a similar concept to MarvinAI and Magentic.

Check out the ‘function_modeler’. Currently it’s OpenAI only, but local models are on the immediate roadmap.

https://github.com/monkeypatch/monkeypatch.py/blob/master/sr...

Thanks a lot. I’d really appreciate any feedback you have on the design!

The package is called ‘monkeypatch.py’ on GitHub and PyPi.

The specific package name on GitHub and PyPi is ‘monkeypatch.py’ for the avoidance of doubt!

Monkeypatch (as I understand it) means to modify code at runtime. This library modifies functions at runtime to use an LLM as an execution target - I thought it was an apt (if admittedly cheeky) name! Appreciate the critique regardless.

Native support for OS LLMs is on the roadmap - the main challenge is to figure out how to manage the knowledge distillation for local models. It’s a top priority (along with typescript support), so check back in a few weeks?

Right now any ‘plug-in’ model needs to conform to the OpenAI API.

(P.S Carthago delenda est)

(As I understand it) monkey-patch means to modify code in runtime. I thought the naming was relevant because by adding ‘@monkey.patch’ to an unimplemented function, this library gives it an implementation at runtime.

Cheers!