Review driven development for ML / LMM applications

https://www.lumoa.me/blog/machine-learning-development-a-comprehensive-review-of-booktest-and-testing-tools/

by arauhala • 2 years ago

1 1 2 years ago

arauhala• 2 years ago

Author here. The article seeks to answer the question "How to make ML / LMM development faster with higher quality?"

The long format article seeks address the pain points in current tooling (Jupyter Notebook, unittests) used to support intelligent application RnD. It proposes a new approach for tooling and development, which combines the benefits of notebooks (review driven process, caches) with benefits of unit testing (repeatiability, regression testing)

The tool has been succesfully used to support the developmvent topic models, analytics and GPT based functionality. Here's an example of how to create a simple test that both creates a snapshot for results that also snapshots the environment and (e.g. GPT) API calls so that the test interaction can be replayed e.g. in CI.

    import booktest as bt
    import os 
    import requests
    import json

    @bt.snapshot_env("HOST_NAME")
    @bt.mock_missing_env({"API_KEY": "mock"})
    @bt.snapshot_requests()
    def test_requests_and_env(t: bt.TestCaseRun):
        t.h1("request:")

        host_name = os.environ["HOST_NAME"]
        response = (
            t.t(f"making post request to {host_name} in ").imsln(
                lambda:
                requests.post(
                    host_name,
                    json={
                        "message": "hello"
                    },
                    headers={
                        "X-Api-Key": os.environ["API_KEY"]
                    })))

        t.h1("response:")
        t.tln(json.dumps(response.json()["json"], indent=4))

https://github.com/lumoa-oss/booktest/blob/main/getting-star...

Review driven development for ML / LMM applications

Related Stories