50% discount on OpenAI pricing if you submit a batch and give them up to 24h

https://twitter.com/simonw/status/1779932174592167965

by tosh • 2 years ago

92 65 2 years ago

twitter.com

9999000009993• 2 years ago

How does this work, do I submit a rest call and get an IOU response to look up the results later?

tedsanders• 2 years ago

Copied from OpenAI's Jeff Harris on Twitter:

1) Upload your batch file

curl https://api.openai.com/v1/files -H "Authorization: Bearer $OPENAI_API_KEY" -F purpose="batch" -F file="@batch_example.jsonl"

2) Create the batch job

curl --request POST --url https://api.openai.com/v1/batches --header "Authorization: Bearer $OPENAI_API_KEY" --header 'Content-Type: application/json' --data '{ "input_file_id": "file-DYJImtYAQ2y3j25b5F27Eefp", "endpoint": "/v1/chat/completions", "completion_window": "24h" }'

3) Check the status of your job

curl https://api.openai.com/v1/batches/batch_f5Hh3MXaM0NNuTGOmutl... -H "Authorization: Bearer $OPENAI_API_KEY"

4) Download the completed file!

curl https://api.openai.com/v1/files/file-Og2t4LcJOCqOFq6rmp22Uzk... -H "Authorization: Bearer $OPENAI_API_KEY" > batch_output.jsonl

9999000009993• 2 years ago

Cool, not really looking to do this just yet, but it's definitely on my radar the next time I have a project.

Would love an option to look up all my open jobs. I imagine doing this in a ci/cd pipeline or GitHub action. But I don't want to have to store state.

silvacarl• 2 years ago

this is crazy cool, will try it out shortly.

i need to batch anaylze 100K whisper transcriptions. 8-)

mike_hearn10• 2 years ago

It's a pity that this sort of feature is done with a custom ad-hoc HTTP protocol instead of using a message queuing protocol like AMQP or STOMP. Those are designed for this kind of thing and already have libraries available, they avoid the need for busywork polling and avoid the need to break work up into batches client side. You should just be able to submit messages containing work to a queue and then let the broker inform you when new responses arrive on the response queue if you're connected. If you're not connected they buffer up for a while until you reconnect and drain the queue.

The advantage of that approach would be that latency could be minimized or tiered. Instead of a single "one batch with maybe up to 24hr latency" the offer could be a series of tiered queues with different SLAs and costs, and you can pick which you submit your work to depending on how important it is.

I wonder why they went for this custom JSON/HTTP approach instead. Maybe they feel that they can't expect developers to understand anything other than POSTing JSON.

dmw_ng2• 2 years ago

There's a reason pretty much everything that does not require low latency replies avoids stateful networking - everything from RSS to video streaming prefers stateless polling designs because it is vastly easier for both parties to implement and scale. Meanwhile, I couldn't name a single system in widespread use built around a MQ paradigm in its public interface, except for actual MQ APIs, and many of those (e.g. from AWS) are still built on polling for the reasons just described

jerf2• 2 years ago

Behind each request made to OpenAI is a staggering amount of GPU computation. If the price of the queue request is even a hundred thousandth of the overall price of a single request I'd be stunned. There is no message queue scaling issue here. Message queue scaling issues arise when you are blasting around a lot of messages, but each of them take minimal resources on an individual basis to service, so it's feasible for the queue itself to be the bottleneck. I wouldn't be surprised a single Raspberry Pi could handle the entire queuing load here, and if it couldn't it's not off by a very large factor, because the GPU resources behind what it would take to service a full RPi's queuing capacity would be staggeringly enormous, I think well beyond what OpenAI actually has.

btown• 2 years ago

Dropbox's approach here in the early 2010s (not sure if still used) was actually quite clever. Both official clients and third parties could open up a long-timeout HTTP connection that wouldn't be responded to until the web server was delivered a message in their internal message queue. Avoided the overhead of polling and allowed for extremely low latency, while letting clients still use their favorite HTTP library.

https://dropbox.tech/developers/low-latency-notification-of-...

dec0dedab0de• 2 years ago

webhooks are a little closer in that they remove the need for constant polling, atleast for longer running processes

mike_hearn10• 2 years ago

Think about every messenger out there, any Slack or Slack-like app that uses WebSockets, email clients that use IMAP etc. They aren't polling once a minute like it's 1995.

It's not really easier to implement or scale, in my view. It may seem that way if you've never worked on large scale stateful connection serving. If you want users to get the answer as soon as it's ready, and you do or at least your users do, then you need users to hold open connections and at that point it's really a question of how much your protocols and libraries do for you. If you use HTTP the answer is "not much" because it was designed to download hypertext. The actual task of managing lots of connections server side isn't especially hard. There are MQ engines that support sharding and can have a regular TCP LB like a NetScaler stuck in front of them, or of course, you can just implement client-side LB as well.

simonw2• 2 years ago

That sounds really difficult to use to me. I'd need to figure out how to connect to a custom messaging protocol, and how to keep that connection online (maybe routing through firewalls etc), and handle retries and connection failures and suchlike.

An HTTP polling API may be less efficient but I know how to use that from all of the programming languages I work with already.

alexchamberlain• 2 years ago

FWIW, AMQP is the same protocol as RabbitMQ- there's a library for pretty much every language/ecosystem.

mike_hearn10• 2 years ago

AMQP isn't a custom messaging protocol and MQ libraries do all of that for you already (handling reconnects and retries etc). Connection interruptions aren't an issue. You're actually worrying about low level details of the sort MQ systems are specifically designed to solve for you, but HTTP stacks do not.

For example, someone below asks "how do I see my submitted jobs". MQ protocols solve that for you out of the box. You can browse the queues that you own. It's all already standardized and implemented. Because they rolled their own protocol, OpenAI lacks this function and will have to develop it themselves. Then because it's all oriented around batches (when the underlying problem does not actually need batching), the users will have to run a local DB to keep track of what prompts map to what batches or they'll find themselves redownloading the whole batch to find out what's in it.

As for firewalls, you open the port. The only issue there is if you're behind a firewall you don't have any control over and where the people who do don't want you using AI, but that's not an issue in the datacenter deployments that are presumably generating inferencing batch workloads, and such firewalls or proxies can break HTTP APIs too anyway.

tuckerconnelly• 2 years ago

Message queues are difficult to maintain.

Webhooks is the best solution IMO.

krisoft• 2 years ago

they avoid the need for busywork polling

The fact that you know that you can only expect an answer after 24h also avoids the need for busywork polling.

Instead of a single "one batch with maybe up to 24hr latency" the offer could be a series of tiered queues with different SLAs and costs, and you can pick which you submit your work to depending on how important it is.

But that is what they have. They have one "ASAP" API and one "within 24h" one. These are two big distinct use cases. You use the fast one when there is a user waiting for an answer realtime, and you use the slow one when nobody is waiting for it realtime.

These are two distinct use cases from the user's perspective.

You are correct in identifying that there could be an API which offers 2h returns, or one which offer 14day returns, but why would you complicate the offering with that? It adds a lot of complication to the documentation, a lot of complication on the scheduling side and a lot of complication on the pricing side, and for what upside?

mike_hearn10• 2 years ago

There's almost no complexity on the scheduling side done properly. Just pop work off the queues in priority order.

There are use cases for multiple latency tiers

1. At the lower end you have realtime-ish responses for when the chat is via SMS/WhatsApp/tickets/email, i.e. the social convention is that responses may be delayed by a few minutes or even 10-20 minutes and that's reasonable, but not by 24 hours.

2. At the high end you have a database with 200 million documents, a prompt that works, and you just want to submit all of them in one go as quickly as your client can work, then get the results as they get processed. There is no latency requirement whatsoever, but you'd prefer not to have to chunk things up and track batch IDs and such yourself. What you DO really want though is to have exactly-once submission, so you don't accidentally lose your connection half way through a batch upload and then end up paying for the same batch twice.

These are all basic problems of the type that occur all the time in the IT industry. The solutions are well known and mature. Your bank runs on message queues of various kinds. The tech really isn't that fancy.

willcipriano• 2 years ago

You'd have to hold a ton of connections open for 24 hours+

mike_hearn10• 2 years ago

That isn't hard. They aren't going to serve a billion connections are they, and WhatsApp scaled a full NxN low latency messaging system up with a handful of people.

At any rate, that's only the case if every request actually sits in the queue for 24 hours. The 24 hour period is clearly chosen to let them absorb free capacity overnight (all such services have diurnal cycles), but unless their implementation is really shoddy at any given moment their workers will be popping work off an internal queue anyway to take advantage of moment-sized lulls in traffic. Done properly you can saturate your hardware 100% all the time... there's certainly enough inferencing demand out there.

So at that point the question is just one of external API.

dragonwriter• 2 years ago

I wonder why they went for this custom JSON/HTTP approach instead. Maybe they feel that they can't expect developers to understand anything other than POSTing JSON.

Its simpler for them to build and operate since it fits their existing API (and user validation, quota tracking, etc.) infrastructure, its simpler for them to update their SDK to handle another HTTP endpoint than to build an AMQP or STOMP client into the SDK for this feature, its easier for most consumers who aren't using the SDK, and a message queue interface for this one bit of functionality would offer minimal benefits. Easy choice.

(If you are consuming from an environment where you are alreayd using AMQP or STOMP, sure, build a quick adapter for the HTTP interface. But I don't see any good reason OpenAI should have done one of those instead of HTTP.)

vibl• 2 years ago

Instead of a single "one batch with maybe up to 24hr latency" the offer could be a series of tiered queues with different SLAs and costs

The API includes a `completion_window` parameter ("The time frame within which the batch should be processed. Currently only 24h is supported.").

https://platform.openai.com/docs/api-reference/batch/create

jasonjmcghee• 2 years ago

Original / official tweet: https://twitter.com/openaidevs/status/1779922566091522492?s=...

API Link: https://platform.openai.com/docs/api-reference/batch

victorbjorklund• 2 years ago

Could def be useful for many use cases.

tony_cannistra6• 2 years ago

Suggests to me that they have a LOT of compute that they can't, or have decided not to, scale down. Interesting way for them to keep the infra up and cover costs.

jstummbillig• 2 years ago

It mostly suggests large spikes in usage, no? Which does not seem very surprising.

tony_cannistra6• 2 years ago

Totally. There are other explanations (like wanting to offer a discount), but my mind initally went to "they have a lot of provisioned compute that they pre-paid for, and they want higher-utilization when API request rates are low."

Just hypothesizing.

blackoil• 2 years ago

Or simply that traffic is uneven. Like the Spot instances of EC2.

danielmarkbruce• 2 years ago

I don't think that's right. There is an optimal batch size, and this allows them to just keep everything optimal or close to it for as much time as possible. I'd bet every single minute they miss chances to run things optimally because they need to hit some latency number, and this enables them to not have to make that trade off.

tony_cannistra6• 2 years ago

Yeah, that's another good explanation.

yumraj2• 2 years ago

Not really. It could be they have massive traffic during US daytime hours, which is not evenly matched by traffic from the rest of the world during US night times, so they'll just run this batch during US night time and hence the 24 hour period.

tony_cannistra6• 2 years ago

Yah, that's what I'm saying. In those low periods they could just scale down the infra, but perhaps there's a constraint that doesn't allow for that. This lets them not waste it.

pornel• 2 years ago

Maybe this gives them more control and improves ability to handle latency-sensitive traffic.

Or maybe they’ve found a way to batch process many requests together in a way that’s more efficient.

8organicbits• 2 years ago

For anyone trying to figure out these sorts of pricing questions when running a service in the cloud: I wrote up a blog post about how (d one of) the math works out.

https://alexsci.com/blog/modeling-on-demand-pricing/

hoseja• 2 years ago

Don't spill the card stack!

namaria• 2 years ago

50 years of Moore's law improvements and we're back to batch processing. Nothing illustrates better to me the regression that using massive amounts of GPU to do inference represents as an approach to computation.

beachy• 2 years ago

50 years of Moore's law improvements and we're back to batch processing.

It never went away.

I did some consulting years ago for a large telco. They had an overnight billing batch process that was obviously critical to the company.

It took about 22 hours to run. They were terrified it was going to start taking longer, if it ever reached 24 hours that would be the death of the company.

Marlinski• 2 years ago

Wish this was done over Bundle Protocol, would love to code that for them!

stygiansonic2• 2 years ago

This is probably using their excess capacity, but not necessarily that their GPUs are idle.

For LLMs/large models the huge cost is memory ops to load each layer weights during the forward pass. This is why doing inference at batch size 1 is extremely wasteful: you pay all the mem ops cost and don’t use enough compute FLOPs to justify.

You want a high enough batch size so that compute:mem ops is close to the ratio for that GPU. This is usually done by batching together multiple user requests.

At times of low usage there is excess capacity because batch size is below this “optimal” ratio. So they can slot in these “relaxed SLA” requests for little marginal increase resource usage on their end.

Basically have a queue of these requests that you use to “top off” your batch size when you can.

Edit: also you may not be able to get optimal batch size depending on when the requests arrive, eg you don’t want to wait forever to fill up a batch. So again having a queue of outstanding/delayed requests to serve allows for smoothing things out and increasing compute utilization

tony_cannistra6• 2 years ago

Yeah this makes sense. I do wonder though how it changes the dynamics around provisioned capacity, if at all.

mike_hearn10• 2 years ago

It reduces the need. If they can get non-latency sensitive users onto this API then they only need to be provisioned to support their max interactive query load (ChatGPT) rather than peak API load, which can be arbitrary high (however fast the program generating the load can run). The lower pricing should move users across quite fast, and the higher efficiency will free up hardware and reduce the rate at which they need to grow it.

jedberg• 2 years ago

Hey there, this comment is super insightful. I'd love to talk to you about this some more, but you don't have contact info. If you're interested my contact info is in my profile.

stygiansonic2• 2 years ago

Thanks - I added my contact info (I don’t comment a lot on HN, mostly just read) but will drop you a line

cavisne• 2 years ago

This awesome talk [1] from OpenAI covers this topic quite a bit, one useful takeaway is how GPU compute is basically static, gone are the days of autoscaling as there is nothing to autoscale to.

I think that beyond optimizing batch size, massive training clusters tend to benefit from scheduled maintenance periods where everything gets fixed vs rolling fixes (as you either need everything to be working or you need to restart the training window). If OpenAI could interleave batch inference with training specific HW downtime like interconnect maintenance it would be another basically free source of GPU FLOPS.

[1] https://www.youtube.com/watch?v=PeKMEXUrlq4

fotcorn• 2 years ago

I wonder if they have a clear hardware separation between each of the API, ChatGPT, their lower-scale experiments and their large scale (e.g. GPT5) training hardware. Or is everything just a big blob of hardware, that dynamically gets allocated to jobs depending on demand?

Hardware demand is so high, having GPUs idling is a massive waste, but you also want to have separation between dev, test and prod environments, so not obvious what to do.

ravetcofx2• 2 years ago

We've gone full circle to an era of Mainframe timeshare scheduling tasks.

VBprogrammer• 2 years ago

Give it 10 years and your mid-price android phone will be running models that would make ChatGPT look like baby talk.

ravetcofx2• 2 years ago

Maybe even talking birthday cards with LLM ASICs in a blob

nextworddev• 2 years ago

Probably a response to provisioned capacity inference from aws gcp

50% discount on OpenAI pricing if you submit a batch and give them up to 24h

Related Stories