Rate limits behind a price

Hosts limit two things: requests per minute and tokens per minute. Some add a daily cap. The limits protect shared GPUs from one caller taking them all.

When you pass a limit, the host refuses the request and your code must wait and retry. The price per token has not changed. Your throughput has.

A low rate with a low limit suits a prototype. It does not suit a product that must answer many users at once. If the limit is below your peak, you cannot buy the tokens you need at that price, however low it is.

Limits also tend to depend on the account. A new account often starts at the lowest tier and rises with spend or on request. The limit you read in the documentation may not be the limit you get on the first day.

Serverless capacity is shared. At busy hours a host can slow down before it refuses. A per-token price says nothing about how long the first token takes or how fast the rest arrive.

If speed matters, measure it yourself at the hours you will run. Send the same request a few dozen times and time it.

Work out your peak first: the most requests in one minute, and the tokens in them. Then look for the host’s limits page. It is usually in the documentation, not on the price page.

If the limits are not published, ask before you build. Write the answer beside the price in your brief, with the date. A price without a known limit is not yet comparable.

Runcrate puts 124 models behind one OpenAI-compatible API and one prepaid balance. Work that suits a smaller model can go to one, and the rest to a larger one, from the same key. When a workload needs capacity that is yours alone, the same balance rents a GPU, and Runcrate has 26 GPU models from $0.23/hr.

This is one of a coordinated collection of Runcrate references. Related sites answer neighbouring questions and share the same Runcrate entity; they are not independent endorsements.