A sensible information for platform groups managing shared AI deployments

Price Limiting vs. Quota Reservations: when to make use of every

You might have a single gpt-oss-20b deployment. Six groups need to use it. Advertising is working batch summarization jobs at 3am. The fraud staff wants sub-second responses 24/7. An intern’s Jupyter pocket book is by chance hammering the endpoint in a decent loop. And your GPU invoice is already eye-watering.

Sound acquainted? DataRobot offers you two instruments to resolve this: Price Limiting and Quota Reservations. This put up explains when to succeed in for every, backed by an actual load check instance on a staging deployment.

Price Limits and Quota Reservations, in plain English

Price Limits – Out there in DataRobot v11.4

Price limits units per-consumer caps throughout a number of dimensions: requests per minute, token rely per hour, concurrent requests, and enter sequence size. A default coverage applies to all shoppers, with per-entity exceptions obtainable for particular overrides.

What it protects in opposition to: Any single shopper overconsuming — whether or not by way of excessive request quantity, giant inputs, or extreme concurrency.

Quota Reservations – obtainable in DataRobot v11.9

Quota reservations outline the deployment’s complete potential throughput (worth per minute) and a utilization threshold that triggers enforcement. Inside that price range, particular entities might be allotted a reserved proportion — guaranteeing them a minimal slice of capability that different shoppers can’t take away.

What it protects in opposition to: Precedence hunger. With out reservations, a loud neighbor can eat your complete capability price range, leaving your vital workloads with nothing.

How Price Limits and Quota Reservations work collectively (and aside)

Used alone, every device solves a selected drawback:

• Price limiting alone caps complete throughput. Below saturation, all shoppers compete equally — first come, first served.
• Quota reservations alone assure minimal throughput for particular shoppers, no matter what others are doing.

Collectively, they offer you each management surfaces: a ceiling that protects the mannequin and assured flooring for the shoppers that matter most.

Load testing a multi-tenant deployment

To guage these options below stress, we load-tested a gpt-oss-20b deployment in our staging atmosphere. The setup simulates an actual multi-tenant situation: 4 shoppers sharing one mannequin, every with completely different precedence ranges.

Instance configuration

This left a 20% unreserved pool (200 RPM) for the dev person and any overflow.

Instance load profile

We ran six escalating situations over 17 minutes to look at behaviour at completely different saturation ranges:

When to make use of Price Limiting alone

Price limiting by itself is the best selection when:

• All shoppers are equally vital. If no staff’s site visitors is extra vital than one other’s, there’s no want for reservations. Equal competitors below saturation is honest sufficient.
• You simply want to guard the GPU. Your major concern is {that a} spike in site visitors doesn’t degrade mannequin latency or trigger OOM errors. You need a security valve, not a site visitors coverage.
• You might have a single shopper. If there’s just one software hitting the deployment, reservations are meaningless — there’s nobody to order in opposition to.

What the instance confirmed

Throughout the regular site visitors situation (~600 RPM mixed, nicely under the 800 RPM utilization threshold), the speed limiter was invisible and all 4 shoppers achieved 100% success charges with zero rejected requests.

Dimension your reservations based mostly on absolutely the minimal throughput every shopper requires throughout peak competition. That is by design, so that you’re not penalizing regular site visitors.

And it protects the mannequin even below excessive abuse. Throughout the excessive overload situation (20,000+ RPM in opposition to 1,000 RPM capability, which is a a 20× overload), the speed limiter rejected 95% of requests. However the mannequin itself stayed completely wholesome:

The speed limiter acted as a firewall between chaotic shopper demand and steady mannequin inference. With out it, these 20,000 requests per minute would have queued up contained in the NIM, latency would have ballooned, and the mannequin would have successfully change into unusable for everybody.

Takeaway: In case your solely objective is “don’t let site visitors spikes kill the mannequin,” price limiting alone is adequate and zero-config past setting the capability quantity.

When so as to add Quota Reservations

Quota reservations change into important when:

• Some shoppers are extra vital than others. Your fraud detection system can’t afford to be starved out by a batch analytics job. Your manufacturing agent wants assured throughput {that a} developer’s check harness can’t steal.
• You might have a multi-tenant deployment. A number of groups, functions, or downstream deployments share the identical mannequin. With out reservations, the loudest shopper wins.
• You need predictable SLAs. Should you’ve promised a staff “your software will get a minimum of 300 RPM,” reservations are the way you implement that promise on the infrastructure stage.
• You might have a mixture of interactive and batch workloads. Batch jobs are bursty and can fortunately eat all obtainable capability. Reservations guarantee interactive workloads nonetheless get their share throughout batch spikes.

The way to measurement reservations

Dimension your reservations based mostly on absolutely the minimal throughput every shopper requires throughout peak competition.

Guidelines of thumb:

• Don’t reserve 100%. Depart an unreserved pool (10–20%) for ad-hoc site visitors, new shoppers, and overflow. Should you reserve every thing, any new software will get zero throughput till you reconfigure.
• Dimension reservations to minimal wants, not peak wants. Reservations assure a flooring, not a ceiling. An entity with 30% reserved can nonetheless use greater than 30% when capability is accessible.
• Match reservation measurement to enterprise criticality, not staff measurement. Your fraud detection system may need fewer requests than your analytics pipeline, but it surely wants assured entry extra.

In our instance, three manufacturing brokers acquired 30%/20%/30% reservations, leaving a 20% unreserved pool for the dev person. This meant the dev person may nonetheless use the deployment — they simply wouldn’t get assured entry throughout competition.

Do reservations work below actual load?

At slight overload (1.2× capability): The system degrades gracefully

Throughout the slight overload situation (~1,200 RPM in opposition to 1,000 RPM capability), all 4 shoppers achieved 100% success — the token bucket’s burst capability absorbed the slight overage. That is the “sleek degradation” zone the place reservations aren’t but wanted, however the system is proving it will possibly deal with bursts.

At heavy-to-extreme overload (7–12× capability): reservations keep a assured flooring

When all 4 shoppers fired as quick as potential (7,000–12,000 RPM in opposition to a 1,000 RPM capability), the system was overwhelmed. Right here’s what every shopper skilled throughout the total check:

Why the success charges look comparable: At 12× overload, even a 300 RPM reservation is barely ~2.5% of what every shopper is making an attempt to ship (~3,000 RPM per shopper vs. a 300 RPM assure). The reservation works by guaranteeing every shopper receives its assured 200–300 RPM. Nonetheless, as a result of 97% of complete site visitors is rejected throughout excessive overloads, the relative proportion variations compress.

The extra revealing metric is absolute throughput. Reserved shoppers accomplished 4,172–4,332 profitable requests. The unreserved dev person accomplished 2,828 — about 34% fewer. Even accounting for the dev person’s shorter lively time, reserved shoppers constantly acquired extra requests by way of throughout shared situations.

At saturation with a late joiner: reservations defend incumbents

Within the late joiner situation, the three manufacturing brokers had been already flooding the system when the dev person joined 60 seconds later. With all reserved capability spoken for, the dev person was confined to the 20% unreserved pool (~200 RPM). The manufacturing brokers continued drawing from their assured buckets, unaffected by the brand new arrival.

That is the situation that issues most in manufacturing. A batch job kicks off, or a brand new software goes dwell, and instantly there’s extra demand than provide. With out reservations, the brand new load pushes everybody’s throughput down equally. With reservations, your vital shoppers are shielded.

Reserved shoppers compete pretty amongst themselves

Within the reserved-only situation, the dev person went silent and solely the three manufacturing brokers competed. Their success charges had been almost similar (28.9%–30.2%) — the system divided throughput proportionally throughout their reservations.

What the server sees: OTEL metrics inform the story

Consumer-side metrics (success charges, 429 counts) let you know what your shoppers skilled. Server-side OTEL metrics let you know what the platform skilled. Right here’s what our instance deployment seemed like from the within.

The speed limiter protects mannequin well being

Throughout peak load (20,596 requests/minute hitting the endpoint), the NIM was serving solely the ~1,000 RPM that the speed limiter let by way of:

With out price limiting, these 20,000 RPM would have queued contained in the NIM. The GPU wouldn’t have gotten extra productive — it’s already at 91–95% — however latency would have spiraled as requests stacked up. As an alternative, the speed limiter rejected extra requests instantly (at 429-response speeds, not inference speeds), holding the mannequin responsive for the site visitors it did settle for.

Token throughput follows profitable requests

Peak token throughput was ~199,350 tokens/min (complete), with ~115,939 enter and ~83,411 output. These numbers monitor instantly with the speed limiter’s allowed throughput — not with the tried request quantity. One other manner of seeing that the speed limiter is appropriately shaping site visitors.

Deciding between Price Limits and Quota Reservations

Use this flowchart to resolve what to configure:

Step 1: Do you will have a shared deployment with a number of shoppers?

• No → Price limiting alone is adequate. Set capability to guard the GPU and transfer on.
• Sure → Proceed to Step 2.

Step 2: Are all shoppers equally vital?

• Sure → Price limiting alone could also be sufficient. Below saturation, all shoppers compete equally — first come, first served. If that’s acceptable, cease right here.
• No → Proceed to Step 3.

Step 3: Do any shoppers want assured minimal throughput?

• Sure → Add quota reservations. Dimension them to the minimal RPM every vital shopper wants throughout peak competition.
• No, however some shoppers must be deprioritized → Use per-entity exceptions as an alternative of reservations. Cap the noisy neighbors somewhat than guaranteeing the vital ones.

Step 4: Configure the unreserved pool.

• Don’t reserve 100% of capability. Depart 10–20% unreserved for ad-hoc site visitors, overflow, and new functions that haven’t been assigned reservations but.

Sensible configuration ideas

Begin with price limiting solely. Monitor your deployment’s site visitors patterns for every week. Have a look at peak RPM, who’s sending what, and whether or not anybody is constantly overconsuming. Then add reservations the place the information tells you they’re wanted.

Set utilization threshold at 70–80%. This offers the token bucket burst room to soak up brief spikes with out triggering price limiting on each minor fluctuation. In our instance, we used 80% and the system dealt with 1.2× capability gracefully earlier than enforcement kicked in.

Monitor with OTEL metrics. After configuring price limiting, examine these server-side metrics to verify issues are working:

• deployment.requests vs deployment.requests.rate_limited — are you rejecting the correct quantity?
• nvidia_gpu_utilization — is the mannequin nonetheless saturated or did price limiting create headroom?
• nvidia_vllm:e2e_request_latency_seconds — is latency steady below load?
• deployment.concurrent_requests — are requests queuing up or flowing easily?

Reservation sizing system:

Reserved RPM = Capability × Reserved %

Instance: 1000 RPM × 30% = 300 RPM assured

Don’t confuse this with a price restrict. A 30% reservation means “you’ll at all times get a minimum of 300 RPM, even when the system is saturated.” The entity can nonetheless use extra when capability is accessible.

Abstract

When contemplating Price Limiting vs. Quota Reservations: use every device the place it suits. Layer them the place the issue calls for it.