Serverless vs dedicated GPUs for inference

Serverless GPUs bill per second only while a worker runs and scale to zero, so they are cheaper when traffic is low or bursty. A dedicated GPU bills for every hour it is on, so it wins once it is busy enough. For the same GPU at the same provider, the break-even share of the month equals the dedicated hourly price divided by the serverless hourly price; the table below computes it from current verified prices. Latency matters as much as cost: cold starts add container boot and model loading time unless you keep workers warm, and warm workers are billed time.

• Serverless: per-second billing, scale to zero, cold starts, no idle cost unless you keep workers warm.
• Dedicated: flat cost while running, no cold starts, full control over batching and the serving stack.
• Break-even share of the month = dedicated price ÷ serverless price, for the same GPU, GPU count and provider.
• Many production setups combine a small dedicated baseline with serverless capacity for peaks.

Serverless GPUs bill for the time a worker exists. Modal, for example, bills “by the second for CPU, memory, and GPU time”, and once an application “has scaled down to zero containers you no longer get charged” [1]. The billed time has three parts: start time (starting the container and loading the model into GPU memory), execution, and an idle window after the last request [1]. Modal keeps an idle container for up to 60 seconds by default before shutting it down, adjustable from 2 seconds to 20 minutes, and you pay for resources while it idles [2].

Dedicated GPUs bill for the time the instance runs, busy or not. Some providers bill them by the second and others by the minute or the hour (each offer’s billing increment is listed on its provider page), but every increment is billed until you stop the instance.

The providers in our dataset that publish scale-to-zero GPU prices, with their lowest price and how finely they bill:

Serverless GPUs usually cost more per hour than a dedicated instance of the same GPU. The question is how many hours you actually use. If a dedicated GPU costs D per hour and the serverless worker S per busy hour, a month of dedicated capacity costs D × 730 and serverless costs S × busy hours. The two are equal when:

Below that share of the month, serverless is cheaper; above it, a dedicated instance is. The table computes it from current verified prices, only where one provider publishes both prices for the same GPU and the same number of GPUs, so providers, GPU models and multi-GPU cluster rates are never mixed:

A worked example with the median pair in the table: the L40S at OVHcloud costs $1.77 per hour serverless and $1.60 per hour on a dedicated instance. A worker that is busy 4 hours a day, about 122 hours a month, costs about $216/mo serverless, against $1,165/mo for the dedicated instance running all month. At 16 busy hours a day (487 hours) serverless comes to about $861/mo, still less than the dedicated instance.

The comparison covers compute only. Add storage, network volumes and egress, which apply to both models, with the GPU cost calculator.

A cold start is the wait for a new worker: booting the container and running your initialisation, which for an LLM means loading gigabytes of weights into GPU memory. Modal reports that its containers boot in about one second, but a container only counts as warm after your setup code has run, and downloading weights ahead of time instead of at boot “can reduce boot times from minutes to seconds” for models in the tens of gigabytes [2]. That start phase is billed time too [1].

Ways to reduce cold starts, and what they cost

• Cache or bake the weights into the image or a volume so each worker skips the download.
• Keep workers warm. Modal’s min_containers sets a floor so a function never scales to zero, and buffer_containers keeps spare containers while it is active [2].
• Stay warm for longer with a longer idle window, such as Modal’s scaledown_window [2].
• Use a smaller or quantised model, which loads faster; see the VRAM guide.

Every warm worker is billed time, so these settings move you along the same curve as the break-even table. Modal states the trade-off directly: they “introduce a trade-off between cold start latencies and cost” [2].

A busy dedicated GPU does more work per dollar when the server batches requests. The PagedAttention paper, which introduced vLLM, reports 2–4× higher throughput than earlier serving systems at the same latency, with larger gains for longer sequences and larger models [3]. A dedicated instance under steady load gets that benefit continuously.

Serverless platforms can batch too, but only when requests reach the same worker. Check how many concurrent requests one worker accepts: one request per worker multiplies cold starts and billed idle windows.

If you plan to run vLLM on your own instance, the vLLM hosting page lists providers with an official vLLM template or documentation.

Estimate your busy hours, compare them with the break-even share above, and price both options in the GPU cost calculator. Providers offering both models are compared under inference hosting and managed AI hosting.