What is a cloud GPU?
A cloud GPU is a graphics processing unit hosted in a remote data center and rented to users over the internet, typically on a per-hour or per-second basis. Instead of buying a physical card, installing it in a workstation, and managing drivers, power, and cooling locally, a tenant provisions a virtual machine or container with one or more GPUs attached, then accesses it remotely through SSH, a remote desktop protocol, a managed notebook, or an API endpoint. Cloud GPUs tend to span a range of classes — from inference-tuned cards like the NVIDIA T4 and L4 to training-class accelerators like the A100, H100, and H200, alongside workstation-class RTX cards used for graphics-heavy workloads. They are commonly used for AI model training and inference, real-time pixel streaming, offline rendering, 3D content preprocessing, and simulation pipelines that benefit from elastic capacity rather than fixed on-premises hardware.
How cloud GPUs work
Cloud GPUs are exposed through a layered stack that combines virtualization, scheduling, and remote networking. The path from a user request to a running GPU workload tends to look something like this:
- Provisioning. A user selects an instance type that pairs a GPU SKU (for example H100 80GB, A100 40GB, L4, or RTX 6000 Ada) with a CPU, memory, and storage profile. The provider's control plane allocates a host with the requested accelerator and boots a VM or container image that contains matching drivers and CUDA libraries.
- Attachment. The GPU is exposed to the guest either through full PCIe passthrough, vGPU partitioning, or Multi-Instance GPU (MIG) slices on supported cards. Passthrough gives the tenant the whole device, while MIG and vGPU let one physical GPU be split across several smaller tenants.
- Networking. For multi-GPU and multi-node jobs, hosts are linked by high-bandwidth fabrics — NVLink and NVSwitch inside a node, and InfiniBand or RoCE between nodes — so that collective operations like all-reduce can scale without saturating standard Ethernet.
- Workload execution. The tenant runs containers, training jobs, inference servers, or interactive sessions. Schedulers such as Kubernetes with the NVIDIA device plugin, Slurm, or provider-native services place work onto GPUs and track utilization.
- Metering and teardown. Usage is metered by GPU-hour, with separate line items for storage, egress, and sometimes interconnect. When the workload finishes or the user stops the instance, the GPU is returned to the pool and may be reallocated to another tenant.
Major providers and offerings
Several categories of providers tend to dominate the cloud GPU market, each with different trade-offs around price, availability, and tooling:
- Hyperscalers. AWS (EC2 P5/P4/G6/G5), Microsoft Azure (ND/NC/NV series), and Google Cloud (A3/A2/G2) offer broad GPU catalogs alongside managed services for training, inference, and orchestration. They tend to suit teams that already run other workloads on the same platform.
- NVIDIA DGX Cloud. A managed offering that exposes DGX-class clusters — H100 or H200 nodes wired together with NVLink and InfiniBand — through partner clouds, aimed at large-scale training and foundation model work.
- Specialist GPU clouds. CoreWeave, Lambda, Crusoe, Nebius, and similar providers focus on GPU-dense infrastructure with bare-metal or thin-virtualization options. They often list newer SKUs sooner and can price aggressively for committed capacity.
- Marketplaces and on-demand pools. Platforms such as RunPod, Vast.ai, and Paperspace aggregate capacity from operators of varying size, which can suit short bursts, experimentation, or interactive notebooks.
Cost model
Cloud GPU pricing is usually quoted as a per-hour rate per GPU class, with multipliers for the rest of the instance. A few patterns tend to recur across providers:
- Inference-tier cards. T4 and L4 instances often land in the lower per-hour range and are common for serving smaller models, video transcoding, and lightweight rendering.
- Training-tier cards. A100 (40GB or 80GB) and H100 (80GB) instances are priced significantly higher and are used for large model training, fine-tuning, and demanding inference. H200 and Blackwell-class parts sit at the top of the range where available.
- Workstation cards. RTX 6000 Ada, RTX A6000, and similar SKUs are typical for graphics, 3D content work, and pixel streaming where display engines and encoder counts matter as much as raw compute.
- Commitment discounts. Reserved, committed-use, and spot or preemptible pricing can reduce the on-demand rate substantially in exchange for term length or interruptibility.
- Hidden line items. Storage IOPS, egress bandwidth, dedicated interconnect, and managed service surcharges can add a meaningful fraction on top of the headline GPU rate, so budgeting tends to require modeling the full workload, not just the accelerator.
Where cloud GPUs are used
Cloud GPUs tend to show up wherever workloads are bursty, parallel, or graphics-heavy enough that buying dedicated hardware would be inefficient. Common workloads include:
- AI training. Large language models, diffusion models, and computer vision networks are commonly trained on clusters of A100, H100, or newer accelerators connected by NVLink and InfiniBand.
- AI inference. Serving generative and predictive models at scale, often on L4, L40S, or H100 instances behind autoscaling endpoints.
- Pixel streaming. Server-side rendering of interactive 3D applications — built in engines like Unreal or Unity — with the encoded video streamed to thin clients in a browser or on a device.
- Render farms. Offline rendering for film, animation, architecture, and product visualization using path tracers and rasterizers that scale across many GPUs.
- 3D content preprocessing. Photogrammetry, neural radiance field training, Gaussian splatting reconstruction, mesh decimation, and asset conversion pipelines that turn captured data into streamable 3D representations.
- Sim-to-real and robotics. Physics simulation, synthetic data generation, and reinforcement learning environments that train policies in parallel before deployment on physical hardware.
Cloud GPUs and 3D streaming
3D streaming workloads tend to use cloud GPUs in two distinct ways. In pixel streaming, the GPU runs the full 3D engine server-side, renders each frame, encodes it as video, and ships pixels to the client; the client decodes video and sends back input. This model leans on cards with strong rasterization and dedicated encoders, and it scales by adding more GPU sessions as concurrent users grow. In spatial or 3D streaming, the GPU is used earlier in the pipeline — reconstructing scenes from captures, training neural representations, baking lightfields, or packaging Gaussian splats and meshes into streamable tiles — while the client device does the final rendering. The same physical hardware can serve both patterns, but the cost profile differs: pixel streaming tends to incur ongoing per-session GPU-hours during playback, whereas preprocessing concentrates GPU spend in batch jobs before content is delivered.
See also
Pixel streaming — server-side rendering and video encoding that depends on cloud or on-prem GPU capacity.
3D streaming — delivering 3D scenes to clients, often using cloud GPUs for preprocessing and packaging.
Spatial streaming — client-side rendering of streamed 3D representations prepared on cloud GPU pipelines.
Server-side rendering — the rendering model behind pixel streaming, sized by GPU-hours per concurrent session.
Additional resources
