Running workloads on shared CPU instances can unlock significant cloud cost savings, sometimes 30-50% compared to dedicated vCPU machines. But deploying critical always-on services on shared CPU brings an inherent risk: noisy neighbors and resource contention can cause unpredictable performance hiccups. Effectively benchmarking shared reduce compute spend CPU requires more than plugging average CPU utilization numbers into a calculator—it demands disciplined observation with the right metrics and horizons.
In this post, I’ll walk you through:
- Why "always-on" small services commonly hide cloud waste when scaled on shared CPU. How AWS, Azure, and other cloud providers define shared CPU differently. Benchmarking best practices: capturing P95/P99 CPU spikes with adequate observation windows. How to use tools like AWS Compute Optimizer and Azure Advisor wisely for risk-aware cost optimization. Why averages lie and how to think in percentiles and spike durations for effective risk management.
Always-On Small Services: The Invisible Cloud Waste
Cloud teams tend to run many small, “always-on” services: monitoring agents, lightweight APIs, auth proxies, small asynchronous workers, etc. Individually, these consume tiny chunks of compute, so loading them onto cheap shared CPU instances looks like a no-brainer for cost reduction. But the aggregate impact is often surprisingly large.
The catch: these small services often run 24/7 without dyno-style scaling and can hide resource inefficiencies. Because CPU usage is intermittent, averages appear low. This invites temptation to pick smaller shared CPU flavors or burstable instances based on average CPU utilization, which dangerously underestimates peak resource demands.
Over time, these inefficiencies accumulate. The danger: unnoticed performance degradation during peak load or traffic spikes, leading to increased error rates, higher latency, or worse, cascading failures into dependent systems.
Shared CPU: What Does It Really Mean Across Providers?
Not all shared CPU flavors are created equal. Terminology like “burstable,” “shared-core,” or “low-priority” compute means different things depending on your cloud provider. Understanding these definitions is fundamental before benchmarking or optimizing.
Provider Shared CPU Flavor Example Key Characteristics AWS T3, T4g (Burstable Performance Instances)- Baseline CPU credits accrue when idle. Credits spend when bursting above baseline. Performance can degrade when credits are exhausted.
- Accrues and spends CPU credits. Provides predictable baseline CPU. Credit depletion throttles CPU performance.
- Shared CPU with equitable resource allocation. Less aggressive bursting, smoother degradation. No explicit CPU credits, relies on platform orchestration.
Note: Many people treat vCPU counts as hard performance guarantees. This is inaccurate on shared CPU instances where hypervisor scheduling, contention, and credit mechanisms influence actual CPU availability. Knowing these differences upfront helps set realistic expectations and prevents premature tuning mistakes.
Measuring Peaks with the Right Observation Window
Before touching instance types or changing your infrastructure, always ask: “What do the P95 and P99 CPU utilization percentiles look like?” And then also ask: “Over what time intervals are these spikes sustained?”
Why?
- Short-term spikes: A 10-second CPU spike during a batch job may be tolerable. Sustained load: A 5-minute sustained high CPU likely indicates a misfit for burstable machines. Observation Window: Too short a window masks variability; too long averages out spikes.
My rule of thumb is to monitor CPU usage in at least 1-minute granularity and collect data over periods of at least 2-4 weeks to cover workload variability, including weekends, patch deployments, or traffic seasonality.
Percentiles Over Averages: The Risk Perspective
Averaging is the simplest metric, but utterly misleading in shared CPU benchmarking. Averages conceal burstiness and peak demands associated with real-world user or batch loads.
Instead, focus on percentiles:
- P50 (Median): Typical CPU usage. Good for understanding baseline consumption. P95: High but infrequent CPU demand. Key for sizing buffers. P99: Rare but impactful peaks. Critical to prevent outages.
For risk management, target your shared CPU sizing on absorbing P95 or lower P99 usage without credit exhaustion or contention. This reduces risk of throttling and makes performance more predictable.
Also, track the duration of CPU spikes above certain thresholds since short bursts are less problematic than prolonged saturation.
Load Testing and Benchmarking: What Actually Worked
In my experience managing staging fleets and internal tooling across AWS and Azure, before migrating workloads to shared CPU instances, the following process worked best:
Establish Baseline Metrics: Collect 4+ weeks of production CPU utilization data with 1-minute granularity. Analyze Percentiles and Spike Durations: Identify P95 and P99 CPU usages and note continuous spike durations beyond 70-80% CPU. Run Load Tests Mirroring Production Peaks: Simulate realistic peak workloads on pilot instances using tools like Apache JMeter or Locust. Observe CPU Credit Balances (if applicable): For AWS T3/T4g or Azure B-series, monitor CPU credit exhaustion during load tests. Define Rollback Criteria: Set thresholds around latency percentiles, error rates, and credit exhaustion for automatic rollback if breached. Run Pilot Deployments on Shared CPU: Validate assumptions, monitor telemetry closely, and compare against baseline. Refine Sizing Based on Pilot: Adjust instance sizes, scale-out approaches, or fallback to dedicated vCPU types for high-risk components.
This strategy reduces operational risk and avoids punching holes through theoretical cost savings.
Using AWS Compute Optimizer and Azure Advisor Wisely
These tools provide cost and performance optimization recommendations leveraging historical telemetry and machine learning. However, they cannot see invisible risk bands—you must feed them the right inputs and interpret outputs cautiously.
- AWS Compute Optimizer: Shows CPU utilization histograms, identifies idle or over-provisioned instances, and suggests EC2 instance types including shared CPU options. Azure Advisor: Offers VM rightsizing suggestions, often recommending B-series for low baseline workloads.
To prevent getting misleading "low CPU average" driven recommendations:
- Export detailed CPU utilization percentiles and durations data to identify risk. Review recommended instance families alongside burst credit documentation. Validate that recommended types maintain P95/P99 CPU headroom before downsizing. Cross-check egress, storage, and IO metrics ignored by these tools, as they impact real-world performance and cost.
Remember, blindly following advisor tools for shared CPU optimizations without considering spike profiles leads to fragility under production load.
Summary: Principles to Benchmark Shared CPU for Risk-Managed Savings
- Always consider the risk of under-provisioning on shared CPU, which can manifest as latency spikes or outages. “Always-on” small services scaled out on shared CPU may deceptively appear cheap but hide cloud waste. Know your cloud provider’s shared CPU definition; CPU credits and scheduling matter. Benchmark based on percentiles (P95/P99) and spike durations, never on simple averages. Use at least 4 weeks of fine-grained monitoring data before making sizing decisions. Run realistic load tests to validate assumptions and set rollback criteria prior to live deployment. Leverage AWS Compute Optimizer and Azure Advisor as starting points, but interpret with care.
Trading lower cloud bills by moving workloads to shared CPU exists on the knife-edge between high savings and high risk. Approach benchmarking systematically with rich telemetry, risk-aware metrics, and staged pilots to get the best of both worlds.
Want to know something interesting? if you found this article useful or want to share your own benchmarking experiences, drop a comment below!