Resources
Edge ai Memory Bandwidth
Why ai compute does not predict deployed inference performance on its own.
By Doruk Sönmez, M.Sc.
AI Solutions Architect, CTai LABS/Connect Tech Inc.
Technical review: Rob Callaghan, P.Eng., Chief Product Officer, Connect Tech Inc.
Key Takeaways
- Peak ai compute and peak memory bandwidth describe different limits. A workload only benefits from available compute when data can be supplied to the processors quickly enough
- The roofline model expresses the relationship directly: attainable performance is bounded by the lower of the compute ceiling and memory bandwidth multiplied by arithmetic intensity (Williams et al., 2009)
- Memory capacity and memory bandwidth are separate constraints. Capacity determines whether the working set fits; bandwidth influences how quickly data can move while the workload runs
- On an Edge system, inference shares memory resources with decode, preprocessing, sensor pipelines, application logic, additional models, and concurrent execution. A model-only benchmark can miss the production bottleneck
- NVIDIA®’s Jetson Orin™ Nano Super update is a useful example: the same hardware platform received higher GPU, CPU, and memory clocks, increasing memory bandwidth from 68 GB/s to 102 GB/s alongside higher ai compute (NVIDIA, 2024)
- Optimization starts with profiling. NVIDIA TensorRT™ profiling guidance emphasize reproducible benchmarking, per-layer analysis, memory monitoring, and measurement before tuning (NVIDIA, 2026a)
Edge ai performance is often discussed in TOPS, TFLOPS, model size, and latency. Those metrics matter, but none describes the rate at which the processor can fetch model weights, activations, intermediate tensors, and application data from memory. When that data movement becomes the limiting resource, additional compute can remain underutilized even though the platform’s headline ai specification is higher.
The economics of memory makes that sizing decision more important than it was even a year ago. Memory prices have risen sharply as ai infrastructure demand absorbs more global DRAM and NAND capacity. TrendForce forecasts conventional DRAM contract prices to rise another 13% to 18% quarter-over-quarter in the third quarter of 2026, while NAND Flash prices were expected to increase 10% to 15% (TrendForce, 2026). J.P. Morgan Global Research estimates DRAM prices could rise more than 400% from the beginning of 2024 through the end of 2026, with higher memory costs increasingly flowing into electronics pricing (J.P. Morgan, 2026). For Edge ai systems, carrying substantially more memory capacity than the workload requires can therefore affect both module selection and production economics, making workload profiling and right-sizing part of the hardware decision rather than an afterthought.
Memory bandwidth therefore belongs in the platform-selection and optimization discussion alongside compute, memory capacity, power, thermals, and I/O. The important question is not whether memory bandwidth is universally the bottleneck. It is whether the deployed workload is compute-bound, memory-bandwidth-bound, memory-capacity-bound, or constrained elsewhere in the pipeline.
The memory wall is also a broader ai systems problem. Gholami et al. (2024) found that peak compute capability has scaled substantially faster than DRAM and interconnect bandwidth, helping explain why memory movement can become a dominant limiter even as accelerator arithmetic throughput increases. For Edge deployments, the practical implication is to match compute and memory behavior to the workload rather than treating either specification in isolation.
Compute, capacity, and bandwidth answer different questions
| Resource | What it describes | Typical symptom when constrained | Engineering question |
|---|---|---|---|
| ai compute | The platform's peak mathematical throughput for supported operations and precisions. | Compute units remain highly utilized and more arithmetic throughput can improve execution. | Is the workload doing enough arithmetic to use the available processors? |
| Memory capacity | How much working data can reside in system memory. | Out-of-memory failures, aggressive eviction or offload, or too little headroom for the complete application. | Do model weights, activations, caches, sensor buffers, and application processes fit at the required concurrency? |
| Memory bandwidth | The peak rate at which data can be transferred through the memory subsystem. | Compute resources wait for data movement, and added compute produces diminishing gains. | Can the memory subsystem supply the workload at the rate required by the target latency or throughput? |
| I/O and pipeline | The movement and processing of data before and after inference. | Decode, copies, preprocessing, sensor ingest, storage, or application work dominates end-to-end latency. | Is the model actually the slowest part of the deployed system? |
The distinction matters because the remedies are different. A capacity problem may require reducing the working set or selecting a higher-memory module. A bandwidth problem may respond to reduced precision, better data reuse, fewer memory transfers, operator fusion, or a platform with a more appropriate memory subsystem. A compute-bound workload may benefit directly from more accelerator throughput. An I/O-bound system may require changes outside the model entirely. CTai LABS’ ai architects and ai engineers have expertise in all of this troubleshooting.
More bytes are moved for each unit of computation. Performance is more likely to scale with memory bandwidth.
Optimization can shift the balance by changing precision, data reuse, fusion, batching, or execution strategy.
More computation is performed per byte moved. Performance is more likely to approach the compute ceiling.
Figure 1. The roofline model separates memory-bound and compute-bound behavior using arithmetic intensity. At low arithmetic intensity, attainable performance rises with memory bandwidth. At sufficiently high arithmetic intensity, the compute ceiling becomes the dominant limit (Williams et al., 2009).
The production workload is larger than the inference engine
An Edge ai system rarely runs a model in isolation. Camera and sensor data must enter the platform, be decoded or transformed, pass through preprocessing, reach one or more inference engines, and then move into postprocessing and application logic. Those stages can share DRAM bandwidth, cache, copy engines, CPU resources, and GPU resources with the model.
Independent benchmarking practice reinforces the need to measure the whole system. MLPerf Inference was developed to provide representative and reproducible comparisons across very different hardware and software combinations, rather than relying on peak specifications alone (Reddi et al., 2020).
NVIDIA® TensorRT™ optimization guidance explicitly notes that concurrent CUDA streams share streaming multiprocessors, register files, L2 cache, and DRAM bandwidth, creating contention that can change which kernels are optimal at runtime (NVIDIA, 2026b). TensorRT also documents device-memory use for weights, activation tensors, scratch memory, and execution contexts, reinforcing that runtime memory behavior extends beyond the serialized model file (NVIDIA, 2026c).
The roofline model uses that relationship to establish two performance regions. In the memory-bound region, performance is limited by bandwidth multiplied by arithmetic intensity. In the compute-bound region, performance is limited by the processor’s compute ceiling. The transition between the two is workload- and platform-specific (Williams et al., 2009).
A larger TOPS number therefore does not guarantee a proportional improvement for every Edge ai model. If execution already spends substantial time waiting on memory, adding arithmetic throughput without reducing data movement can leave part of the accelerator idle.
Arithmetic intensity explains why TOPS is not a universal performance predictor
Arithmetic intensity describes how much computation is performed for the data moved from memory. An operation with high arithmetic intensity reuses data effectively and performs substantial work before requiring more memory traffic. At low arithmetic intensity, comparatively more data is transferred for each unit of computation.
The roofline model uses that relationship to establish two performance regions. In the memory-bound region, performance is limited by bandwidth multiplied by arithmetic intensity. In the compute-bound region, performance is limited by the processor’s compute ceiling. The transition between the two is workload- and platform-specific (Williams et al., 2009).
A larger TOPS number therefore does not guarantee a proportional improvement for every Edge ai model. If execution already spends substantial time waiting on memory, adding arithmetic throughput without reducing data movement can leave part of the accelerator idle.
Figure 2. Where memory traffic occurs in an Edge ai pipeline
Figure 2. Memory bandwidth is consumed across the complete sensor-to-application path. Profiling only the inference engine can miss contention introduced by capture, decode, preprocessing, concurrent models, postprocessing, and application work.
Sensor-rich systems can create bandwidth pressure before the model changes
Multi-camera and multi-sensor systems increase the volume of data entering and moving through the platform even when the inference model is unchanged. Higher resolutions, frame rates, additional streams, multiple models, and concurrent preprocessing can all reshape the memory-traffic profile.
End-to-end profiling is especially important for Robotics & Logistics, Industrial Automation, and rugged Construction, Agriculture & Mining deployments where several sensors and software stages run concurrently. The production question is whether the complete system sustains the required frame rate, latency, and concurrency under representative sensor load, not whether one inference engine reaches an acceptable result with synthetic input.
Reduced precision can affect both compute and data movement
Precision changes are often discussed as a compute optimization, but lower bit-widths can also reduce the memory footprint of weights and activations. Research on mixed-precision Edge inference shows that quantization can lower memory and energy demands while preserving useful accuracy when the precision strategy is selected for the model and hardware (Schaefer et al., 2024).
The performance result still must be measured. Reduced precision does not guarantee that every layer becomes faster, and it introduces numerical considerations that may require mixed precision or wider accumulation for sensitive operations. The correct precision strategy is therefore a performance-and-quality decision, not a blanket conversion.
Generative ai changes the memory profile again
Transformer inference introduces a different memory profile from many fixed-shape vision pipelines. Model weights remain a major part of the working set, while runtime state grows with context length, sequence length, batching, and concurrency. Gholami et al. (2024) identify memory bandwidth as an increasingly important limiter in ai serving, particularly for decoder-style models.
Autoregressive serving also creates dynamic KV-cache requirements. Kwon et al. (2023) showed that inefficient KV-cache allocation can waste substantial memory and constrain serving throughput, especially as sequences and request concurrency grow. Capacity still determines whether weights, runtime allocations, and caches fit; bandwidth governs how quickly the system can service the resulting data movement during execution.
A real Jetson™ example: compute and bandwidth can change together
NVIDIA’s Jetson Orin™ Nano Super update provides a useful, controlled example because the performance increase came from a software-enabled power mode on the same developer-kit hardware. NVIDIA increased the GPU clock, CPU clock, and memory clock, raising sparse INT8 ai performance from 40 TOPS to 67 TOPS and memory bandwidth from 68 GB/s to 102 GB/s (NVIDIA, 2024).
NVIDIA reported up to a 1.7x generative-ai performance increase for the updated configuration (NVIDIA, 2024). The example is important because it avoids the simplistic conclusion that the gain came from compute alone: the memory subsystem accelerated at the same time.
Jetson Orin Nano
Original Configuration
40 TOPS
SPARSE INT8 ai COMPUTE
68 GB/s
MEMORY BANDWIDTH
RESULT
Same developer-kit hardware
Jetson Orin Nano Super Mode
Software-enabled power mode
67 TOPS+68%
SPARSE INT8 ai COMPUTE
102 GB/s+50%
MEMORY BANDWIDTH
CONSTANT
Compute and memory bandwidth increased together
Figure 3. Peak memory bandwidth increases from 102 GB/s on NVIDIA Jetson Orin Nano Super to 204.8 GB/s on NVIDIA Jetson AGX Orin™ 64GB and 273 GB/s on NVIDIA Jetson Thor™ T5000. These values describe the platform memory subsystem, not expected application throughput. Jetson Thor uses FP4/FP8 compute metrics that are not directly comparable to the INT8 TOPS figures used for Jetson Orin, so compute values are intentionally not plotted on the same axis (NVIDIA, 2026e; NVIDIA, 2026f; NVIDIA, 2026g).
Why the Jetson comparison needs more than one axis
These Jetson platforms provide progressively higher peak memory bandwidth, but their ai compute specifications do not use one directly comparable precision metric. Jetson Orin Nano Super and Jetson AGX Orin are commonly specified in INT8 TOPS, while Jetson Thor T5000 is specified in FP4 and FP8 TFLOPS. Plotting those values as a single continuous compute series would imply a comparison the specifications do not support.
A better platform-selection method is to benchmark the intended model and pipeline on candidate hardware, then inspect utilization and memory behavior. Published peak specifications define ceilings; they do not predict arithmetic intensity, kernel mix, data reuse, sensor traffic, or contention in the deployed application.
How to tell whether an Edge ai workload is memory-bandwidth-bound
No single metric proves that a complete application is bandwidth-bound. Diagnosis starts with a representative, reproducible baseline across the model and surrounding pipeline. MLPerf Inference was created around architecture-neutral, reproducible system benchmarking for exactly this reason: inference results depend on the interaction of model, software, hardware, and deployment scenario (Reddi et al., 2020). For TensorRT-specific tuning, NVIDIA’s guidance adds per-layer profiling, memory monitoring, and control of clocks, power, thermals, and transfer behavior so comparisons remain stable (NVIDIA, 2026a).
- Profile end-to-end latency and throughput, then separate model execution from decode, preprocessing, copies, postprocessing, and application work.
- Inspect GPU utilization and Tensor Core activity. Low accelerator utilization can indicate that the workload is waiting elsewhere, although it does not identify memory bandwidth by itself.
- Inspect DRAM and cache behavior with the profiling tools supported for the target Jetson software and hardware configuration.
- Test changes that alter memory traffic, including precision, fusion, data layout, copies, batching, or concurrency, and measure whether latency or throughput responds.
- Repeat the benchmark under the real sensor count, resolution, frame rate, model concurrency, power mode, and thermal conditions expected in production.
- Compare candidate platforms using the same model, runtime, inputs, and success criteria rather than comparing peak specifications alone.
Optimization levers when memory traffic is the constraint
The correct intervention depends on where the traffic originates. TensorRT’s optimizer can eliminate dead computation, fold constants, reorder operations, combine layers, and select implementations with different data formats (NVIDIA, 2026h). Its performance guidance also covers fusion, batching, multi-streaming, tensor alignment, and memory-access patterns for specific layer types (NVIDIA, 2026b).
- Use an appropriate precision strategy to reduce tensor and weight traffic where the model’s accuracy requirements allow it.
- Reduce unnecessary host-to-device and device-to-device copies. Keep data resident where possible and remove intermediate transformations that exist only because of pipeline boundaries.
- Use operator and pointwise fusion where the runtime can eliminate intermediate reads and writes.
- Review tensor shapes and layouts so optimized kernels and Tensor Core paths are available where appropriate.
- Tune batching and concurrency against the production request pattern. More parallel work can improve utilization, but simultaneous streams also compete for DRAM bandwidth and other shared resources.
- Move suitable preprocessing or data-path work to dedicated hardware engines where the platform and software stack support it, then verify the effect on end-to-end latency and throughput.
- If software changes cannot create sufficient headroom, select a module whose memory subsystem, compute, capacity, power, and I/O match the complete application.
Memory bandwidth is part of platform right-sizing
A system can carry more compute than the application can use if another resource is limiting execution. It can also have enough memory capacity for the model while lacking the bandwidth needed to meet the production latency or throughput target. Platform right-sizing should therefore follow measured bottlenecks rather than a single headline specification.
For CTai LABS, this resource sits between ai Model Optimization and Memory Optimization. ai Model Optimization addresses model and inference performance across runtime, precision, quantization, graph and engine behavior. Memory Optimization addresses working-set footprint, headroom, memory pressure, and validated module right-sizing. Memory-bandwidth analysis identifies whether the rate of data movement is constraining the deployed workload and whether software optimization or a different platform is the appropriate response.
Within the CTI EdgeAI Stack, memory-bandwidth analysis connects the compute platform to carrier I/O, sensor traffic, runtime software, power, and thermal constraints. Treating those elements as one deployment system helps CTai LABS determine whether the next improvement should come from model/runtime optimization, Memory Optimization, data-path changes, or a different platform.
A practical measurement checklist
| Measure | Why it matters | Compare against |
|---|---|---|
| End-to-end latency and tail latency | Shows whether the deployed pipeline meets the application deadline. | Production requirement, not model-only latency. |
| Throughput / FPS / tokens per second | Shows sustained application output under representative load. | Required sensor rate, concurrency, or user load. |
| Model and application memory | Separates capacity pressure from bandwidth pressure. | Available memory plus required operating headroom. |
| GPU / accelerator utilization | Shows whether compute resources are being kept busy. | Expected utilization under a stable benchmark. |
| DRAM / cache activity | Helps identify memory-system pressure and data-movement behavior. | Platform peak and changes between controlled test cases. |
| Sensor and preprocessing load | Captures traffic outside the inference engine. | Production camera count, resolution, frame rate, and sensor mix. |
| Power and thermals | Prevents throttling or power-mode differences from contaminating results. | Target deployment mode and sustained operating condition. |
| Precision and runtime configuration | Changes both compute paths and memory behavior. | Validated accuracy and performance target. |
The distinction matters because the remedies are different. A capacity problem may require reducing the working set or selecting a higher-memory module. A bandwidth problem may respond to reduced precision, better data reuse, fewer memory transfers, operator fusion, or a platform with a more appropriate memory subsystem. A compute-bound workload may benefit directly from more accelerator throughput. An I/O-bound system may require changes outside the model entirely. CTai LABS’ ai architects and ai engineers have expertise in all of this troubleshooting.
ABOUT THE AUTHOR
Doruk Sönmez, M.Sc.
AI Solutions Architect, CTai LABS
Doruk is an AI Solutions Architect at CTai LABS, the Physical AI and Edge AI services division of Connect Tech Inc., an NVIDIA Elite Partner. An NVIDIA DLI Certified Instructor, he specializes in deploying vision-language models, agentic AI workflows, and accelerated video pipelines on NVIDIA Jetson platforms.
Book a Demo
Have an Edge ai workload that needs to move from model to deployment? Talk to CTai LABS about your application, performance requirements, data constraints, and deployment environment.
Learn More
Resources and Frequently Asked Questions
Related
Sources
Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., & Keutzer, K. (2024). AI and memory wall. IEEE Micro, 44(3), 33–39.
https://doi.org/10.1109/MM.2024.3373763J.P. Morgan. (2026, August 6). Overdrawn at the memory bank: How AI is creating memory shortage, and what happens next.
https://www.jpmorgan.com/insights/global-research/artificial-intelligence/dram-memory-shortage-from-aiKwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. Proceedings of the 29th Symposium on Operating Systems Principles, 611–626.
https://doi.org/10.1145/3600006.3613165NVIDIA. (2024, December 17). NVIDIA Jetson Orin Nano Developer Kit gets a “Super” boost. NVIDIA Technical Blog.
https://developer.nvidia.com/blog/nvidia-jetson-orin-nano-developer-kit-gets-a-super-boost/NVIDIA. (2026a). Best practices. NVIDIA TensorRT documentation.
https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/best-practices.htmlNVIDIA. (2026b). Optimizing TensorRT performance. NVIDIA TensorRT documentation.
https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/optimization.htmlNVIDIA. (2026c). How TensorRT works. NVIDIA TensorRT documentation.
https://docs.nvidia.com/deeplearning/tensorrt/latest/architecture/how-trt-works.htmlNVIDIA. (2026d). Accuracy considerations. NVIDIA TensorRT documentation.
https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/accuracy-considerations.htmlNVIDIA. (2026e). Jetson Orin Nano Super Developer Kit.
https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/nano-super-developer-kit/NVIDIA. (2026f). NVIDIA Jetson AGX Orin Developer Kit quick specs. NVIDIA Jetson developer kits.
https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson/back-to-school/NVIDIA. (2026g). NVIDIA Jetson T5000 Modules data sheet (DS-11945-001, v1.2).
https://developer.download.nvidia.com/assets/embedded/secure/jetson/thor/docs/Jetson_T5000_Modules_Datasheet_DS-11945-001_v1.2.pdfNVIDIA. (2026h). TensorRT’s capabilities. NVIDIA TensorRT documentation.
https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/capabilities.htmlReddi, V. J., Cheng, C., Kanter, D., Mattson, P., Schmuelling, G., Wu, C.-J., Anderson, B., Breughe, M., Charlebois, M., Chou, W., Chukka, R., Coleman, C., Davis, S., Deng, P., Diamos, G., Duke, J., Fick, D., Gardner, J. S., Hubara, I., et al. (2020). MLPerf inference benchmark. 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 446–459.
https://doi.org/10.1109/ISCA45697.2020.00045Schaefer, C. J. S., Joshi, S., Li, S., & Blazquez, R. (2024). Edge inference with fully differentiable quantized mixed precision neural networks. 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 8460–8469.
https://doi.org/10.1109/WACV57701.2024.00827TrendForce. (2026, July 3). AI server demand continues to support memory prices in 3Q26, but gains moderate as consumer demand weakens and high base effects take hold.
https://www.trendforce.com/presscenter/news/20260703-13134.htmlWilliams, S., Waterman, A., & Patterson, D. (2009). Roofline: An insightful visual performance model for multicore architectures. Communications of the ACM, 52(4), 65–76.
https://doi.org/10.1145/1498765.1498785Frequently Asked Questions
What is memory bandwidth in Edge ai?
Memory bandwidth is the peak rate at which data can move through the platform’s memory subsystem. For ai inference, that data can include model weights, activations, intermediate tensors, sensor buffers, and application data. It is distinct from memory capacity, which describes how much data can reside in memory.
Why can a model be slow even when the NVIDIA Jetson module has high TOPS?
TOPS describes a peak compute capability at a specified precision. A workload may be limited by memory bandwidth, memory capacity, sensor or preprocessing work, unsupported or inefficient kernels, power and thermals, or application overhead. The bottleneck has to be measured on the target system.
What is the difference between memory capacity and memory bandwidth?
Capacity answers whether the working set fits. Bandwidth describes how quickly data can be transferred while the workload executes. A workload can fit in memory and still be bandwidth-bound, or it can have adequate bandwidth but fail because the complete working set exceeds available capacity.
What does arithmetic intensity mean for ai inference?
Arithmetic intensity is the amount of computation performed relative to the amount of data moved from memory. In the roofline model, low-arithmetic-intensity workloads are more likely to be constrained by memory bandwidth, while sufficiently high-arithmetic-intensity workloads can approach the processor’s compute ceiling (Williams et al., 2009).
Does INT8 or FP16 always solve a memory-bandwidth problem?
No. Reduced precision can lower memory traffic and improve compute efficiency, but the result depends on the model, supported kernels, tensor shapes, runtime, and accuracy requirements. Mixed-precision research shows the memory benefits can be substantial, while NVIDIA’s TensorRT guidance still recommends validating numerical behavior on the actual model and target platform (Schaefer et al., 2024; NVIDIA, 2026d).
Can multiple cameras affect inference performance even if the model does not change?
Yes. Camera capture, decode, preprocessing, buffering, copies, and additional application work can share memory and compute resources with inference. The production benchmark should therefore use the intended camera count, resolution, frame rate, and pipeline configuration.
Does Jetson Thor have more memory bandwidth than Jetson AGX Orin?
Yes. NVIDIA specifies 273 GB/s peak memory bandwidth for Jetson Thor T5000 and 204.8 GB/s for Jetson AGX Orin 64GB. Those figures are platform ceilings and do not imply a fixed application-performance ratio (NVIDIA, 2026f; NVIDIA, 2026g).
How does CTai LABS determine whether memory bandwidth is the bottleneck?
CTai LABS profiles the model within the complete Edge ai application under representative sensor and software load. The analysis compares latency, throughput, memory use, accelerator utilization, data movement, power, and thermal behavior so the limiting resource is identified before optimization or hardware changes are recommended.
Ready to Build Smarter?
Let’s create the intelligent Edge AI solution that moves your business forward.