Technology · 07 AI Compute & Chipsets
Most AI projects never need a training cluster. We match each workload to the right silicon, rent before we buy, and confirm availability, region and cost at scoping, because chip names and supply change quickly.
Updated September 2026 You own everything Reply within one business day
Vision - line 3 camera
Warehouse detector v4 - 640x640 - centre aisle
38ms
Inference
24/s
Frames read
1,284
Objects tracked
OlDevs, a full-stack technology studio in Vancouver, sizes AI compute from the model outward.We measure the memory a model needs at its target precision, then choose between rented cloud accelerators, a workstation in the client's office, or an edge device. Inference is the usual workload and gets the smaller hardware; training and fine-tuning are scoped separately and rented by the hour until steady demand justifies owning anything.
Key facts
07 · AI Compute & Chipsets
Every option below is a trade between memory, throughput, availability and who has to keep it running after launch.
H100 and H200 are the safe pick for fine-tuning and serving large models: CUDA tooling is mature and both are bookable in Canadian and US regions. Blackwell (B200, GB200 NVL72, B300 Blackwell Ultra) suits the largest jobs but is still allocated, so we reserve at scoping. Later cost: rental that runs until switched off.
MI300X, MI325X and the MI350 series carry more HBM per card than the matching NVIDIA generation, so a large model fits on fewer GPUs. Gaudi 3 fits a client already committed to Intel or wanting a second source. Later cost: fewer engineers who know ROCm or Gaudi, and slower support for new models.
Trillium (v6e) and Ironwood (v7) on Google Cloud, and Trainium2, Trainium3 and Inferentia on AWS, fit a client who already lives on that cloud and runs JAX or PyTorch XLA. Queues are shorter than for NVIDIA parts. Later cost: portability, since leaving that silicon means re-testing the whole serving stack.
Groq and Cerebras serve open-weight models over an API with very low latency per token, ideal for assistants and agents where response time is the product, and for clients who do not want to run a GPU at all. Later cost: a limited model catalogue and one provider's roadmap, so we keep a standard API in front of it.
A desktop with an RTX PRO 6000 Blackwell or GeForce RTX 5090, or a Mac with an M4 Max or M5 and generous unified memory, runs mid-size open models for prototyping and private data work with no cloud bill. Later cost: one box cannot serve a production audience, so the cloud path is designed before the first demo.
Jetson Orin and Thor modules run vision and small language models on a machine, vehicle or kiosk; Raspberry Pi 5 with the AI HAT+ (Hailo) and Google Coral cover low-power detection. Right when the connection is unreliable or data must never leave the site. Later cost: updating models on devices you cannot walk to.
Stack
Concrete tools, current at the time of writing; confirmed for your project at scoping.
Cloud accelerators
Serving and inference
Training and fine-tuning
Quantisation and model formats
Workstations and edge devices
Where it runs
What changed
What moved in this area and what it means for your build.
Early 2025
GeForce RTX 50-series cards and the RTX PRO Blackwell workstation line shipped, with the RTX PRO 6000 carrying far more memory than earlier single workstation cards. A mid-size open model can now be prototyped and evaluated on one machine in the office before any cloud spend.
2025
B200 instances and GB200 NVL72 racks came online at the hyperscalers and specialist GPU clouds through the year, and H100 and H200 queues shortened as a result. Fine-tuning on Hopper is now a routine booking; we hold Blackwell for jobs that need the interconnect.
Mid 2025
NVIDIA announced B300 and GB300 Blackwell Ultra, and AMD shipped the MI350 series, both with a larger memory stack per accelerator. Larger models fit on fewer cards, but since a new SKU appears every year we specify by memory and interconnect rather than by part name.
2025
Google introduced Ironwood (TPU v7) alongside Trillium, and AWS moved Trainium2 to general availability with Trainium3 following. Clients already on Google Cloud or AWS gain an inference option that avoids the NVIDIA queue, with a serving stack tied to that cloud.
Late 2025
The M5 generation improved the neural accelerators inside Apple's GPU cores, and the Foundation Models framework introduced with iOS 26 gave apps a supported way to run small models on the device. An iOS or macOS product can now ship private, offline AI features without a server round trip.
2026
The trend at the time of writing is falling cost per token through quantisation, speculative decoding and specialist providers, alongside a growing sovereign compute effort in Canada. In-country residency is more achievable than it was, though GPU choice in Canadian regions is still narrower than in the US; we confirm both at scoping.
How we choose
A rented H100 with a managed serving layer suits a client with no infrastructure team. A workstation or Jetson fleet is only right if someone on the client's side, or on ours under a support agreement, will patch drivers and roll out models.
Health, public-sector and financial data often has to stay in Canada. That narrows the accelerator list before performance is discussed, and sometimes a slightly older GPU in Montreal beats a newer one in Oregon.
We compute the VRAM or HBM a model needs at its precision, with headroom for context and batch size. If it does not fit, no amount of compute helps. Only then do we look at tokens per second and interconnect.
Hourly cloud GPUs cost nothing when idle and let us change chip generation freely. We recommend buying hardware only when utilisation has been steady for about a year and the client has somewhere secure and cooled to put it.
Every accelerator outside the NVIDIA and CUDA mainstream saves money now and costs portability later. If a client may need to move clouds within three years, we stay on portable formats and standard serving APIs and say so in the proposal.
Chip names, quotas and regional stock change month to month. Before any proposal we check what is actually bookable in the target region, hold the capacity, and write the plan around that rather than around a vendor slide.
Who it's for
Whether the audience is a customer, a member, a citizen, or your own team, the choice has to hold up under real use.
Fine-tuning on private documents, internal assistants and forecasting models. We usually start on rented Hopper GPUs in a Canadian region with a serving layer your IT team can audit, and plan reserved or owned capacity only once usage settles.
Residency and procurement rules come first. We favour in-country cloud regions, open-weight models and portable formats so a future vendor change does not strand the work, and we document the hardware choice in language an auditor can follow.
Many locations, thin margins per site. Edge devices such as Jetson or a Raspberry Pi 5 with an AI HAT+ handle in-store vision locally, with one central cloud model for the analytics head office needs. Fleet updates are planned from day one.
Speed and burn rate. We prototype on a workstation or an inference API, quantise aggressively, and delay owning any GPU until there is a paying audience. When investors ask about compute, you will have a written sizing rationale.
Related technology
AWS, Microsoft Azure and Google Cloud, plus Vercel and Cloudflare where a smaller footprint fits. Canadian…
Explore08Phones, tablets and laptops changed more in 2025 and 2026 than in the five years before, mostly because of…
Explore09From the workstations we build on to the kiosks, terminals, scanners and sensors your product ships with,…
ExploreFAQ
Almost never at the start. Rented cloud GPUs, an inference API or a single workstation cover prototyping and the first production release for most clients. Owning hardware only pays when utilisation is high and steady for many months, and it brings power, cooling, security and driver maintenance with it. We revisit the question once real usage data exists.
By memory first. If the model, its context window and the batch fit in an H100, that is the cheapest bookable option. H200 adds memory for larger models or longer contexts on the same software. Blackwell earns its place for multi-GPU training or very large serving jobs where the interconnect matters, and only where the region actually has stock.
Usually, yes. AWS, Google Cloud and Microsoft Azure all run Canadian regions with GPU instances, and OVHcloud has GPU capacity in Quebec. The catch is choice: the newest accelerators reach US regions first, so an in-country requirement can mean a previous-generation GPU or a longer wait. We confirm what is bookable in Canada during scoping.
Quantisation stores a model's weights in fewer bits, which shrinks memory use and speeds up inference. Going from 16-bit to 8-bit is usually invisible in output quality; 4-bit formats such as GGUF or AWQ trade a little accuracy for running on much smaller hardware. We measure the difference on the client's own evaluation set before committing.
Small and mid-size models can. Apple's M4 and M5 chips pair a Neural Engine with unified memory, so a MacBook can run a quantised open model locally, and iOS offers Core ML and the Foundation Models framework for on-device features. Larger models still need a server. We decide per feature: private and offline tasks on the device, heavy reasoning in the cloud.
When the device must act without a reliable connection, when video or sensor data is too large to stream, or when data must never leave the site. Jetson Orin and Thor handle real-time vision and small language models; Raspberry Pi 5 with an AI HAT+ or a Coral accelerator suits simpler detection at low power. Every edge project includes a plan for remote model updates.
Yes, when the fit is right. AMD Instinct offers more memory per card, Google TPU and AWS Trainium suit clients already on those clouds, and Groq or Cerebras give very fast inference over an API. NVIDIA stays the default because its software ecosystem is the broadest and hiring is easier. We keep the serving interface standard so a later switch is configuration, not a rewrite.
Let’s connect
Tell us what you’re building. We’ll reply within one business day with recommended platforms, structure and a tailored quote — no obligation.