Running a local LLM feels fiscally clever right up until the invoice arrives in the form of a GPU, a PSU, more RAM than your laptop has ever seen, and a workstation that starts ageing the moment you boot it. Token fees look painful because they are visible. Hardware costs are sneakier. They hide inside a capital purchase, then quietly rot in a cupboard while the next model family makes your shiny box look merely adequate.
That is the real tax on local AI. You do not stop paying. You just change the line item. Instead of feeding a frontier model by the token, you buy your way into the privilege of hosting inference yourself, then accept that the machine is likely to be obsolete before the accountant has finished smiling about the capex.
The seduction of owning the stack
There is a sensible case for running an LLM on your own machine. If you need tight control over data, predictable latency, or an internal workflow that cannot keep spraying prompts at an external API, local deployment starts to look attractive fast. A 7B or 8B model, properly quantised, can already do useful work for drafting, classification, extraction, search augmentation, and rough internal assistants. On a decent box, it is not a toy.
The mistake is assuming that the jump from “useful” to “financially superior” is automatic.
Frontier API pricing is visible and repeatable. Local infrastructure is neither. You buy the machine upfront, you pay for electricity, cooling, maintenance, downtime, and staff time, then you absorb the brutal little fact that AI hardware depreciates at a speed that would embarrass most corporate assets. If your plan depends on the machine holding value for five years, you are already telling yourself a story that the market will not tell back.
VRAM is the real bill
People love to talk about the model name. Llama 3. Mistral. Mixtral. That is the marketing layer. The practical layer is video memory.
For local inference, the big constraint is VRAM, because the model weights have to live close enough to the GPU to be processed efficiently. A 7B or 8B model in 4 bit quantised form usually lands around 8GB of VRAM. That is why consumer cards such as an RTX 3060 with 12GB or an RTX 4070 with 12GB can be perfectly respectable starting points.
Move up to something like Mixtral 8x7B, which sits in the roughly 50B class, and the numbers jump hard. A 4 bit version wants about 48GB of VRAM. Now you are either stitching together two consumer GPUs, such as a pair of RTX 3090s or RTX 4090s, or you are shopping in professional territory with a card like an NVIDIA A100 with 80GB. That is not a casual upgrade. That is a budget meeting.
Llama 3 70B is where the fantasy usually dies. A 4 bit quantised setup needs roughly 40GB of VRAM, while an 8 bit version climbs towards 80GB. One consumer card does not solve that. You are into high-end hardware, multi-GPU planning, and the sort of thermal and power management that makes a normal office PC look like a calculator.
Inference and fine tuning are different games
A lot of local LLM talk blurs two separate jobs together. Running a model and training a model are not the same thing, and the hardware gap between them is where many budgets go to die.
Inference is the easier task. It is the act of making the model answer questions, classify text, draft content, or power an internal assistant. For that, a single RTX 4090 with 24GB of VRAM is a serious and popular option. It can handle models up to roughly the 34B range in practical terms, depending on quantisation and context length. For a small team, that can be enough.
Fine tuning is nastier. The model is no longer just being used, it is being adjusted. That means more memory pressure because you are storing weights, gradients, and optimiser state. Even a 7B model can need more than 24GB of VRAM for full fine tuning. Parameter efficient methods such as QLoRA reduce the burden, which is useful, but they do not repeal physics. If you want to fine tune a 70B model, you are in professional multi GPU territory, typically involving interconnected NVIDIA H100 or A100 cards.
This is where the local AI pitch becomes slippery. A sales pitch will talk about “owning your model”. A working system asks a more annoying question: do you want to host inference, or do you want to train something that actually changes behaviour in a meaningful way? Those are different spend profiles, and the second one gets expensive in a hurry.
RAM and storage still matter
GPU VRAM gets all the attention because it is the bottleneck that breaks dreams first. System RAM and storage are the boring bits that break them second.
A bare minimum of 32GB of system RAM is sensible for local LLM work. If you are dealing with larger models, 64GB starts to look safer, and 128GB is where the machine stops feeling like it is gasping for air every time you load data, embeddings, or supporting services. If your AI box also needs to act as a development workstation, a vector store host, and a content pipeline machine, skimping on RAM is a false economy.
Storage matters because the files are enormous. A model like Llama 3 70B is about 140GB on disk. That is not something you want on a slow, tired drive. A fast NVMe SSD, ideally 1TB or more, keeps load times tolerable and stops the machine from spending half its life pretending to be a waiting room. If you are using multiple models, experiments, fine tuning checkpoints, and application data, 1TB disappears faster than finance expects.
What this costs in South Africa
This is where the pretty theory runs into local reality.
A capable AI workstation in South Africa is rarely a modest purchase. By the time you factor in import duties, VAT, distributor markups, and exchange rate wobble, the sticker price climbs quickly. A build that actually does something useful for local LLM work is usually a five figure or six figure rand purchase, depending on whether you are aiming for strong inference, serious fine tuning, or multi GPU headroom.
An RTX 3060 or RTX 4070 based setup can be kept relatively contained if your target is 7B and 8B class models. That still means a proper workstation, decent CPU, enough RAM, a large NVMe SSD, and a PSU that can keep up without sounding like it is lifting off. Move into RTX 4090 territory and the budget changes sharply. Move into dual 3090 or 4090 territory and the budget starts to look like an IT procurement decision, not a hobby purchase.
For South African businesses, the hidden sting is that the machine is bought in rand while its useful lifespan is measured against a global hardware cycle. The currency already works against you. The depreciation clock does the rest.
Depreciation eats the win
The local model argument often leans on savings. You stop paying per token. You gain independence from API pricing. You may even reduce privacy risk. All of that can be true.
It still does not rescue the asset from depreciation.
AI hardware loses value fast because the market keeps moving. New generations improve performance per watt, memory bandwidth, and inference throughput. What was a flagship card can become yesterday’s compromise remarkably quickly. A GPU that looks excellent on day one is often merely acceptable by year two, then awkward by year three, especially if your use case grows or the model sizes you care about keep increasing.
That means the purchase is not a stable long-term asset. It is a consumable with ambition. If you spend heavily on a local AI box, then amortise it over five years, you may be flattering yourself. In practice, the useful financial life is often shorter than the accountants would like and shorter than the hardware vendor would prefer you to notice.
The ugly part is that the machine keeps existing after the market has moved on. It does not become useless. It just becomes less defensible as a capital expense. That is the hardware tax. You are paying for the privilege of being early to a tool that ages like milk.
Fine tuning is the one real reason to do it
If local LLMs have a strong argument, it is fine tuning, not vague sovereignty theatre.
A business that needs a model trained on internal product language, customer service patterns, SEO templates, support articles, or sector-specific terminology can extract real value from a tuned local setup. A model that knows your data and your phrasing can outperform a generic API call in a narrow workflow. That is especially true where output consistency matters more than raw general intelligence.
For example, a South African agency might want a model tuned to write metadata and product copy in a house style, or a retailer might want internal classification of inventory descriptions and search queries without sending all of that data to a third party. A law firm, medical practice, or financial services company may care less about cleverness and more about keeping material inside its own environment.
That is the best local case. But even here, the hardware maths is not forgiving. Once fine tuning enters the picture, the rack room fills with expensive compromises: more VRAM, more cooling, more maintenance, more time spent making the infrastructure behave. The more specialised the model, the more likely you are to need the sort of hardware that makes the finance team ask whether you are building a product or a monument.
A simple cost comparison
Here is the question most teams should ask before buying anything:
| Option | Upfront cost | Ongoing cost | Flexibility | Risk |
|---|---|---|---|---|
| Frontier API | Low | Predictable per token | High, model updates handled externally | Vendor pricing changes |
| Local 7B or 8B inference | Medium | Electricity, support, replacement parts | Moderate | Hardware ages, limited scale |
| Local fine tuned 70B stack | Very high | High operational burden | High for narrow internal use | Rapid depreciation, integration complexity |
The table hides an important truth. The API bill feels infinite because it is visible. The hardware bill feels finite because it lands once. But if you are using AI seriously, the hardware does not stay paid for. It gets replaced, upgraded, expanded, cooled, repaired, and eventually retired long before the rest of the IT estate has finished arguing about refresh cycles.
When local makes sense
Local LLMs are not a bad idea. They are a niche idea that gets oversold.
They make sense when one or more of these are true:
- your prompts contain sensitive data that should not leave your environment;
- your workflow needs low latency and you cannot tolerate API dependency;
- you have enough workload to justify the machine’s idle time;
- you have a narrow fine tuning target that actually improves output quality;
- you already own the infrastructure and the marginal cost is acceptable.
They make less sense when the goal is simply to avoid API costs out of principle. If your usage is uneven, your team is small, or your model needs change every few months, paying for a depreciating GPU tower is a clumsy way to escape a token bill. You may save money on inference and lose it on capital expenditure, support overhead, and lost flexibility.
The blunt truth is that most businesses do not need to own the stack. They need to own the workflow.
The practical test before you buy
Before anyone signs off on local AI hardware, run the numbers like you mean it.
- Estimate monthly token spend for the current API workflow.
- Compare that with the full cost of a capable workstation, including VAT, import overhead, and local support.
- Add electricity, cooling, and staff time for setup and maintenance.
- Decide whether you need inference only, or whether fine tuning is the actual requirement.
- Check how fast the hardware will be obsolete relative to your model roadmap.
- Ask whether 32GB, 64GB, or 128GB RAM is realistic for the rest of the stack.
- Make sure storage is planned around model files that can reach 140GB for a single 70B model.
If the answer only works when you pretend the machine will hold its value, the answer does not work.
Local LLMs can be useful, especially when you need privacy, control, or tuned output for a narrow internal task. But the promise of escaping API costs comes with a hardware tax that is easy to underestimate and hard to reverse. In South Africa, where every imported component arrives with its own markup, the bet is even worse. A frontier model bill is annoying. A six figure GPU purchase that turns into a paperweight in five years is worse.
