The Workshop Diary is an ongoing series about how tools really come to be, in my own and clients’ projects, from idea through error to iteration. This is part 2; part 1 was about local AI and RAG.
Part 1 ended on a question I brushed aside: local AI keeps the data at home, but it demands power from the hardware. What if the power isn’t there, or if you need more than your own machine can give? You can buy power. But the moment you buy power, you hand over something far more important than computation.
Why a local model eats power
Running a language model, inference, is heavy work. The model consists of billions of parameters, and for answers to appear in a reasonable time those parameters have to fit into the graphics card’s memory (VRAM). That is the bottleneck that decides more than anything else: not so much the processor’s speed, but whether the model fits in memory or not.
You can shrink the size with quantisation, packing the parameters into a coarser precision, so a larger model fits into a more modest card at a small quality cost. In practice a moderate, quantised model and a decent consumer graphics card get you surprisingly far: single-user document search and RAG run well on that.
Three things, though, quickly push you up against the ceiling of your own machine: a clearly bigger and more accurate model, a large and constantly growing amount of data (vectorising is computation too), or many simultaneous users. A machine that is enough for one person chokes when ten ask at once. Then you face a choice: compromise on the model, or buy power elsewhere.
Three ways to buy power
There are roughly three routes, and they differ above all in how much data and responsibility they move outside.
1. A rented GPU server. You rent pure computation, a virtual server or a GPU instance, and run your own model and RAG on it. The data goes to the provider’s data centre, but the software, the models and the logic are yours. Control is high: you decide what runs and how the data is handled. Responsibility is shared, yours for the environment, theirs for the hardware.
2. A managed inference API. You use a ready-made model as a service: you send a question to an interface, you get an answer. The easiest and often the cheapest way, but your data passes through a third party’s model. Here you read especially carefully what the provider does with your inputs, whether it retains them, logs them, or uses them to train the model. The difference between “we don’t retain” and “we retain for 30 days against misuse” is decisive for sensitive data.
3. A private cloud. Your own isolated environment in the cloud, preferably in the EU, where you run your model under contract without sharing the model or hardware with anyone. The most expensive option, but closest to “your own machine” control when the scale grows large and a desktop no longer suffices.
In practice the best solution is often a hybrid: the most sensitive data and everyday use run locally on your own machine, and bought power is used only for heavy, rarer runs, for example a one-off vectorising of a large body of material, and only with data that can bear to leave. That way you pay for power only when you need it, and you don’t expose sensitive data needlessly.
When the data leaves the house, responsibility moves to the contract
The whole idea of part 1 was that the data does not leave the machine. The moment you buy power from outside, the data leaves, and security stops being a technical setting on your own machine and becomes a matter of contract.
That doesn’t mean control disappears. It means control moves from the machine to the negotiating table. This is an important distinction, because many people think moving to the cloud means giving up responsibility. It is the opposite: responsibility for the data stays with you, but the means of controlling it are now contract terms, not firewall rules.
What to read in the DPA
The GDPR’s Article 28 requires a data processing agreement (DPA) whenever an outside party processes personal data on your behalf. And even where there is no personal data, the same lines decide the fate of trade secrets. Five points I always read first:
Data location, where geographically the data rests and is processed (EU/EEA or elsewhere). Sub-processors, which sub-processors the provider uses and how changes to them are announced. Encryption, at rest and in transit, and who holds the keys. Deletion, whether you get your data out and destroyed when the contract ends, and with what guarantee. Use, often the most important line: whether the provider uses your inputs for its own purposes, such as training a model. Availability and the service level (SLA) are their own chapter, but for data protection it is these five that decide.
Where the data rests, and whose law reaches it
Location is more than a technical detail. It determines whose legislation your data falls under. Data held by a US provider can fall within the reach of US law (such as the CLOUD Act) even if the server sits in Europe, and transfers between the EU and the US have been legally shifting ground ever since the Schrems rulings.
The practical conclusion for a small operator is simple: if the data is genuinely sensitive, a service operating in the EU, under European law, is a safer starting point than a cheaper option whose jurisdiction is unclear. This is exactly where the local solution (part 1) is unbeatable: when the data never leaves the house, the question of jurisdiction doesn’t even arise.
The cloud takes on the computation, not the responsibility. The responsibility stays with you, and the only way to manage it is a contract you have actually read.
NIS2 makes your supplier your responsibility
Here many an SME is surprised. NIS2, and security-of-supply thinking more broadly, does not look only at your own security. It looks at your supply chain. When you buy computation or an AI service, that supplier becomes part of your security, and you have to be able to show that you have assessed it.
In practice that means a few concrete things: supplier due diligence before the contract, written security requirements in the contract, an agreed procedure for reporting incidents, and an exit plan in case the cooperation ends or the supplier goes under. Buying power does not outsource the risk. It adds a link to your supply chain that you have to manage just like your own environment. A chain is exactly as strong as its weakest link.
How to choose: the data class decides
A simple framework to guide the decision: don’t ask first “what is the cheapest”, ask “what data goes into this”. Once the data is classified, the right solution almost falls out by itself.
Public or low-sensitivity data (general guidance, public material), a managed API is entirely sufficient, cheap and effortless. Confidential data (trade secrets, contracts, pricing), an EU private cloud or a rented server, always with a DPA. Personal data, keep it local; if that is otherwise impossible, a strict DPA and data minimisation. And as I said in part 1: personal data is best excluded from the whole system entirely, already at the mapping stage.
So the decision-maker’s framework is clear: local when you can, bought power when you must, always a contract behind it, and the data class as the guide. But there is a large group of organisations whose data already lives in the cloud, in Microsoft 365. There the question is not “do we move to the cloud” but “who controls access”. That is the next part.
Part 3: SharePoint, the Graph API and Intune, when the data already lives in the cloud, the security boundary moves to identity.