There is a category of AI deployment that almost nobody talks about publicly. Not because it doesn’t exist but because the people doing it are under NDAs, operating in classified environments, or simply too busy keeping systems alive to write blog posts about it.

I’ve spent the last several years in that category. AI in defense environments. AI in medical environments under HDS certification. On-prem LLMs in air-gapped networks where a single dependency on an external API would be a security incident.

This blog is my attempt to write about it honestly.

“Can’t fail” is not a marketing phrase

When I say I build AI in places where it can’t fail, I mean something precise. I don’t mean “we have good uptime.” I mean environments where failure has consequences that can’t be undone: a misclassification in a triage system that delays treatment, an inference error in a threat detection pipeline that creates a false negative, a data exfiltration through a model endpoint in a classified network.

The mainstream AI industry has a framework for this: it’s called “safety.” But in practice, safety conversations in the industry mostly focus on LLM alignment, preventing models from saying harmful things. That’s a real problem, but it’s not the same problem.

The problem I work on is closer to what Charles Perrow described in Normal Accidents (1984): in complex, tightly coupled systems, failures are not exceptional, they are structurally inevitable.[1] Knight Capital Group lost $440 million in 45 minutes on August 1, 2012, because a deployment script accidentally activated legacy code that had been dormant for years.[2]

These weren’t AI systems. The AI version of these failures will be harder to diagnose, harder to attribute, and harder to prevent.

The gap nobody is naming

The current AI discourse operates at two poles. On one side: capability research — benchmarks, context windows, reasoning scores. On the other: governance debates about bias, copyright, and existential risk. Both are legitimate. Neither addresses the question I face every day: how do you deploy a model in a production environment where the cost of failure is not a bad user experience, but something irreversible?

The EU AI Act (Regulation 2024/1689) is the first serious legislative attempt to define “high-risk AI systems”, systems used in critical infrastructure, medical devices, law enforcement, and border control.[3] It’s a start. But legislation describes what is prohibited; it doesn’t tell you how to build. The FDA’s evolving guidance on AI/ML-based Software as a Medical Device (SaMD) is more operational, but it’s still catching up to what practitioners are actually deploying.[4]

The engineering standards that come closest to what I actually need were not written for AI. IEC 62304 governs medical device software lifecycle processes.[5] DO-178C covers software certification in airborne systems.[6] ECSS standards were developed for the European space sector.[7] These frameworks were built for deterministic systems, systems where you can trace a failure to a specific line of code. Neural networks don’t work that way.

That gap between the standards we have and the systems we’re building, is where I spend most of my time.

What high-constraint environments teach you

Defense environments don’t tolerate ambiguity in their tooling. Not because institutional culture is rigid (though it can be), but because ambiguity in a threat environment costs lives. What this translates to, technically, is a set of requirements that sounds simple and is extremely hard to meet:

• The system must behave predictably under adversarial conditions
• Dependencies on external services are not permitted
• The model must run on hardware that can be physically controlled and audited
• Explainability is not a nice-to-have, it’s a legal and operational requirement

The medical environment adds another layer: regulatory compliance under HDS (Hébergement de Données de Santé) in France[8], GDPR, and increasingly the EU AI Act’s medical provisions. A model that processes patient data cannot leave a certified perimeter. Period. This eliminates every major cloud AI provider from the conversation immediately.

What these environments share: the system is downstream of real-world consequences, and those consequences cannot be undone.

Illich’s question, applied to AI

In 1973, Ivan Illich published Tools for Conviviality.[9] His central argument was that tools; technologies, institutions, systems; exist on a spectrum. At one end are “convivial tools”: those that enhance human autonomy, that can be understood and controlled by the people who use them, that serve the user. At the other end are “manipulative tools”: those that create dependency, that require experts to operate, that serve the system rather than the person.

Illich was writing about medical systems, transport, and education. He was not writing about AI. But the question he was asking “who controls this tool, and in whose interest does it operate?” is exactly the question I ask about every system I build.

A model that runs on a vendor’s cloud infrastructure, that requires a continuous API connection, that can be updated or deprecated without the operator’s consent: that is a manipulative tool in Illich’s sense. The operator does not control it. They consume it.

An on-prem LLM running on hardware the organization owns, with weights that can be audited, with inference that can be explained and challenged: that is closer to a convivial tool. The organization can adapt it, constrain it, understand it.

This is not an argument against cloud AI for every use case. For a startup building a consumer product, cloud AI makes complete sense. But for a hospital system, a defense contractor, or a space agency, handing control of your inference pipeline to a third party is not a technical decision, it’s a sovereignty decision. And most of the people making it don’t frame it that way.

The design philosophy that follows

From these environments, I’ve developed a set of principles I apply by default:

Assume the network is hostile or absent. If a system requires external connectivity to function, it’s not deployable in half the environments I work in. Design for air-gap first; add connectivity as a deliberate feature.

Explainability is an engineering requirement, not a PR talking point. In regulated environments, a model that produces a result without a traceable rationale is not useful, it’s a liability. This shapes architecture choices before a single line of training code is written.

Rust is a design philosophy, not just a language. I rebuilt a security detection engine in Rust not because I enjoy fighting the borrow checker, but because memory safety is not negotiable in adversarial environments. The language enforces constraints that prevent entire categories of failure. That’s what a good tool does.[10]

Treat sovereignty as a first-class requirement. SecNumCloud (the ANSSI qualification for sovereign cloud infrastructure in France)[11] exists for a reason. Sovereign AI is not nationalism, it is the recognition that control over your inference stack determines who has access to your data, your decisions, and your operational continuity.

The failure mode that matters is the one you didn’t model. I keep Perrow’s thesis on my desk as a reminder: in tightly coupled systems, you don’t fail on the scenarios you planned for. You fail on the interactions between systems that nobody anticipated. The job is to reduce coupling, increase observability, and build in the assumption that something will go wrong in a way you didn’t predict.

Why I’m writing about it

C.A.R. Hoare, in his 1981 Turing Award lecture, said: “I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.”[12]

The mainstream AI industry has chosen the second path. I’m interested in the first one.

Go deeper

Two concrete things to walk away with, not just a manifesto:

• Framework: the NIST AI Risk Management Framework (AI RMF 1.0)[13] is the closest thing to an operational checklist for high-consequence AI systems available today. Start with its “Map” function, it forces you to classify your system’s failure modes before you write a line of training code, which is the same discipline Perrow’s thesis demands.
• Project: Ollama makes it trivial to run open LLMs entirely on hardware you control, with no external API dependency. If “on-prem” still feels theoretical, this is where to make it concrete.

Citations

[1]: Charles Perrow, Normal Accidents: Living with High-Risk Technologies (New York: Basic Books, 1984). [2]: U.S. Securities and Exchange Commission, In the Matter of Knight Capital Americas LLC, Administrative Proceeding File No. 3-15570 (October 16, 2013). [3]: Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 (Artificial Intelligence Act), Official Journal of the European Union. [4]: U.S. Food & Drug Administration, Artificial Intelligence and Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan (January 2021). [5]: IEC 62304:2006+AMD1:2015 — Medical device software: Software life cycle processes. International Electrotechnical Commission. [6]: RTCA DO-178C, Software Considerations in Airborne Systems and Equipment Certification (2011). [7]: European Cooperation for Space Standardization (ECSS), ECSS-Q-ST-80C — Software product assurance (2009). [8]: Décret n° 2018-137 relatif à l’hébergement de données de santé à caractère personnel. [9]: Ivan Illich, Tools for Conviviality (New York: Harper & Row, 1973). [10]: Nicholas D. Matsakis and Felix S. Klock II, “The Rust Language,” ACM SIGADA Ada Letters 34, no. 3 (2014): 103–104. [11]: Agence nationale de la sécurité des systèmes d’information (ANSSI), Référentiel SecNumCloud, version 3.2 (2022). [12]: C.A.R. Hoare, “The Emperor’s New Clothes,” Communications of the ACM 24, no. 2 (1981): 75–83. [13]: National Institute of Standards and Technology (NIST), Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (January 2023).