In Why MLflow? I argued for running your experiment tracker on infrastructure you control. This post is the sequel, written for a specific reader: the CI/CD engineer who is very good at pipelines and has never trained a model. It walks through the full life of a model in a typical enterprise setup: data scientists working in a dev environment (a data science platform with notebooks, jobs and an embedded MLflow), two Git repositories, an artifact registry, and a Kubernetes or OpenShift cluster for production. Everything comes with an open source repository you can run on a laptop and deploy to a cluster: https://github.com/max-endeavour/mlOps.
I first presented this as a talk. The slides carry the big ideas and this article is the long version, with every detail, the code and the MLflow screens. The repository goes one step further: a multi project pipeline with Docker Compose, Kubernetes, Helm, OpenShift and CI templates.
TL;DR
- In ML you don’t ship code, you ship a model. Same commit plus different data gives a different model. The model is a versioned binary artifact, published once to an artifact registry, like any other dependency.
- There are two MLflow instances and they never talk. The dev one is for training and picking the best run. The prod one, in the cluster, is for watching the model in production. The artifact registry is the only bridge.
- The story has two acts. Act 1, Deliver, is CI/CD: a commit or a merge request triggers pipelines that end with an image in production. Act 2, Operate, is not CI/CD: it is run, it never ends, and it watches whether the model is still right.
- CI never retrains. It pulls a published version. A one-line merge request on
model.lockis the hand-off between the two repositories. - In production, only performance metrics tell you the model is wrong. Data and prediction drift are early warnings, real accuracy arrives weeks later, with the ground truth.
- Security controls belong along the whole chain, because most attacks on ML go through data and dependencies, not through code.
1. In ML you ship a model, not code
For a software engineer, a build is deterministic: the same commit produces the same binary. Machine learning breaks that assumption.
| Classic software | Machine learning | |
|---|---|---|
| Input | source code | code + training data + hyperparameters |
| Build step | CI build, a few minutes | training in the dev environment, minutes to hours, sometimes on GPU |
| Output | a jar, a wheel, an image | a trained model (here a model.pkl) |
| Reproducibility | same commit = same binary | same commit + other data = another model |
Three consequences follow for a CI/CD team:
- The model is an artifact in its own right. It gets a semantic version, it is published to the artifact registry (Artifactory, Nexus, S3…), and it is consumed like a binary dependency.
- CI does not build the model. It downloads a published version. The registry is the only link between the dev environment and the cluster.
- Green is not the same as right. A model service can be 100% healthy on the infrastructure side (pods up, latency fine) and still return wrong predictions. That is why we also monitor the model itself in production.
2. A glossary: MLflow concepts in CI/CD terms
Abbreviations used below: DS = data scientists, devs = developers who own the CI, prod = the production / operations team.
| ML / MLflow concept | CI/CD equivalent | What it is | Who does what |
|---|---|---|---|
| Experiment | A project / pipeline | All the attempts at one problem | DS create it |
| Run | One pipeline execution | One training: params, metrics, artifacts, commit | DS launch, dev MLflow records |
| Params / Metrics | Variables / test results | Hyperparameters in, quality scores out | DS choose and compare |
| Parent run + child runs | Matrix build | A sweep: N combinations compared | DS launch the sweep |
| Model Registry (dev MLflow) | Internal release branch | Models shortlisted before publication | DS pick the run |
| Model artifact | Versioned binary (jar, wheel) | Model + manifest + reference profile | DS trigger, devs publish |
manifest.json |
Build metadata | Run, commit, data, hyperparameters, metrics | The release CI writes it, audit reads it |
| Reference profile | Performance-test baseline | Data distributions at training time | DS compute it, a prod CronJob compares against it |
| Signature | API contract / OpenAPI | Input and output schema | DS define it, devs test it |
| Drift | Silent regression | Production data diverging from training data | Prod raises the alert, DS retrain |
3. The architecture: two worlds, one artifact, two acts

The colour of each step number tells you who owns it: purple for data scientists, orange for devs, red for the production team, green for steps shared between DS and devs.
Act 1: Deliver (CI/CD), steps 1 to 7
- Read data (DS). Training jobs in the dev environment read a dated snapshot of the datalake.
- Log runs and sweeps (DS). Every training attempt becomes a run in the dev MLflow.
- Pick the run (DS). Data scientists compare runs and choose the one to release.
- Push 1.4.0 (shared). The release pipeline of the training repository validates the run and publishes an immutable version to the artifact registry. DS trigger it; devs own and maintain it.
- Merge request on
model.lock(devs). The release pipeline opens an automatic MR in the serving repository that bumps one line: the model version. - Build image + staging (devs). The serving pipeline pulls that version, builds an image and deploys it to the staging namespace.
- CD, same image (prod). The exact same image (same digest) is promoted to production.
Act 2: Operate (run, not CI/CD), steps 8 to 12
- Log inferences (prod). Every prediction is logged with a request ID and the model version, into S3 compatible storage next to the cluster.
- Daily read (shared). A monitoring CronJob reads the day’s inferences and the reference profile that travelled inside the artifact.
- PSI + real AUC (shared). It computes drift and, once the ground truth is in, real performance, and logs one run per day into the prod MLflow.
- Alert thresholds (prod). Prometheus holds the thresholds and raises alerts.
- Alert → retrain (DS). A confirmed drift sends the data scientists back to act 1.
The two MLflow instances never talk to each other. The only thing that crosses from dev to prod is the versioned artifact: model, manifest and reference profile.
4. The running example: predicting churn
To make this concrete, the hands-on project predicts churn (attrition): a customer who cancels, leaves for a competitor or does not renew. For each customer the model gets four features (tenure in months, monthly charges, number of support calls, contract length) and returns a risk score between 0 and 1. For a customer with 4 months of tenure, €89.90 a month, 3 support calls and a monthly contract, model 1.0.0 answers 0.80.
Three notions matter for everything that follows:
- The model does not decide. It returns a risk; the action threshold (here 0.5, above which the customer gets a retention offer) is a business choice.
- Ground truth arrives late. You only know whether a customer really left weeks later, so real performance is always measured with a delay.
- The demo data is synthetic: 5,000 customers, roughly one in four churns.
5. Act 1: Deliver (CI/CD)
5.1 Training: every attempt becomes a tracked run
A run in the dev MLflow records:
- Params: the hyperparameters;
- Metrics: AUC, RMSE and friends;
- Artifacts: the serialised model, plots;
- Input dataset: the source and version of the data;
- Git commit: added automatically;
- User: who launched the training.
Access to training data. Training reads from the datalake, databases or platform datasets (snapshots mounted into the container). Data scientists have broad read access to history; that access must never be copied into production. For traceability, log_input or a dated snapshot plays the role of a lockfile for data.
Pitfall #1: training/serving skew. Preprocessing must be identical in training and in production. Ideally it lives inside the packaged model, so the serving code cannot reimplement it slightly differently.
For CI people: a run is a build job with its logs and artifacts, launched from the dev environment rather than by a push. It never leaves the dev environment.
5.2 Parameters, hyperparameters and sweeps
This is the distinction that confuses most newcomers.
- A parameter is learned by the model from the data during training. In a random forest, the variables and thresholds of every split; in a neural network, the weights. Nobody chooses them.
- A hyperparameter is chosen before training and controls how the model learns:
max_depth,n_estimators, a learning rate. It cannot be learned directly from the data, so we try several combinations: that is a sweep.
The CI/CD analogy: a hyperparameter is a compiler flag (-O2); a parameter is what ends up inside the binary. A sweep is a matrix build.
In MLflow, a sweep is one parent run with one child run per combination. In the repository the estimator and the grid come from each project’s project.yaml; simplified:
mlflow.set_tracking_uri(os.getenv("MLFLOW_TRACKING_URI")) # dev MLflow
mlflow.set_experiment(f"{project.name}-training")
df = project.load_dataset(data_version) # dated datalake snapshot
X_train, X_val, _, y_train, y_val, _ = split(df, ...) # test set kept for the release
with mlflow.start_run(run_name=f"sweep-{project.name}-{data_version}") as parent:
mlflow.log_input(mlflow.data.from_pandas(df, name=data_version), context="training")
mlflow.log_params({"data_version": data_version, "data_sha256": fingerprint(df)})
for values in itertools.product(*project.grid.values()): # max_depth x n_estimators
params = {**dict(zip(project.grid, values)), **project.fixed_params}
with mlflow.start_run(run_name=..., nested=True):
mlflow.log_params(params)
model = build_estimator(project.estimator, params).fit(X_train, y_train)
mlflow.log_metric("val_auc", roc_auc_score(y_val, model.predict_proba(X_val)[:, 1]))
pickle.dump(model, open("model.pkl", "wb")) # the "compiled" model
mlflow.log_metric("model_size_kb", os.path.getsize("model.pkl") / 1024)
mlflow.log_artifact("model.pkl")
Rules for a good sweep:
- Select on the validation set; the test set is used once, at release time.
- Same data and same random seed for every attempt.
- Log training time and model size, not just the score.
- Run the attempts in parallel as jobs, grouped under the same parent run.
- Keep heavy artifacts for the best run only.
- Only the best run leaves the dev environment: it becomes a release.
Here is what it looks like in the dev MLflow. The best run, rf-depth6-trees200, reaches a validation AUC of 0.7662 (highlighted in yellow):

Comparing attempts shows something CI people will appreciate: the scores are almost identical (0.75 to 0.77) while the pickle size ranges from 75 KB to 8.9 MB. The chosen run gets the best score for 1.8 MB.

The detail page of the chosen run shows its metrics, its hyperparameters (including data_version), its parent run, and the run ID that the release pipeline will receive:

5.3 Data versioning: every run knows what it learned from
Because the data is an input of the build, it needs a version too. The training script logs:
data_version: the name of the snapshot, filterable in MLflow;data_sha256: a fingerprint of the content; the release is refused if the data changed under the same name;data_rowsand the positive rate: a sanity check at a glance (here the churn rate drops from 26% to 19% between October and November).
These values are copied into the manifest and into a tag on the production side.

5.4 Release: a CI pipeline publishes the model
The release pipeline lives in the training repository. It has three stages:
- Trigger. Data scientists pick the run in the dev MLflow and start the pipeline with the project, the run ID and the version.
- Validate. A quality gate on the held-out test set. The thresholds live in the project’s
project.yamland are changed by merge request by the data scientists:
release_criteria: # owned by data scientists, changed by MR
min_test_auc: 0.70
max_model_size_kb: 20000
- Publish. Package, compute fingerprints, push to the registry with an immutable semantic version, and open the merge request on
model.lockin the serving repository.
# 1. Fetch the pickle of the chosen run
path = mlflow.artifacts.download_artifacts(run_id=RUN_ID, artifact_path="model.pkl")
model = pickle.load(open(path, "rb"))
# 2. Quality gate on the test set
test_auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
if test_auc < criteria["min_test_auc"]:
sys.exit("Quality gate failed")
# 3. Reference profile for drift monitoring
profile = {f: histogram_profile(X_train[f]) for f in FEATURES}
# 4. Fingerprints, package, push to the artifact registry
manifest["model_sha256"] = sha256("model.pkl")
tar.add("model.pkl"); tar.add("manifest.json"); tar.add("reference_profile.json")
registry.publish(archive, "churn-model", VERSION) # refuses to overwrite an existing version
The resulting manifest:
{
"name": "churn-model",
"version": "1.0.0",
"mlflow_run_id": "f86285e4...",
"training_data": "customers-2026-10",
"training_data_sha256": "4f3584b1d8b8",
"model_sha256": "30dc3cb3...",
"hyperparameters": { "max_depth": "6", "n_estimators": "200" },
"metrics": { "val_auc": 0.7662, "test_auc": 0.7407 }
}
This is the equivalent of mvn deploy. From here on, churn-model 1.0.0 is immutable in the registry. Versions accumulate (1.2.0 archived, 1.3.0 in production, 1.4.0 just published) and model.lock says which one the serving side wants.
Who does what: devs own the pipeline; data scientists decide which run and which thresholds. One thing to check early: the CI runner must be able to reach the API of the dev MLflow.
5.5 Two decoupled pipelines, linked by a merge request
training repo (dev side)
chosen run → quality gate → package → push registry → auto MR
│
MR "model.lock: 1.3.0 → 1.4.0", reviewed by a human ◄─┘
│
serving repo (prod side) ▼
MR merged → pull 1.4.0 → build image → tests → staging → CD
The decoupling gives you two independent change paths:
| New model, same code | New code, same model |
|---|---|
Only the model.lock line changes |
Security patch, new endpoint, new base image |
| Serving code is untouched | No retraining needed |
| Rollback = revert the MR | A classic serving pipeline, like any other service |
5.6 The artifact: the contract between dev and prod
churn-model-1.4.0.tar.gz
├─ model/ trained model
├─ requirements.txt
├─ signature.json
├─ manifest.json run, commit, data, hyperparameters, metrics
└─ reference_profile.json distributions at training time
The serving pipeline then does Pull (version read from model.lock), Build (minimal base image, non-root, Kaniko, Buildah or BuildKit), Test (smoke test, signature contract, scan) and Push (image churn-model:1.4.0).
Model format. Two reasonable choices:
- MLflow pyfunc:
load_model()+predict(). Simple, but the serving side needs the same Python libraries as training. - ONNX: a real compiled binary with no pickle, but preprocessing is sometimes hard to export.
Classic pitfalls: unpinned dependencies; preprocessing rewritten on the serving side; multi-gigabyte models and their pull time; and the fact that a pickle executes code when you load it (more on that in the security section).
The hyperparameters are already “compiled” into the model. The manifest is there for traceability and audit only.
5.7 Deployment: a microservice like any other, almost
The path to production is familiar: MR merged, CI builds and pushes the image, staging runs automated tests, CD promotes the same image to production, and the new model goes through shadow, then canary, then 100% of the traffic. The switch is decided by the production team, with the data scientists’ opinion.
The serving code reads the pinned version, checks the model fingerprint before unpickling, and logs every inference asynchronously:
MODEL_VERSION = open("model.lock").read().strip() # 1.0.0
with tarfile.open(f"churn-model-{MODEL_VERSION}.tar.gz") as t:
t.extractall(MODEL_DIR, filter="data") # done at image build time
manifest = json.loads((MODEL_DIR / "manifest.json").read_text())
if sha256(MODEL_DIR / "model.pkl") != manifest["model_sha256"]:
raise RuntimeError("model has been tampered with") # security
model = pickle.load(open(MODEL_DIR / "model.pkl", "rb"))
app = FastAPI()
@app.post("/predict")
def predict(customer: Customer, background: BackgroundTasks):
X = pd.DataFrame([customer.model_dump()])[FEATURES]
proba = float(model.predict_proba(X)[0, 1])
request_id = uuid.uuid4().hex
background.add_task(log_inference, { # asynchronous
"request_id": request_id, **customer.model_dump(),
"score": proba, "model_version": MODEL_VERSION})
return {"request_id": request_id, "score": proba, "model_version": MODEL_VERSION}
The serving code knows neither the dev environment nor MLflow, only an artifact version. In the repository the same app serves every project: the request schema is generated from the artifact’s signature.json, and the Dockerfile fetches and verifies the model at build time, so the running pod never talks to the registry.
Five things differ from a classic microservice:
- Slow start. The pod loads the model into memory: long
startupProbe, RAM sized to the model, a GPU node for deep learning. Liveness and readiness probes on/ping. - Test the answers, not only the service. In shadow mode, the new model receives real traffic and its predictions are compared with the current model’s, but never returned to the client. Canary then sends it a small share of the traffic, then all of it.
- Rollback = revert the
model.lockMR, or go back to the previous image. No rebuild between staging and production: same digest. - Production data access. Often the features arrive in the request. Otherwise: a dedicated service account, read-only, secrets in Kubernetes or Vault. Never the data scientists’ access rights.
- Every inference is logged with a request ID and the model version. That log is the fuel of act 2.
The usual tooling applies: Helm, Kustomize or Argo CD; Deployment, Service, Ingress or Route, HPA. The repository ships a Helm chart that creates all of it per project, plus the monitoring CronJob of act 2.
6. Act 2: Operate, this is not CI/CD anymore
This is where CI/CD people get confused, so let me be explicit. Everything in act 1 stops when the image runs in production. Act 2 starts at that moment and never stops.
| Act 1 · Deliver (CI/CD) | Act 2 · Operate (run) | |
|---|---|---|
| Trigger | a commit, an MR, a release | time: every day, continuously |
| Duration | a pipeline that ends | as long as the model is in production |
| Question | is the deliverable good? | is the model still right? |
| Tools | CI pipelines, artifact registry, Argo CD | CronJobs, S3, prod MLflow, Prometheus |
| Owned by | devs (CI), prod (CD) | prod (operates), DS (analyse) |
| Output | an image in production | an alert, then a retraining |
Monitoring does not run in any pipeline. It is run, like the supervision of any service, except that we also watch whether the answers are right. When the model drifts, act 2 sends work back to act 1: retraining, a new release, a new pass through the pipelines.
6.1 The monitoring circuit and who owns it
Serving logs every inference (devs) → S3 storage, Parquet per day, plus the ground truth (prod operates) → a daily drift-monitor CronJob computes PSI and real performance (devs write it) → the prod MLflow keeps one run per day (prod operates) → alerting (prod) → data scientists decide whether the drift is real and whether to retrain.
| Data scientists | Devs / CI | Production team |
|---|---|---|
| Define metrics and drift thresholds | Log every inference in the serving code | Operate S3, prod MLflow and alerting |
| Read the runs in the prod MLflow | Write and maintain the CronJobs | Receive and route alerts |
| Decide on retraining | Join the ground truth by request_id |
Set retention and GDPR rules |
The prod MLflow is only a journal. It collects nothing, computes nothing in real time and raises no alerts. Serving collects, the CronJob computes, alerting warns. On Kubernetes the natural alerting stack is Prometheus + Alertmanager (built in on OpenShift as user workload monitoring); in the repository the CronJob simply fails with a dedicated exit code on drift, which your alerting already watches, and can post to a webhook.
How to store what monitoring needs. You do not need the test set in production: the reference is the profile shipped inside the artifact. Store inferences in S3 inside the cluster, as Parquet partitioned by model version and day. Store the ground truth in a table (Postgres works) and join it by request_id. Pseudonymise, and set a retention period (GDPR).
6.2 Four levels of monitoring
| Level | Question | What you look at | Who |
|---|---|---|---|
| 1. Technical | Does the service answer? | latency, errors, CPU / RAM | prod / devs |
| 2. Data drift | Do inputs still look like training? | feature distributions, unknown values | DS |
| 3. Predictions | Are outputs drifting? | score distribution | DS |
| 4. Real performance | Is the model right? | metric recomputed when ground truth arrives | DS + business |
Level 1 lives in Prometheus / Grafana. Levels 2 to 4 are computed by CronJobs, recorded in the prod MLflow and alerted through Prometheus.
6.3 A short metrics primer
The confusion matrix. Model 1.0.0 on the test set: 1,000 customers, 245 of whom left. Decision threshold: score ≥ 0.5.
| Predicted: leaves | Predicted: stays | |
|---|---|---|
| Actually left | 78 true positives (churners caught) | 167 false negatives (churners missed) |
| Actually stayed | 61 false positives (false alarms) | 694 true negatives (loyal customers, correctly left alone) |
| Metric | Formula | Question it answers | Value |
|---|---|---|---|
| Accuracy | (TP + TN) / total | What share of predictions is right? | 0.772 |
| Precision | TP / (TP + FP) | When the model raises an alert, is it right? | 0.561 |
| Recall | TP / (TP + FN) | What share of real churners does it catch? | 0.318 |
| F1-score | 2 × P × R / (P + R) | Harmonic mean of precision and recall | 0.406 |
Two traps. First, a model that predicts “nobody leaves” already scores 0.755 accuracy, so accuracy alone says almost nothing. Second, all four metrics depend on the threshold: at 0.3, recall climbs to 0.653 and precision drops to 0.416. The threshold is a business decision.
AUC: can the model rank customers by risk? The model does not answer yes or no; it gives a risk score. A good model puts the customers who leave at the top of the list. The simplest way to read AUC is the pair test: pick at random one customer who left and one who stayed. Does the one who left have the higher score? Repeat many times; the success rate is the AUC. Model 1.0.0 scores 0.74: 74 pairs out of 100 are correctly ordered.

Rough reading: 0.5 is a coin toss, 0.7 is useful, 0.8 is good, 0.9 and above is excellent, or suspicious (look for leakage). The name “area under the curve” comes from the ROC curve, which plots the true-positive rate against the false-positive rate as the threshold varies, and the pair test gives the same number (0.742 here, by simulation). Because it does not depend on the threshold, AUC is used both to compare models (val_auc) and to follow production (perf_auc).
PSI: have the customers changed since training? A model learns on one population. If the population changes, the model extrapolates, like a poll whose sample no longer looks like the country. The Population Stability Index measures that in three steps:
- Cut the training customers into 10 buckets of monthly price, 10% each.
- Every day, compute the share of production customers that falls into each bucket.
- Every gap adds points; the total is the PSI.

Formally, PSI = Σ (prod share – reference share) × ln(prod share / reference share) over the buckets. Each term is positive or zero, so a gap in either direction adds points. On 8 November, the €91 to €134 bucket went from 10% to 52% (+0.70) and the €15 to €40 bucket from 10% to 0.8% (+0.23); the total over the 10 buckets is 1.46. The usual thresholds, from credit scoring: below 0.1, same population; 0.1 to 0.2, worth watching; above 0.2, a different population.
PSI’s big advantage: you do not have to wait to know who left. It alerts the same day, whereas real AUC is only available when the ground truth arrives, sometimes weeks later.
6.4 Three families of metrics, only one measures performance
| Family | Metrics in the prod MLflow | Question | Available | What it triggers |
|---|---|---|---|---|
| Data | data_psi_<feature>, data_n_requests |
Have the customers changed? | same day | investigation: business change, feed bug? |
| Predictions | pred_psi, pred_mean_score |
Does the model answer differently? | same day | early warning |
| Real performance | perf_auc, perf_precision, perf_recall, perf_f1, perf_accuracy |
Was the model right? | when ground truth arrives, weeks later | decision to retrain |
Data and prediction metrics are warning signals, available immediately. Only the third family tells you the model is wrong. In the demo the ground truth is simulated the same day; in real life it arrives weeks later.
The daily CronJob, simplified:
def psi(values, ref): # 0 = stable, > 0.2 = drift
edges, expected = np.array(ref["bins"]), np.array(ref["freq"])
counts, _ = np.histogram(np.clip(values, edges[0], edges[-1]), edges)
actual = counts / counts.sum()
e, a = np.clip(expected, 1e-4, None), np.clip(actual, 1e-4, None)
return float(np.sum((a - e) * np.log(a / e)))
# 1. Data: have the customers changed?
metrics = {f"data_psi_{f}": psi(df[f], profile[f]) for f in FEATURES}
# 2. Predictions: does the model answer differently?
metrics["pred_psi"] = psi(df.score, profile["prediction"])
# 3. Performance: was it right? (ground truth, joined by request_id)
metrics["perf_auc"] = roc_auc_score(truth.churn, truth.score)
metrics["perf_precision"] = precision_score(truth.churn, truth.score >= 0.5)
mlflow.set_tracking_uri(os.getenv("MLFLOW_TRACKING_URI")) # prod MLflow
with mlflow.start_run(run_name=f"monitoring-{DAY}"):
mlflow.set_tags({"model_version": VERSION, "drift_status": status,
"training_data": manifest["training_data"]})
mlflow.log_metrics(metrics)
Note that production never needs the training data: the reference profile travels inside the artifact. The real script also keeps one “timeline” parent run per model version, so MLflow draws day-by-day curves, with one nested run per day.
6.5 The demo: 14 days of drift in the prod MLflow
The simulated scenario: 14 days, 400 requests per day. From 1 November, prices go up (data drift) and support calls stop predicting churn (concept drift). The screenshots come from the first version of the hands-on; in the repository the metric and run names are generic (score instead of churn_proba, pred_mean_score instead of pred_mean_risk, one experiment per project), and make demo reproduces the same story for both example projects.

- On 3 November,
data_psi_monthly_chargescrosses 0.2 anddrift_statusswitches to ALERT. - The tag
training_datashows which data the production model learned from.
Data and predictions drift first:

data_psi_monthly_chargesis flat, then takes off from day 8 up to 1.46: prices changed.pred_psirises to 0.27: the model’s scores shift too.- the mean score goes from about 0.25 to 0.33: the model sees more customers at risk. Is it right? Only performance will tell.
Then real performance degrades (first 8 days vs last 4):

perf_precision: 0.63 → 0.37. When the model raises an alert, it is wrong almost two times out of three.perf_auc: 0.75 → 0.69. It ranks customers less well.perf_accuracy: 0.77 → 0.75. Almost flat: it sees nothing, which is exactly the accuracy trap from the primer.
Decision: retrain on customers-2026-11 and publish 1.1.0, which beats 1.0.0 by 0.02 to 0.04 AUC on the last three days. That is act 2 handing work back to act 1.
7. Security: attacking a model without touching the code
Most attacks on ML do not go through a code vulnerability but through data or dependencies, so controls have to sit along the whole chain, not only at deployment.
| Attack | Consequence | Data | Training | Release CI | Serving | Monitoring |
|---|---|---|---|---|---|---|
| Poisoned data | biased model or backdoor | key | key | backup | backup | |
| Vulnerable or trojaned library | code executed in dev or in the cluster | backup | key | key | ||
| Replaced model, trojaned pickle | code executed at load time | key | key | |||
| Prompt injection (LLM) | the assistant obeys the attacker | backup | backup | key | key | |
| Crafted inputs (evasion) | undetected fraud or risk | backup | key | key | ||
| Model theft by querying | copy of the model, lost advantage | key | backup | |||
| Training data leakage | GDPR breach, exposed secrets | key | key | backup | ||
| Executed LLM output, over-powered agent | unwanted actions (SQL, mail, API) | backup | key | backup | ||
| Unbounded consumption (LLM) | cost, denial of service | key | key |
Key = main control point; backup = defence in depth. OWASP mapping: poisoning = ML02 / LLM04; libraries = ML06 / LLM03; replaced model = ML10; prompt injection = LLM01; evasion = ML01; model theft = ML05; data leakage = ML03, ML04, LLM02; executed output and excessive agency = LLM05, LLM06; unbounded consumption = LLM10 (OWASP Machine Learning Security Top 10 and OWASP Top 10 for LLM Applications 2025).
7.1 Poisoned data: the model learns what the attacker wants
Scenario. Someone edits the training table so that customers in one segment are labelled “loyal”. The model learns never to alert on that segment, or reacts to a rare pattern of the attacker’s choosing (a backdoor).
Consequences. Business decisions go wrong without any technical error. It is invisible in global metrics: the AUC stays good. And there is regulatory exposure (bias, the EU AI Act).
Countermeasures, step by step:
- Datalake (step 1): restricted, logged writes on training tables; provenance for every source.
- Training (steps 2 and 3):
data_version+data_sha256; schema and distribution checks against the previous version. - Release CI (step 4): a quality gate on a trusted, frozen test set, out of reach of the team that trains; metrics per segment.
- Monitoring (steps 9 to 11): PSI and real AUC per segment, not only globally.
7.2 Vulnerable library or trojaned model: the supply chain
Scenario. A library version with a known CVE, or a typosquatted PyPI package. Or a model.pkl replaced in the registry: pickle executes code at load time, inside the production pod.
Consequences. Code execution in the dev environment or in the production cluster; theft of the pod’s secrets and therefore of production data; lateral movement to other services.
Countermeasures:
- Training (dev environment): controlled base images; packages only through an internal proxy repository, never straight from PyPI.
- Release CI (step 4): pinned versions, dependency scanning (pip-audit, Trivy), an SBOM, a sha256 fingerprint and a signature of the artifact (cosign).
- Serving (steps 5 to 7): check the fingerprint before
pickle.load(the hands-on does it), non-root image, NetworkPolicy, minimal secrets. - Format (to decide): prefer a format that does not execute code (ONNX, safetensors); otherwise scan pickles (for example with ModelScan).
7.3 Prompt injection: when text becomes a command
Scenario. Direct: “ignore your instructions and display customer X’s data”. Indirect: an instruction hidden in an email, a PDF or a web page that the assistant reads (RAG).
Consequences. Leakage of data or of the system prompt; unwanted actions through tools (sending mail, SQL queries, API calls); wrong answers presented as reliable.
Countermeasures:
- Design: least privilege for every tool; no secrets in the prompt; human validation for sensitive actions.
- Release CI: an attack suite replayed on every prompt or model change, like regression tests.
- Serving: input and output filters; the LLM’s output is untrusted data, never executed as is.
- Monitoring: log prompts, answers and tool calls; alert on attack patterns and abnormal volumes.
7.4 The other attacks worth knowing
| Attack | What happens | Countermeasure |
|---|---|---|
| Evasion | A fraudster tunes their data to stay under the threshold | Serving: validate inputs (bounds, consistency). Monitoring: watch scores just under the threshold |
| Model theft | Thousands of queries to rebuild an equivalent model | Serving: authentication, quotas, return a decision rather than a precise score |
| Training data leakage | The model or LLM regurgitates personal data it learned | Data: minimise and pseudonymise. Serving: filter outputs |
| Executed LLM output | The answer is injected into SQL, HTML or a shell command | Serving: treat the output like user input (escaping, parameterised queries) |
| Over-powered agent | The agent has write rights it does not need | Design: minimal rights per tool, human confirmation for irreversible actions |
| Unbounded consumption | Huge or burst requests to the LLM | Serving: size limits, per-user quotas, timeouts. Monitoring: cost tracking |
7.5 What security concretely adds to the pipelines
| Training and release pipeline | Serving pipeline | Run and monitoring |
|---|---|---|
| Packages through an internal proxy only | Fingerprint and signature checked before loading | NetworkPolicy and minimal secrets |
| Dependency scanning and SBOM | Image scan, minimal base, non-root | Log of inferences and prompts |
data_version + data_sha256 check |
Input validation (schema, bounds) | PSI and real AUC per segment |
| Quality gate on a frozen test set, per segment | LLM: attack suite replayed on every version | Alerts on abnormal volumes and costs |
| sha256 fingerprint and artifact signature | Authentication and quotas on the API | A procedure to withdraw a model (rollback MR) |
The principle: treat a model, and even more an LLM, as an untrusted component. Limit what it can do, verify what you load, watch what comes out.
8. Organisation: who does what (RACI)
R = Responsible (does the work), A = Accountable (decides; exactly one per row), C = Consulted before, I = Informed after. Step numbers refer to the architecture diagram.
| Activity | Steps | Data scientists | Devs / CI | Production team |
|---|---|---|---|---|
| Train, launch sweeps, compare runs | 1, 2 | A/R | ||
| Pick the run and trigger the release | 3 | A/R | I | |
| Set the quality-gate thresholds | 4 | A/R | C | I |
| Maintain the release CI pipeline | 4 | C | A/R | |
Review and merge the model.lock MR |
5 | C | A/R | I |
| Build the image, test, deploy to staging | 6 | I | A/R | I |
| Go to production, run canary and rollback | 7 | I | C | A/R |
| Production data access, secrets, network flows | 8 | C | A/R | |
| Inference storage: S3, retention, GDPR | 8 | C | R | A |
| Write the monitoring CronJobs | 9, 10 | C | A/R | I |
| Define drift metrics and thresholds | 9 to 11 | A/R | I | C |
| Operate the prod MLflow and Prometheus | 10, 11 | I | C | A/R |
| Receive and route alerts | 11 | I | I | A/R |
| Analyse drift, decide on retraining | 12 | A/R | I | C |
The only row where A and R sit in different teams, inference storage, is the interface to settle first.
9. Decisions to make before the first model reaches production
- Can the CI runner call the API of the dev MLflow?
- Who can publish a release, and on which numeric criteria?
- Model format: MLflow pyfunc or ONNX?
- Is the preprocessing inside the packaged model?
- Are dependencies pinned, and is the dev environment aligned with the serving base image?
- Real time (API) or batch (CronJob)?
- Where are inferences stored, for how long, under which GDPR rules?
- Which drift thresholds, and who receives the alerts?
- Do data scientists get access to the prod MLflow UI?
- Is copying inferences back to the datalake for retraining approved?
10. Hands-on: from a laptop to a cluster
Everything above is in an open source repository: github.com/max3ndeavour/mlops-project. It started as the small demo behind the screenshots and grew into a complete, multi project pipeline you can copy:
| Folder | What it holds |
|---|---|
mlops_core/ |
the shared code: train.py (sweep + data versioning), release.py (quality gate + publish), registry.py (local folder, HTTP registry such as Artifactory or Nexus, or S3), serving.py (FastAPI, sha256 check, inference log), monitor.py (data, prediction and performance metrics) |
projects/ |
one folder per model: churn, fraud and a _template. A project is a project.yaml (features, estimator, grid, thresholds), a data.py and a model.lock |
steps/ |
the guide, one page per step, from the two MLflow servers to the cluster |
deploy/ |
Dockerfiles, Docker Compose (MinIO, two PostgreSQL, two MLflow), kustomize for MLflow dev and prod, a Helm chart per model with its monitoring CronJob, OpenShift notes |
ci/ |
release and serving pipelines for GitLab CI and GitHub Actions, including the automatic merge request on model.lock |
On a laptop:
git clone https://github.com/max3ndeavour/mlops-project.git && cd mlops-project
python -m venv .venv && source .venv/bin/activate
make install
make test # end to end tests, no server needed
make mlflow-up # dev MLflow on :5001, prod MLflow on :5002
make demo # two projects: sweeps, releases, 14 days of traffic with drift, daily monitoring
Step by step, for one project:
make train PROJECT=churn DATA=customers-2026-10 # step 2-3, dev MLflow
make release PROJECT=churn RUN=<best_run_id> VERSION=1.0.0 # step 4-5, quality gate + registry
make serve PROJECT=churn # step 6, serves projects/churn/model.lock
python scripts/simulate_traffic.py projects/churn --days 14 # step 8, demo traffic with drift
make monitor PROJECT=churn DAY=2026-11-08 STEP=14 # step 9-11, prod MLflow
make new-project NAME=upsell # step 12, the next model
On a cluster, one Helm release per project:
kubectl apply -k deploy/kubernetes/mlflow/overlays/prod
helm upgrade --install churn deploy/helm/model-service -n mlops-prod \
-f deploy/helm/values/churn.yaml --set modelVersion=$(cat projects/churn/model.lock)
What stays simplified: the data is synthetic (replace load_dataset() with a read of your datalake), the models are scikit-learn pickles (keep the sha256 check, or move to ONNX), and the demo writes the ground truth on the same day instead of weeks later.
Conclusion
If you come from CI/CD, three ideas carry most of the weight. The model is a binary dependency, built elsewhere and pulled by version. The hand-off between data science and delivery is a one-line merge request on model.lock. And once the model is in production, a second story starts that no pipeline covers: watching whether the model is still right, and sending work back to the start when it is not.
MLflow sits on both sides of that story (in the dev environment to choose, in the cluster to watch) and never as the link between them. That link is, and should stay, a boring, immutable, versioned artifact.
The code, the deployment templates and the step by step guide are on GitHub: github.com/max3ndeavour/mlops-project.
Leave a Reply