MLflow End-to-End: Experiment Tracking, Model Registry, and GitHub Actions CI/CD

You trained 47 models last month. You can't reproduce the best one because you didn't log the hyperparameters. MLflow costs 3 lines of code to fix this. But logging a metric is the tip of the iceberg. The real pain starts when you need to explain to your PM why the model that worked perfectly in staging is now throwing TypeError: expected float32, got int64 in production, or when your data scientist overwrites the production model because there's no approval gate.

This is where MLOps stops being a buzzword and starts being your lifeline. With 85% of ML projects never reaching production (Gartner 2025), ad-hoc scripts and local .pkl files are professional malpractice. We're going to build a hardened, professional pipeline using MLflow—the tool with 17M+ monthly downloads (PyPI stats, Jan 2026)—from experiment chaos to governed, CI/CD-driven deployment.

The 5 Things You Must Always Log (Or Your Future Self Will Hate You)

You fire up mlflow.start_run() and log a metric. Great. You've solved 10% of the problem. The other 90% is in the metadata you're probably ignoring. Here’s your non-negotiable checklist for every single run:

1. Hyperparameters & Architecture: Not just learning_rate, but the model class name and any configuration that defines its structure. This is your only ticket to reproduction.
2. Dataset Version: The exact commit hash from DVC (used by 60% of ML teams for dataset versioning (DVC user survey 2025)). Without this, you're debugging against a moving target.
3. Evaluation Metrics Across Slices: Log overall AUC, but also log auc_validation_region_north. Drift often hits specific segments first.
4. Model Signature: The single most effective guardrail against serving errors. More on this in a moment.
5. Environment Snapshot: conda.yaml or requirements.txt. That torch==1.9.0 vs torch==1.13.1 difference will matter.

import mlflow
import dvc.api


data_path = 'data/train.csv'
repo = 'https://github.com/your/repo'
rev = 'v1.0'  # or a git commit hash
data_url = dvc.api.get_url(path=data_path, repo=repo, rev=rev)

with mlflow.start_run(run_name="xgboost_final_v1") as run:
    # 2. Log PARAMETERS & DATA VERSION
    mlflow.log_params({
        "learning_rate": 0.1,
        "max_depth": 6,
        "model_class": "xgboost.XGBClassifier"
    })
    mlflow.log_param("data_url", data_url)  # Critical for lineage

    # Train model...
    model = xgb.train(params, dtrain)

    # 3. Log METRICS (overall and sliced)
    mlflow.log_metric("auc", 0.92)
    mlflow.log_metric("auc_region_north", 0.87)

    # 4. INFER & LOG SIGNATURE (Saves you from production type errors)
    from mlflow.models import infer_signature
    signature = infer_signature(train_features.head(), model.predict(train_features.head()))
    
    # 5. Log MODEL with signature and environment
    mlflow.xgboost.log_model(
        xgb_model=model,
        artifact_path="model",
        signature=signature,
        input_example=train_features.head(),  # Bonus: adds a concrete example
        registered_model_name="CreditRiskClassifier"  # Prep for registry
    )
    # Environment is automatically captured via `conda.yaml` or `pip requirements.txt`

Autologging: The Free Lunch That Isn't

Hit mlflow.autolog() and watch metrics, params, and models magically appear. For quick prototyping, it's a godsend. For production, it's a trap.

What you get for free: Basic parameters, metrics, and the model artifact with frameworks like sklearn, PyTorch, and XGBoost. It's perfect for that first experiment to see if an idea has legs.

What you miss (catastrophically):

• Dataset Versioning: Autolog doesn't know about your DVC pipeline.
• Business Logic Metrics: Your custom profit_per_decile metric won't be auto-logged.
• Artifact Overload: It logs everything—every checkpoint, every graph. Your S3 bill will notice.
• Signature Inference: It may not correctly infer complex signatures for custom prediction functions.

The rule: Use autolog for exploration, manual logging for production. Start a run with autolog to capture the basics, then manually log the critical, project-specific metadata shown in the code block above.

Your Get-Out-of-Jail-Free Card: Model Signature and Input Example

Here’s the exact error you prevent:

Error: TypeError: expected float32, got int64

Fix: You prevented it by inferring and logging the model signature during training.

A signature defines the schema of your model's inputs and outputs. Was the training data pd.DataFrame or np.ndarray? float32 or float64? The signature enforces this at serving time. The input_example goes further, providing a concrete sample that can be used for automatic testing.

When you load a model for serving, MLflow uses this signature to validate incoming requests. No more "it worked on my machine."

Model Registry: Staging → Production with an Approval Gate

The registry is where your model goes from a scientist's artifact to a production asset. The key is the stage lifecycle: None → Staging → Production → Archived.

The most common error message you'll hit here is:

Model version conflict: model 'CreditRiskClassifier' version 3 is already in stage 'Production'

Fix: You must transition the old Production version to Archived before promoting the new one. Do this in the MLflow UI or via the API: client.transition_model_version_stage(name="CreditRiskClassifier", version=2, stage="Archived").

Promotion should never be a manual click in a UI. It should be gated by a test suite and require approval. Here's how you set up a promotion workflow:

1. Register on evaluation: When a run meets a threshold (e.g., AUC > 0.90), register it as a new version.
2. Send to Staging: Automatically transition the new version to Staging.
3. Gate with Approval: Require a manual approval (or an integration with your CI/CD) to transition from Staging to Production. This is your chance to run integration tests, shadow deployments, or business reviews.
4. Archive the Old: As shown in the fix above, always clean up the previous production model.

GitHub Actions: Auto-Train on Data Change, Auto-Promote on AUC

This is where your pipeline becomes autonomous. We'll create two workflows:

1. on: [push] to data.dvc: Trigger training when the data changes.
2. on: [workflow_run] of training: Evaluate the resulting model and conditionally promote it.

Here’s a skeleton for the training trigger:

# .github/workflows/train-on-data-change.yml
name: Train Model on Data Update
on:
  push:
    paths:
      - 'data.dvc' # Trigger when DVC data file is updated

jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with: { fetch-depth: 0 } # Get full history for DVC
      - uses: actions/setup-python@v4
        with: { python-version: '3.10' }
      
      - name: Install DVC & Pull Data
        run: |
          pip install dvc dvc-s3
          dvc pull -v  # Pulls the actual dataset from S3
      
      - name: Train and Log to MLflow
        env:
          MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_TRACKING_URI }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        run: |
          pip install mlflow xgboost
          python train.py  # Your script that uses the code from section 1

The promotion workflow would run after training, call the MLflow API to evaluate the newly registered model on a holdout set, and if auc > 0.90, automatically transition it to Staging, perhaps pausing for a manual approval before Production.

Setting Up a Battle-Ready Tracking Server

mlflow ui on your laptop is not a tracking server. For teams, you need persistence and scalability.

• Backend Store: Use PostgreSQL. The default SQLite file corrupts under concurrent access.
• Artifact Store: Use S3 (or GCS/Azure Blob). Don't store models on the server's local disk.

A common deployment error:

DVC push fails: 403 Forbidden on S3

Fix: Check the IAM policy has s3:PutObject on the bucket, and verify AWS credentials with aws s3 ls <your-bucket>.

You can deploy the server via Docker, Kubernetes, or a managed service (Databricks, AWS SageMaker). The key is that your MLFLOW_TRACKING_URI points to this central server.

Multi-Team Setup: Conventions and Access Control

When the Data Science team's "experiment_5" collides with the NLP team's "experiment_5", chaos ensues.

Naming Convention (Enforce via PR review):

• Experiments: [team]-[project]-[objective] → fraud-predictor-churn-q3
• Runs: [algorithm]-[data-version]-[timestamp] → xgboost-dv1.2-20250115
• Models: [business-function]-[output] → credit-risk-classifier

Access Control: MLflow doesn't have built-in fine-grained RBAC. For serious multi-tenancy, you have two options:

1. Proxy + Database Per Team: Run an MLflow server per team (or with a separate PostgreSQL schema) behind a proxy that handles authentication.
2. Use a Managed Platform: Consider Databricks MLflow, which adds enterprise access controls on top of the open-source core.

Performance Benchmarks: Choosing Your Serving Engine

Once your model is in the registry, you need to serve it. MLflow's built-in serving is fine for testing, but for production throughput, you need a dedicated engine. Let's compare two heavyweights.

Benchmark: ResNet-50 inference on an 8-core CPU. Your mileage will vary.

The choice often comes down to your stack. For a heterogeneous model zoo, Ray Serve is more flexible. For a pure PyTorch shop, TorchServe is optimized. BentoML is another excellent contender, packaging models with all dependencies into a clean, scalable Docker image.

Next Steps: From Tracking to Full Production Monitoring

You now have a reproducible training pipeline and a governed registry. The next frontiers are:

1. Drift Detection: Remember, model drift causes 30% accuracy degradation within 6 months in production without monitoring (Evidently AI 2025). Integrate Evidently AI into your prediction pipeline to run statistical tests on incoming data vs. your training set baseline. Schedule reports or trigger retraining alerts.
2. Feature Store: Are you recomputing the same features in training and serving? A Feast feature store ensures consistency. The online store serves low-latency features for real-time inference, while the offline store feeds historical data for training.
3. Canary & A/B Testing: Use Seldon Core or KServe to deploy your new model alongside the old, routing 5% of traffic to it. Compare business metrics, not just accuracy, before a full rollout.
4. Continuous Training: The holy grail. Use Prefect or ZenML to orchestrate a pipeline that triggers on schedule, data drift, or performance decay. This is what reduces model redeployment time from 2 weeks to 4 hours on average.

Your MLflow tracking server is the system of record. Every other tool—the feature store, the orchestrator, the serving engine—should be configured to log back to it. This creates a single, searchable lineage from the S3 data path, through the experiment run, to the model version in production, and finally to the performance metrics and drift reports from live traffic. That’s how you move from 47 lost models to a single, accountable, and continuously improving production system.