MLflow: Track Every Experiment

What is MLflow?

MLflow concepts

Logging with MLflow in Azure ML

When you run a training script in Azure ML, MLflow tracking is automatic. Here’s how to use it:

import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, f1_score

# MLflow auto-connects to your Azure ML workspace
# No manual server configuration needed

# Start a run
with mlflow.start_run(run_name="rf-baseline"):
    # Log parameters (inputs)
    mlflow.log_param("n_estimators", 100)
    mlflow.log_param("max_depth", 10)
    mlflow.log_param("dataset_version", "v2")

    # Train the model
    model = RandomForestClassifier(n_estimators=100, max_depth=10)
    model.fit(X_train, y_train)

    # Log metrics (outputs)
    predictions = model.predict(X_test)
    mlflow.log_metric("accuracy", accuracy_score(y_test, predictions))
    mlflow.log_metric("f1_score", f1_score(y_test, predictions))

    # Log the model as an artifact
    mlflow.sklearn.log_model(model, "churn-model")

    # Log additional artifacts
    mlflow.log_artifact("feature_importance.png")

What’s happening:

• Line 10: Opens a named run — everything logged inside this block is grouped together
• Lines 12-14: Records the input configuration so you can reproduce this exact setup
• Lines 22-23: Records how well the model performed
• Line 26: Saves the model in MLflow’s standard format (can be deployed to any MLflow-compatible platform)
• Line 29: Saves additional files (charts, reports) alongside the model

# Find the best run across all experiments
runs = mlflow.search_runs(
    experiment_names=["genomics-variant-calling"],
    order_by=["metrics.f1_score DESC"],
    max_results=1
)
print(runs[["run_id", "params.model_type", "metrics.f1_score"]])

Autologging

MLflow can automatically log parameters and metrics for popular frameworks — no manual log_param calls needed:

# Enable autologging for scikit-learn
mlflow.sklearn.autolog()

# Just train the model — MLflow captures everything
model = RandomForestClassifier(n_estimators=100, max_depth=10)
model.fit(X_train, y_train)

Supported frameworks for autologging:

Comparing runs

One of MLflow’s most powerful features is comparing runs side by side:

# Search and compare runs
import mlflow

runs = mlflow.search_runs(
    experiment_names=["churn-prediction-v2"],
    filter_string="metrics.accuracy > 0.90",
    order_by=["metrics.f1_score DESC"]
)

# View top runs
print(runs[["run_id", "params.n_estimators", "params.max_depth",
            "metrics.accuracy", "metrics.f1_score"]].head(5))

What’s happening:

• Line 6: Filters to only runs with accuracy above 90%
• Line 7: Sorts by F1 score (descending) — best runs first
• Lines 10-12: Shows the key parameters and metrics for comparison

In the Azure ML Studio UI, you can also visually compare runs — select multiple runs and view metrics in parallel charts, scatter plots, or tables.

# Find the top 5 runs from the sweep
best_runs = mlflow.search_runs(
    experiment_names=["churn-sweep-apr-2026"],
    order_by=["metrics.f1_score DESC"],
    max_results=5
)

# Log the winner for the team
winner = best_runs.iloc[0]
print(f"Best run: {winner.run_id}")
print(f"  F1: {winner['metrics.f1_score']:.4f}")
print(f"  Learning rate: {winner['params.learning_rate']}")
print(f"  Max depth: {winner['params.max_depth']}")

Key terms flashcards

Knowledge check

Next up: AutoML & Hyperparameter Tuning — letting Azure find the best model for you.