Secrets & Authentication
Access Azure Key Vault secrets from Databricks, authenticate with service principals, and use managed identities — the three authentication patterns the exam expects you to know.
Why secrets and authentication matter
Never put passwords in your code. Ever.
Think of it this way: your notebook is like a recipe card. You write “add the secret spice” — but you don’t write the actual spice name on the card where anyone could read it. Instead, you keep the spice in a locked cabinet (Azure Key Vault) and only people with the right key can open it.
Similarly, when your pipeline needs to connect to a database or storage account, it shouldn’t carry credentials around like a sticky note. Instead, it uses a service principal (an ID card for apps) or a managed identity (an ID card that Azure manages for you — no password at all).
Azure Key Vault secrets
How secret scopes work
Databricks uses secret scopes as a bridge to Azure Key Vault:
Notebook code
→ dbutils.secrets.get("scope-name", "secret-key")
→ Secret scope (Databricks)
→ Azure Key Vault (stores the actual secret value)Setting up a Key Vault-backed secret scope
- Create an Azure Key Vault and add your secrets
- Create a secret scope in Databricks that points to the Key Vault
- Reference secrets in code using
dbutils.secrets.get()
# Read a secret from Key Vault via the secret scope
storage_key = dbutils.secrets.get(scope="kv-production", key="adls-access-key")
# Use it in a connection (the value is never printed or logged)
spark.conf.set(
"fs.azure.account.key.myaccount.dfs.core.windows.net",
storage_key
)Critical security feature: Secret values are redacted in notebook output. If you try to print(storage_key), Databricks shows [REDACTED] — not the actual value.
Exam tip: Secret scope types
There are two types of secret scopes:
| Type | Backend | Use Case |
|---|---|---|
| Azure Key Vault-backed | Azure Key Vault | Production — centralised secret management |
| Databricks-backed | Databricks internal store | Simple setups — secrets stored in Databricks |
The exam strongly favours Key Vault-backed scopes because they integrate with Azure’s security and compliance tooling (audit logs, access policies, key rotation).
Service principals
A service principal is an application identity in Microsoft Entra ID. It’s like giving your ETL pipeline its own ID badge instead of using a human user’s credentials.
When to use service principals
| Scenario | Why Service Principal |
|---|---|
| Automated ETL pipelines | No human to log in; pipeline needs its own identity |
| Cross-workspace access | Service principal can be granted access across workspaces |
| External system integration | Azure Data Factory, Logic Apps connecting to Databricks |
| Unity Catalog automation | Grant table permissions to pipelines, not people |
Configuring a service principal
# Authenticate ADLS access using a service principal
spark.conf.set("fs.azure.account.auth.type.myaccount.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.myaccount.dfs.core.windows.net",
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.myaccount.dfs.core.windows.net",
dbutils.secrets.get("kv-production", "sp-client-id"))
spark.conf.set("fs.azure.account.oauth2.client.secret.myaccount.dfs.core.windows.net",
dbutils.secrets.get("kv-production", "sp-client-secret"))
spark.conf.set("fs.azure.account.oauth2.client.endpoint.myaccount.dfs.core.windows.net",
"https://login.microsoftonline.com/<tenant-id>/oauth2/token")Notice how the client ID and secret come from Key Vault — never hardcoded.
Tomás uses a service principal at NovaPay for the fraud detection pipeline. The pipeline runs on a schedule with no human interaction — it authenticates using its service principal to access both ADLS storage and Unity Catalog tables.
Managed identities
A managed identity is the simplest authentication method — Azure manages everything:
- No credentials to store or rotate — Azure handles token generation
- No Key Vault needed for the identity itself (though you still use Key Vault for other secrets)
- Two types: system-assigned (tied to one resource) and user-assigned (reusable across resources)
| Feature | Service Principal | Managed Identity |
|---|---|---|
| Credential management | You manage client secret/certificate | Azure manages automatically |
| Secret rotation | You must rotate | Azure rotates automatically |
| Stored in | Entra ID app registration | Tied to Azure resource |
| Cross-tenant | Yes (multi-tenant app) | No (same tenant only) |
| Best for | Cross-workspace, cross-tenant, ADF | Storage access, Azure-to-Azure |
| Exam preference | When question mentions 'automated pipeline' | When question mentions 'no credentials' or 'least management overhead' |
# Configure ADLS access using managed identity (much simpler!)
spark.conf.set("fs.azure.account.auth.type.myaccount.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.myaccount.dfs.core.windows.net",
"org.apache.hadoop.fs.azurebfs.oauth2.ManagedIdentityTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.myaccount.dfs.core.windows.net",
"<managed-identity-client-id>")Dr. Sarah Okafor configures Athena Group’s Databricks workspace to use a user-assigned managed identity for accessing ADLS Gen2 storage. No secrets to rotate, no Key Vault entries to maintain for storage access.
The modern UC approach: storage credentials and external locations
The spark.conf.set(...) patterns above are direct file access — they bypass Unity Catalog governance. The modern, exam-preferred approach uses Unity Catalog’s built-in storage management:
| Concept | What It Does |
|---|---|
| Storage credential | A UC object that wraps a managed identity or service principal for storage access |
| External location | A UC object that maps an ADLS path to a storage credential |
| Access Connector | An Azure resource that provides a managed identity for Databricks to access storage |
Exam tip: UC-governed access vs direct file access
Direct file access (spark.conf.set patterns above) gives you full control but bypasses Unity Catalog governance — no lineage, no audit, no centralised permissions.
UC-governed access (storage credentials + external locations) integrates with Unity Catalog’s permission model. External tables and volumes use external locations, which reference storage credentials. This is the recommended pattern for production and is heavily tested on the exam.
If the question mentions “governance,” “audit,” or “centralised access management” — UC storage credentials are the answer.
Exam decision tree: which authentication method?
Follow this logic for exam scenarios:
- “Store and retrieve secrets/passwords” → Azure Key Vault + secret scope
- “Automated pipeline needs identity” → Service principal
- “Minimise credential management” or “no secrets to rotate” → Managed identity
- “Cross-tenant access” → Service principal (managed identities don’t cross tenants)
- “Access from Azure Data Factory” → Service principal or managed identity (ADF supports both)
When the question doesn’t specify constraints, managed identity is usually the “best practice” answer because it eliminates credential management.
Knowledge check
Ravi's ETL pipeline at DataPulse Analytics needs to connect to an Azure SQL Database to ingest customer records. The connection string contains a password. Where should Ravi store this password?
Dr. Sarah Okafor wants Athena Group's Databricks workspace to access ADLS Gen2 storage with the LEAST management overhead. No manual secret rotation, no Key Vault entries for storage access. Which authentication method should she configure?
Next up: Data Discovery & Attribute-Based Access — descriptions, tags, and ABAC policies.