Create a storage credential for connecting to Azure Data Lake Storage

This article describes how to create storage credentials in Unity Catalog to connect to Azure Data Lake Storage.

A storage credential contains a long-term cloud credential that provides access to cloud storage. You reference storage credentials, along with the cloud storage path, when you create external locations in Unity Catalog to govern access to external storage.

For more information about storage credentials and external locations, see Connect to cloud object storage using Unity Catalog.

For information about other cloud storage options supported by Unity Catalog, see Cloud storage options supported by Unity Catalog.

Create a storage credential that accesses Azure Data Lake Storage

You can use either an Azure managed identity or a service principal as the identity that authorizes access to your storage container. Managed identities are strongly recommended. They have the benefit of allowing Unity Catalog to access storage accounts protected by network rules, which isn’t possible using service principals, and they remove the need to manage and rotate secrets. If you want to use a service principal, see Create Unity Catalog managed storage using a service principal (legacy).

Requirements

In Azure Databricks:

• Azure Databricks workspace enabled for Unity Catalog.
• CREATE STORAGE CREDENTIAL privilege on the Unity Catalog metastore attached to the workspace. Account admins and metastore admins have this privilege by default.

In your Azure tenant:

• An Azure Data Lake Storage storage container. To avoid egress charges, this should be in the same region as the workspace you want to access the data from.

  The Azure Data Lake Storage storage account must have a hierarchical namespace.

• Contributor or Owner of an Azure resource group.

• Owner or a user with the User Access Administrator Azure RBAC role on the storage account.

Create the storage credential

1. In the Azure portal, create an Azure Databricks access connector and assign it permissions on the storage container that you would like to access, using the instructions in Configure a managed identity for Unity Catalog.

   An Azure Databricks access connector is a first-party Azure resource that lets you connect managed identities to an Azure Databricks account. To add the storage credential, you must have the Contributor role or higher on the access connector resource in Azure.

   Make a note of the access connector's resource ID.

2. Log in to your Unity Catalog-enabled Azure Databricks workspace as a user who has the CREATE STORAGE CREDENTIAL privilege.

   The metastore admin and account admin roles both include this privilege.

3. Click Catalog.

4. On the Quick access page, click the External data > button, go to the Credentials tab, and select Create credential.

5. Select Storage credential.

6. Select a Credential Type of Azure Managed Identity.

7. Enter a name for the credential, and enter the access connector's resource ID in the format:

   /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Databricks/accessConnectors/<connector-name>
   

8. (Optional) If you created the access connector using a user-assigned managed identity, enter the resource ID of the managed identity in the User-assigned managed identity ID field, in the format:

   /subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<managed-identity-name>
   

9. (Optional) If you want users to have read-only access to the external locations that use this storage credential, select Read only. For more information, see Mark a storage credential as read-only.

10. Click Create.

11. (Optional) Bind the storage credential to specific workspaces.

    By default, any privileged user can use the storage credential on any workspace attached to the metastore. If you want to allow access only from specific workspaces, go to the Workspaces tab and assign workspaces. See (Optional) Assign a storage credential to specific workspaces.

12. Create an external ___location that references this storage credential.

(Optional) Assign a storage credential to specific workspaces

By default, a storage credential is accessible from all of the workspaces in the metastore. This means that if a user has been granted a privilege (such as CREATE EXTERNAL LOCATION) on that storage credential, they can exercise that privilege from any workspace attached to the metastore. If you use workspaces to isolate user data access, you may want to allow access to a storage credential only from specific workspaces. This feature is known as workspace binding or storage credential isolation.

A typical use case for binding a storage credential to specific workspaces is the scenario in which a cloud admin configures a storage credential using a production cloud account credential, and you want to ensure that Azure Databricks users use this credential to create external locations only in the production workspace.

For more information about workspace binding, see (Optional) Assign an external ___location to specific workspaces and Limit catalog access to specific workspaces.

Bind a storage credential to one or more workspaces

To assign a storage credential to specific workspaces, you can use Catalog Explorer or the Databricks CLI.

Permissions required: Metastore admin, storage credential owner, or MANAGE on the storage credential.

Catalog Explorer

1. Log in to a workspace that is linked to the metastore.

2. In the sidebar, click Catalog.

3. On the Quick access page, click the External data > button and go to the Credentials tab.

4. Select the storage credential and go to the Workspaces tab.

5. On the Workspaces tab, clear the All workspaces have access checkbox.

   If your storage credential is already bound to one or more workspaces, this checkbox is already cleared.

6. Click Assign to workspaces and enter or find the workspaces you want to assign.

To revoke access, go to the Workspaces tab, select the workspace, and click Revoke. To allow access from all workspaces, select the All workspaces have access checkbox.

CLI

There are two Databricks CLI command groups and two steps required to assign a storage credential to a workspace.

In the following examples, replace <profile-name> with the name of your Azure Databricks authentication configuration profile. It should include the value of a personal access token, in addition to the workspace instance name and workspace ID of the workspace where you generated the personal access token. See Azure Databricks personal access token authentication.

1. Use the storage-credentials command group's update command to set the storage credential's isolation mode to ISOLATED:

   databricks storage-credentials update <my-storage-credential> \
   --isolation-mode ISOLATED \
   --profile <profile-name>
   

   The default isolation-mode is OPEN to all workspaces attached to the metastore.

2. Use the workspace-bindings command group's update-bindings command to assign the workspaces to the storage credential:

   databricks workspace-bindings update-bindings storage-credential <my-storage-credential> \
   --json '{
     "add": [{"workspace_id": <workspace-id>}...],
     "remove": [{"workspace_id": <workspace-id>}...]
   }' --profile <profile-name>
   

   Use the "add" and "remove" properties to add or remove workspace bindings.

   Note

   Read-only binding (BINDING_TYPE_READ_ONLY) is not available for storage credentials. Therefore there is no reason to set binding_type for the storage credentials binding.

To list all workspace assignments for a storage credential, use the workspace-bindings command group's get-bindings command:

databricks workspace-bindings get-bindings storage-credential <my-storage-credential> \
--profile <profile-name>

Unbind a storage credential from a workspace

Instructions for revoking workspace access to a storage credential using Catalog Explorer or the workspace-bindings CLI command group are included in Bind a storage credential to one or more workspaces.

Next steps

• View, update, delete, and grant other users permission to use a storage credential. See Manage storage credentials.

• Define external locations using a storage credential. See Create an external ___location to connect cloud storage to Azure Databricks.