Manage external locations

This article describes how to list, view, update, grant permissions on, enable file events for, and delete external locations.

Describe an external ___location

To see the properties of an external ___location, including permissions and workspace access, you can use Catalog Explorer or a SQL command.

Catalog Explorer

1. In the sidebar, click Catalog.
2. On the Quick access page, click the External data > button to go to the External Locations tab.
3. Click the name of an external ___location to view its properties.

SQL

Run the following command in a notebook or the Databricks SQL editor. Replace <___location-name> with the name of the external ___location.

DESCRIBE EXTERNAL LOCATION <___location-name>;

Show grants on an external ___location

To show grants on an external ___location, use a command like the following. You can optionally filter the results to show only the grants for the specified principal.

SHOW GRANTS [<principal>] ON EXTERNAL LOCATION <___location-name>;

Replace the placeholder values:

• <___location-name>: The name of the external ___location that authorizes reading from and writing to the storage container or bucket in your cloud tenant.
• <principal>: The email address of an account-level user or the name of an account-level group.

Grant permissions on an external ___location

This section describes how to grant and revoke permissions on an external ___location using Catalog Explorer and SQL commands in a notebook or SQL query. For information about using the Databricks CLI or Terraform instead, see the Databricks Terraform documentation and What is the Databricks CLI?.

You can grant the following permissions on an external ___location:

• CREATE EXTERNAL TABLE
• CREATE EXTERNAL VOLUME
• CREATE MANAGED STORAGE

Permissions required: The CREATE EXTERNAL LOCATION privilege on both the metastore and the storage credential referenced in the external ___location or the MANAGE privilege on the external ___location. Metastore admins have CREATE EXTERNAL LOCATION on the metastore by default.

To grant permission to use an external ___location:

Catalog Explorer

1. In the sidebar, click Catalog.
2. On the Quick access page, click the External data > button to go to the External Locations tab.
3. Click the name of an external ___location to open its properties.
4. Click Permissions.
5. To grant permission to users or groups, select each identity, then click Grant.
6. To revoke permissions from users or groups, select each identity, then click Revoke.

SQL

Run the following SQL command in a notebook or SQL query editor. This example grants the ability to create an external table that references the external ___location:

GRANT CREATE EXTERNAL TABLE ON EXTERNAL LOCATION <___location-name> TO <principal>;

Replace the placeholder values:

• <___location-name>: The name of the external ___location that authorizes reading from and writing to the storage container or bucket in your cloud tenant.
• <principal>: The email address of an account-level user or the name of an account-level group.

Change the owner of an external ___location

An external ___location's creator is its initial owner. To change the owner to a different account-level user or group, run the following command in a notebook or the Databricks SQL editor or use Catalog Explorer.

Permissions required: External ___location owner or a user with the MANAGE privilege.

Replace the placeholder values:

• <___location-name>: The name of the credential.
• <principal>: The email address of an account-level user or the name of an account-level group.

ALTER EXTERNAL LOCATION <___location-name> OWNER TO <principal>

Mark an external ___location as read-only

If you want users to have read-only access to an external ___location, you can use Catalog Explorer to mark the external ___location as read-only.

Making external locations read-only:

• Prevents users from writing to files in those external locations, regardless of any write permissions granted by the Azure managed identity that underlies the storage credential, and regardless of the Unity Catalog permissions granted on that external ___location.
• Prevents users from creating managed tables or volumes in those external locations.
• Enables the system to validate the external ___location properly at creation time.

You can mark external locations as read-only when you create them.

You can also use Catalog Explorer to change read-only status after creating an external ___location:

1. In the sidebar, click Catalog.
2. On the Quick access page, click the External data > button to go to the External Locations tab.
3. Select the external ___location, click the menu next to the Test connection button, and select Edit.
4. On the edit dialog, click Advanced Options and select the Limit to read-only use option.
5. Click Update.

Configure an encryption algorithm on an external ___location (AWS S3 only)

AWS supports server-side encryption (SSE) with Amazon S3 managed keys (SSE-S3) or AWS KMS keys (SSE-KMS) for protecting data in S3. If your S3 bucket requires SSE encryption, you can configure an encryption algorithm in your external ___location to allow external tables and volumes in Unity Catalog to access data in your S3 bucket. SSE is not supported with external tables shared using Delta Sharing. For more information, see Configure encryption for S3 with KMS.

1. In the sidebar, click Catalog.

2. At the top of the Catalog pane, click the gear icon and select External Locations.

3. Select the external ___location. The external ___location must use an IAM role for a storage credential.

4. Click the kebab menu next to the Test connection button, and select Edit.

5. On the edit dialog, click Advanced Options.

6. Under Encryption Algorithm select SSE-SE or SSE-KMS depending on your encryption key.

   For SSE-KMS, under Encryption KMS key arn paste the ARN of the KMS key referenced by clients when accessing the S3 ___location.

7. Click Update.

(Recommended) Enable file events for an external ___location

If you want to ingest change notifications that are pushed by the cloud provider, enabling file events for the external ___location has the following advantages:

• Makes it easier to set up file notifications for Auto Loader. Specifically, it enables incremental file discovery with notification-like performance in Auto Loader by setting cloudFiles.useManagedFileEvents to true. See Configure Auto Loader streams in file notification mode.
• Improves the efficiency and capacity of file arrival triggers for jobs. See Trigger jobs when new files arrive.

Before you begin

If you want Azure Databricks to configure Azure Data Lake Storage Queues or SQS queues on your behalf, your external ___location must reference a storage credential that gives adequate permissions to do so. See the next step for instructions.

If you want to create your own Azure Data Lake storage queue, the identity represented by the storage credential must have the following permission on those Azure Data Lake storage queues:

• Storage Queue Data Contributor

If you want to create your own SQS queues, the identity represented by the storage credential must have the following permissions on those SQS queues:

• sqs:ReceiveMessage
• sqs:DeleteMessage
• sqs:PurgeQueue

Step 1: Confirm that Azure Databricks has access to file events in Azure Data Lake Storage or AWS S3

Before you can enable file events for the external ___location securable object, you must ensure that your Azure Data Lake Storage or AWS S3 account is configured to give Azure Databricks access to the file events that it emits. If you want Azure Databricks to configure file events in Azure Data Lake Storage or AWS S3 for you, you must also ensure that Azure Databricks has the proper access.

Assigning this access is a recommended step when you configure storage credentials.

For Azure Data Lake Storage containers

To confirm that the managed identity that gives access to the external ___location is configured properly:

1. Get the Azure managed identity ID.

   1. In the sidebar, click Catalog.

   2. On the Quick access page, click the External data > button to go to the External Locations tab.

   3. Select the external ___location.

   4. On the Overview tab, click the Credential name.

   5. On the storage credential Overview tab, copy the Connector Id or User Assigned Managed Identity Id.

      You will use this in the next step.

2. Log into your Azure Data Lake Storage account.

3. Go to Access Control (IAM) and click the Check access button.

4. In Managed identity, select either User-assigned managed identity or Access Connector for Azure Databricks, depending on the managed identity type.

5. Search for the managed identity that you copied in step 1.

6. Confirm that the managed identity has the following roles:

   • Storage Blob Data Contributor
   • EventGrid EventSubscription Contributor
   • Storage Queue Data Contributor: Required only if you want Azure Databricks to create a subscription and events in Azure Data Lake Storage for you. If you do not enable this role, you must create the Azure storage queue yourself).

7. If any of these roles is missing, go to the Eligible assignments tab and add them.

For more information about assigning these roles, see Step 3: Grant the managed identity access to file events and Step 4: Grant Azure Databricks access to configure file events on your behalf.

For S3 buckets

To verify that Databricks can configure and subscribe to your bucket's event notifications:

1. Get the IAM role.

   1. In the sidebar, click Catalog.

   2. On the Quick access page, click the External data > button to go to the External Locations tab.

   3. Select the external ___location.

   4. On the Overview tab, click the Credential name.

   5. On the storage credential Overview tab, copy the IAM role (ARN).

      You will use this in the next step.

2. Log into your AWS account.

3. Go to IAM and search for the role that you copied in step 1.

4. Under Permissions policies, find the IAM policy or policies associated with the IAM role and open it.

5. Open the policy or policies and confirm that there is one that includes the following properties.

   This policy allows your Azure Databricks account to update your bucket's event notification configuration, create an SNS topic, create an SQS queue, and subscribe the SQS queue to the SNS topic.

   Replace <BUCKET> with the name of your S3 bucket.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "ManagedFileEventsSetupStatement",
         "Effect": "Allow",
         "Action": [
           "s3:GetBucketNotification",
           "s3:PutBucketNotification",
           "sns:ListSubscriptionsByTopic",
           "sns:GetTopicAttributes",
           "sns:SetTopicAttributes",
           "sns:CreateTopic",
           "sns:TagResource",
           "sns:Publish",
           "sns:Subscribe",
           "sqs:CreateQueue",
           "sqs:DeleteMessage",
           "sqs:ReceiveMessage",
           "sqs:SendMessage",
           "sqs:GetQueueUrl",
           "sqs:GetQueueAttributes",
           "sqs:SetQueueAttributes",
           "sqs:TagQueue",
           "sqs:ChangeMessageVisibility",
           "sqs:PurgeQueue"
         ],
         "Resource": ["arn:aws:s3:::<BUCKET>", "arn:aws:sqs:*:*:csms-*", "arn:aws:sns:*:*:csms-*"]
       },
       {
         "Sid": "ManagedFileEventsListStatement",
         "Effect": "Allow",
         "Action": ["sqs:ListQueues", "sqs:ListQueueTags", "sns:ListTopics"],
         "Resource": ["arn:aws:sqs:*:*:csms-*", "arn:aws:sns:*:*:csms-*"]
       },
       {
         "Sid": "ManagedFileEventsTeardownStatement",
         "Effect": "Allow",
         "Action": ["sns:Unsubscribe", "sns:DeleteTopic", "sqs:DeleteQueue"],
         "Resource": ["arn:aws:sqs:*:*:csms-*", "arn:aws:sns:*:*:csms-*"]
       }
     ]
   }
   

   See also Step 1: Create an IAM role.

6. If this policy is missing, add it to the IAM role.

Step 2: Enable file events for the external ___location using Catalog Explorer

To enable file events:

1. In the sidebar, click Catalog.

2. On the Quick access page, click the External data > button to go to the External Locations tab.

3. Select the external ___location.

4. Click the kebab menu next to the Test connection button, and select Edit.

5. On the edit dialog, click Advanced Options.

6. Select Enable file events.

7. Select the File event type:

   Automatic: (Recommended) select this if you want Azure Databricks to set up subscriptions and events for you.

   Provided: Select this if you have already configured an Azure storage queue or SQS queue yourself.

8. If you selected the Provided file event type, enter the Queue URL of the existing storage queue:.

   • Azure storage queue: https://<storage account>.queue.core.windows.net/<queue>

   • AWS SQS queue: https://sqs.<region>.amazonaws.com/<account-ID>/<queue-name>

9. Click Update.

10. Wait a few seconds and click Test connection on the main external ___location edit page to confirm that file events have been enabled successfully.

File events limitations

File events on external locations have the following limitations:

• Event throughput is limited to 2000 files ingested per second.

• You cannot tag cloud resources using the resourceTags option. Instead, tag resources using the cloud console after the Auto Loader service creates your queue and subscription resources.

• You can't set up file events for the Unity Catalog metastore root storage ___location if no external ___location is defined for that storage ___location.

  You can't set up file events on storage locations that have no external ___location object defined in Unity Catalog. Some implementations of Unity Catalog include a metastore root storage ___location that is not associated with an external ___location. To determine if this is the case for your metastore root storage ___location:

  1. As an account admin, log in to the account console.

  2. Click Catalog.

  3. Click the metastore name.

  4. Go to the Configuration tab.

  5. If there is a ADLS Gen 2 path value, then your metastore root has no external ___location object defined for it.

     To update the metastore root storage to use an external ___location, click the Remove button. An external ___location will be created for you. For details, see Remove metastore-level storage.

• Azure Blob Storage is not supported.

Modify an external ___location

An external ___location's owner or a user with the MANAGE privilege can rename, change the URI, and change the storage credential of the external ___location.

To rename an external ___location, do the following:

Run the following command in a notebook or the Databricks SQL editor. Replace the placeholder values:

• <___location-name>: The name of the ___location.
• <new-___location-name>: A new name for the ___location.

ALTER EXTERNAL LOCATION <___location-name> RENAME TO <new-___location-name>;

To change the URI that an external ___location points to in your cloud tenant, do the following:

Run the following command in a notebook or the Databricks SQL editor. Replace the placeholder values:

• <___location-name>: The name of the external ___location.
• <url>: The new storage URL the ___location should authorize access to in your cloud tenant.

ALTER EXTERNAL LOCATION location_name SET URL '<url>' [FORCE];

The FORCE option changes the URL even if external tables depend upon the external ___location.

To change the storage credential that an external ___location uses, do the following:

Run the following command in a notebook or the Databricks SQL editor. Replace the placeholder values:

• <___location-name>: The name of the external ___location.
• <credential-name>: The name of the storage credential that grants access to the ___location's URL in your cloud tenant.

ALTER EXTERNAL LOCATION <___location-name> SET STORAGE CREDENTIAL <credential-name>;

Delete an external ___location

To delete (drop) an external ___location you must be its owner or have the MANAGE privilege on the external ___location. To delete an external ___location, do the following:

Run the following command in a notebook or the Databricks SQL editor. Items in brackets are optional. Replace <___location-name> with the name of the external ___location.

DROP EXTERNAL LOCATION [IF EXISTS] <___location-name>;