Configure private connectivity to resources in your VNet

This page explains how to use the Azure Databricks account console to configure Private Link connections from serverless compute to resources in your virtual network (VNet) through an Azure load balancer.

Configuring private connectivity for serverless compute provides:

• A dedicated and private connection: Your private endpoint is exclusively tied to your Azure Databricks account, ensuring that access to your VNet resources is restricted to authorized workspaces only. This creates a secure, dedicated communication channel.
• Enhanced data exfiltration mitigation: While Azure Databricks Serverless with Unity Catalog offers built-in data exfiltration protection, Private Link provides an additional layer of network defense. By placing your VNet resources in a private subnet and controlling access through dedicated private endpoints, you significantly reduce the risk of unauthorized data movement outside your controlled network environment.

Requirements

• The workspace is on the Enterprise plan.
• You are the account admin of your Azure Databricks account.
• You have at least one active serverless workspace deployed in a region with private connectivity enabled. For supported regions, see Serverless availability
• Your load balancer has a virtual network and subnet, and your resource is located in this subnet.

Step 1: Create an Azure load balancer

Create an Azure Load Balancer that serves as the frontend for your VNet resources. This load balancer is linked to your Private Link service.

To create a load balancer, follow the instructions in the Quickstart: Create an internal load balancer to load balance VMs using the Azure portal. Complete the following:

1. Create a load balancer resource.
2. Add a frontend IP configuration: This is the entry point for your Private Link service.
3. Add a backend pool: This pool contains the IP addresses of your VNet resources.
4. Create a Health Probe: Configure a health probe to monitor the availability of your backend resources.
5. Add Load Balancing Rules: Define rules to distribute incoming traffic to your backend pool.

Step 2: Create a Private Link service

You must create a Private Link service to securely expose your load balancer to your private endpoint. Ensure the Private Link service is created in the same region as your load balancer.

For instructions, refer to the Azure documentation: Create a Private Link service using the Azure portal.

Step 3: Create or use an existing Network Connectivity Configuration (NCC) object

The NCC object in Azure Databricks defines the private connectivity settings for your workspaces. Skip this step if an NCC already exists. To create an NCC object:

1. As an account admin, go to the account console.
2. In the sidebar, click Cloud Resources.
3. Click Network.
4. Click Network Connectivity Configuration.
5. Click Add Network Connectivity Configuration.
6. Enter a name for the NCC.
7. Choose the region. This must match your workspace region.
8. Click Add.

Step 4: Create a private endpoint

This step links your Private Link service to your Azure Databricks NCC. To create a private endpoint:

1. In the account console, click Cloud resources.
2. Click Network Connectivity Configurations.
3. Select the NCC object you created in Step 3.
4. In the Private endpoint rules tab, click Add private endpoint rule.
5. In the Azure resource ID field, paste the full resource ID of your Private Link service. Find this ID in the Azure portal on the Overview page of your Private Link service. Example ID: /subscriptions/\<subscription-id\>/resourceGroups/\<resource-group-name\>/providers/Microsoft.Network/privateLinkServices/\<private-link-service-name\>.
6. In the Domain names field, add the custom ___domain names that your VNet resources use. These ___domain names should map to the IP configurations in your load balancer's backend pool.
7. Click Add.
8. Confirm that the Status column for your newly added private endpoint rule is PENDING.

Step 5: Accept the private endpoint on your resource

After creating the private endpoint rule in Databricks, you must approve the connection request in the Azure portal. To approve the connection:

1. Navigate to the Private Link center from the Azure portal.
2. Select Private Link services.
3. Find and select the Private Link service associated with your load balancer.
4. In the left sidebar under Settings, select Private endpoint connections.
5. Select the pending private endpoint.
6. Click Approve to accept the connection.
7. When prompted, select Yes.
8. After approval, the connection state changes to Approved.

It can take ten minutes for the connection to fully establish.

Step 6: Confirm private endpoint status

Verify the private endpoint connection is successfully established from the Azure Databricks side. To confirm the connection:

1. Refresh the Network Connectivity Configuration page in the Azure Databricks account console.
2. On the Private endpoint rules tab, confirm that the Status column for your new private endpoint is ESTABLISHED.

Step 7: Attach the NCC to one or more workspaces

This step associates your configured private connectivity with your Azure Databricks workspaces. Skip this step if your workspace is already attached to the desired NCC. To attach the NCC to a workspace:

1. Navigate to Workspaces in the left-hand navigation.
2. Select an existing workspace.
3. Select Update Workspace.
4. Under Network Connectivity Configuration, select the dropdown and choose the NCC you’ve created.
5. Repeat for all workspaces you’d like this NCC to apply to.

What's next

• Configure private connectivity to Azure resources: Use Private Link to establish secure and isolated access to Azure services from your virtual network, bypassing the public internet. See Configure private connectivity to Azure resources.
• Manage private endpoint rules: Control network traffic to and from your Azure private endpoints by defining specific rules that permit or deny connections. See Manage private endpoint rules.
• Configure a firewall for serverless compute access: Implement a firewall to restrict and secure inbound and outbound network connections for your serverless compute environments. See Configure a firewall for serverless compute access.
• Understand data transfer and connectivity costs: Data transfer and connectivity refer to moving data into and out of Azure Databricks serverless environments. Networking charges for serverless products only apply to customers using Azure Databricks serverless compute. See Understand Databricks serverless networking costs.