Share via

Serverless environment version 2

This article outlines the system environment information for serverless environment version 2. To ensure compatibility for the application, serverless workloads use a versioned API, known as the environment version, which remains compatible with newer server versions.

You can select the environment version using the Environment side panel in your serverless notebooks. See Select an environment version.

New features and improvements

The following new features and improvements are available in serverless environment 2.

Dashboards, alerts, and queries are supported as workspace files

May 20, 2025

Dashboards, alerts, and queries are now supported as workspace files, which means you can programmatically interact with these Databricks objects like any other file, from anywhere the workspace filesystem is available. See What are workspace files? and Programmatically interact with workspace files.

Web terminal enabled on serverless compute

April 3, 2025

The web terminal is now enabled on serverless environment version 2. For more information on how to use the web terminal, see Run shell commands in Azure Databricks web terminal.

The VARIANT data type can no longer be used with operations that require comparisons

February 5, 2025

You cannot use the following clauses or operators in queries that include a VARIANT data type:

  • DISTINCT
  • INTERSECT
  • EXCEPT
  • UNION
  • DISTRIBUTE BY

Additionally, you cannot use these DataFrame functions:

  • df.dropDuplicates()
  • df.repartition()

These operations perform comparisons, and comparisons that use the VARIANT data type produce undefined results and are not supported in Databricks. If you use the VARIANT type in your Azure Databricks workloads or tables, Databricks recommends the following changes:

  • Update queries or expressions to explicitly cast VARIANT values to non-VARIANT data types.
  • If you have fields that must be used with any of the above operations, extract those fields from the VARIANT data type and store them using non-VARIANT data types.

See Query variant data.

Notebooks are supported as workspace files

January 23, 2025

Notebooks are supported as workspace files in serverless environment 2. You can programmatically write, read, and delete notebooks just as you would any other file. This allows for programmatic interaction with notebooks from anywhere the workspace filesystem is available. For more information, see Programmatically create, update, and delete files and directories.

Task progress bar added to serverless compute

December 16, 2024

A new task progress bar has been added to notebook cells running on serverless compute environment version 2. This progress bar indicates the execution progress of the cell's Spark code.

Serverless progress bar

System environment

  • Operating System: Ubuntu 22.04.4 LTS
  • Python: 3.11.10
  • Databricks Connect: 15.4.5

Installed Python libraries

Library Version Library Version Library Version
asttokens 2.0.5 astunparse 1.6.3 autocommand 2.2.2
azure-core 1.31.0 azure-storage-blob 12.19.1 azure-storage-file-datalake 12.14.0
backports.tarfile 1.2.0 black 23.3.0 blinker 1.4
boto3 1.34.39 botocore 1.34.39 cachetools 5.5.0
certifi 2023.7.22 cffi 1.15.1 chardet 4.0.0
charset-normalizer 2.0.4 click 8.0.4 cloudpickle 3.0.0
comm 0.1.2 contourpy 1.0.5 cryptography 41.0.3
cycler 0.11.0 Cython 0.29.32 databricks-connect 15.4.5
databricks-sdk 0.36.0 dbus-python 1.2.18 debugpy 1.6.7
decorator 5.1.1 dill 0.3.6 distlib 0.3.9
entrypoints 0.4 executing 0.8.3 facets-overview 1.1.1
filelock 3.13.4 fonttools 4.25.0 gitdb 4.0.11
GitPython 3.1.43 google-api-core 2.18.0 google-auth 2.35.0
google-cloud-core 2.4.1 google-cloud-storage 2.18.2 google-crc32c 1.6.0
google-resumable-media 2.7.2 googleapis-common-protos 1.65.0 grpcio 1.67.0
grpcio-status 1.67.0 httplib2 0.20.2 idna 3.4
importlib-metadata 6.0.0 importlib_resources 6.4.0 inflect 7.3.1
ipyflow-core 0.0.201 ipykernel 6.28.0 ipython 8.25.0
ipython-genutils 0.2.0 ipywidgets 7.7.2 isodate 0.7.2
jaraco.collections 5.1.0 jaraco.context 5.3.0 jaraco.functools 4.0.1
jaraco.text 3.12.1 jedi 0.18.1 jeepney 0.7.1
jmespath 0.10.0 joblib 1.2.0 jupyter_client 7.4.9
jupyter_core 5.3.0 keyring 23.5.0 kiwisolver 1.4.4
launchpadlib 1.10.16 lazr.restfulclient 0.14.4 lazr.uri 1.0.6
matplotlib 3.7.2 matplotlib-inline 0.1.6 mlflow-skinny 2.11.4
more-itertools 8.10.0 mypy-extensions 0.4.3 nest-asyncio 1.5.6
numpy 1.23.5 oauthlib 3.2.0 packaging 23.2
pandas 1.5.3 parso 0.8.3 pathspec 0.10.3
patsy 0.5.3 pexpect 4.8.0 pillow 10.3.0
pip 24.2 platformdirs 3.10.0 plotly 5.9.0
prompt_toolkit 3.0.48 proto-plus 1.25.0 protobuf 5.28.3
psutil 5.9.0 psycopg2 2.9.3 ptyprocess 0.7.0
pure-eval 0.2.2 py4j 0.10.9.7 pyarrow 14.0.1
pyasn1 0.4.8 pyasn1-modules 0.2.8 pyccolo 0.0.65
pycparser 2.21 pydantic 1.10.6 Pygments 2.15.1
PyGObject 3.42.1 PyJWT 2.3.0 pyodbc 4.0.39
pyparsing 3.0.9 python-dateutil 2.8.2 python-lsp-jsonrpc 1.1.2
pytz 2022.7 PyYAML 6.0 pyzmq 25.1.2
requests 2.31.0 rsa 4.9 s3transfer 0.10.3
scikit-learn 1.3.0 scipy 1.11.1 seaborn 0.12.2
SecretStorage 3.3.1 setuptools 75.1.0 six 1.16.0
smmap 5.0.1 sqlparse 0.5.1 ssh-import-id 5.11
stack-data 0.2.0 statsmodels 0.14.0 tenacity 8.2.2
threadpoolctl 2.2.0 tokenize-rt 4.2.1 tomli 2.0.1
tornado 6.3.2 traitlets 5.13.0 typeguard 4.3.0
typing_extensions 4.10.0 tzdata 2022.1 ujson 5.4.0
unattended-upgrades 0.1 urllib3 1.26.16 virtualenv 20.26.6
wadllib 1.3.6 wcwidth 0.2.5 wheel 0.38.4
zipp 3.11.0 zstandard 0.23.0