Hi @MRTN,
Errors reading external tables from serverless compute are almost always caused by how serverless handles cloud storage access compared to classic compute. Serverless compute runs in a Databricks-managed compute plane, so it cannot use DBFS mounts, instance profiles, or direct cloud credentials the way classic clusters can. Instead, it requires Unity Catalog external locations and storage credentials for all external storage access.
Here is a checklist to work through:
STEP 1: CONFIRM THE EXTERNAL TABLE IS REGISTERED IN UNITY CATALOG
Serverless compute only supports Unity Catalog. If your external table is registered in the legacy Hive metastore (hive_metastore), it will not be accessible from serverless. You can verify where the table lives by running:
DESCRIBE EXTENDED catalog_name.schema_name.table_name;
Look at the "Catalog" field in the output. If it shows "hive_metastore", you will need to migrate or re-create the table in a Unity Catalog catalog.
STEP 2: VERIFY THE STORAGE CREDENTIAL EXISTS AND IS VALID
A storage credential is required to authorize access to the cloud storage location where your external table data lives. Check existing storage credentials:
SHOW STORAGE CREDENTIALS;
If no credential covers the storage path of your external table, your workspace admin needs to create one. For AWS, this is typically an IAM role. For Azure, it would be a managed identity or service principal. For GCP, a service account.
Documentation: https://docs.databricks.com/en/connect/unity-catalog/storage-credentials.html
STEP 3: VERIFY THE EXTERNAL LOCATION IS CONFIGURED
An external location maps a cloud storage path to a storage credential. Without this mapping, serverless compute has no way to reach the data. Check your external locations:
SHOW EXTERNAL LOCATIONS;
Confirm there is an external location whose URL path is a prefix of your external table's data path. You can check the table's location with:
DESCRIBE DETAIL catalog_name.schema_name.table_name;
The "location" field shows the cloud storage path. That path must fall under a registered external location.
Documentation: https://docs.databricks.com/en/connect/unity-catalog/external-locations.html
STEP 4: CHECK PERMISSIONS
Even if the external location and storage credential exist, your user (or the service principal running the job) needs the right Unity Catalog privileges:
- USE CATALOG on the parent catalog
- USE SCHEMA on the parent schema
- SELECT on the table itself
The storage credential and external location also need appropriate grants. You can check with:
SHOW GRANTS ON EXTERNAL LOCATION location_name;
SHOW GRANTS ON TABLE catalog_name.schema_name.table_name;
STEP 5: RULE OUT DBFS MOUNTS
If the table was originally created using a DBFS mount path (e.g., /mnt/my-storage/...), this will not work on serverless. DBFS mounts with AWS instance profiles are explicitly not supported on serverless compute. The table needs to be re-created using a direct cloud storage URI (e.g., s3://bucket-name/path/ or abfss://container@account.dfs.core.windows.net/path/) that is covered by an external location in Unity Catalog.
STEP 6: NETWORK CONNECTIVITY (IF APPLICABLE)
If your workspace uses private endpoints or firewall rules on the storage account, make sure the serverless compute plane has network access. Serverless compute resources run in a Databricks-managed VPC/VNet, so firewall rules that allow only your workspace VPC may block serverless. Check the documentation on serverless networking for your cloud provider:
- AWS: https://docs.databricks.com/en/compute/serverless/network-connectivity.html
- Azure: https://learn.microsoft.com/en-us/azure/databricks/compute/serverless/network-connectivity
COMMON ERROR PATTERNS
- "Access Denied" or "403 Forbidden": Usually means the storage credential's IAM role or managed identity does not have permission to the S3 bucket/ADLS container, or the trust policy does not allow the Databricks serverless account to assume it.
- "External location not found": The table's data path is not covered by any registered external location.
- "Cannot access data": The table may be in hive_metastore instead of a Unity Catalog catalog.
If you can share the exact error message you are seeing, I can narrow down the specific cause further.
* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.
If this answer resolves your question, could you mark it as "Accept as Solution"? That helps other users quickly find the correct fix.
