Databricks

dvmentalmadess · 02-15-2024

We run `OPTIMIZE` on our tables every 24 hours as follows:spark.sql(f'OPTIMIZE {catalog_name}.{schema_name}.`{table_name}`;') This morning one of our hourly jobs started failing on the call to `OPTIMIZE` with the error:org.apache.spark.SparkException...

dvmentalmadess · 11-28-2023

We create storage credentials using Terraform. I don't see any way to specify a given External ID (DBR Account ID) when creating the credentials via Terraform or in the web UI console. However, today when I tried creating a new set of credentials usi...

dvmentalmadess · 11-28-2023

I'm getting the following error: module.consumer_stage_catalog.databricks_external_location.catalog: Creating... ╷ │ Error: cannot create external location: AWS IAM role does not have READ permissions on url s3://[bucket name]/catalogs. Please conta...

dvmentalmadess · 09-18-2023

I am trying to setup s3 as a structured streaming source. The bucket receives ~17K files/day and the original load to the bucket was ~54K files. The bucket was first loaded 3 months ago and we haven't started reading from it since. So let's say there...

dvmentalmadess · 08-01-2023

We use a private PyPI repo (AWS CodeArtifact) to publish custom python libraries. We make the private repo available to DBR 12.2 clusters using an init-script as prescribed here in the Databricks KB. When we tried to upgrade to 13.2 this stopped wor...

dvmentalmadess

I'm coming back to provide an updated solution that doesn't rely on the implementation detail of the user name (e.g., libraries) - which is not considered a contract and could potentially change and break in the future.The key is to use the --global ...

dvmentalmadess · 02-15-2024

The job that was triggering this was an hourly job and the OPTIMIZE operation finally succeeded on the third attempt. I've submitted a ticket to support so hopefully this gets logged as a bug, but at the very least it appears to be a transient issue ...

dvmentalmadess · 01-17-2024

The problem is your host should be the workspace url, not the account url. Also as mentioned before the credentials should have account admin privilege if you want to have it return a known external id.

dvmentalmadess · 01-17-2024

Configure your provider with credentials that have been granted the account admin role. This solution means every storage credential created by this provider will have the Databricks Account ID as the external id value.If that doesn't work for you th...

dvmentalmadess · 01-04-2024

I was able to get some help from Databricks support and finally confirmed the cause of the behavior: IAM policies for storage credentials now require an external ID (Nov 30, 2023). We have been using external ID since April '23, but this change broke...

Databricks

User Stats

User Activity

OPTIMIZE: Exception thrown in awaitResult: / by zero

Terraform databricks_storage_credential has wrong External ID

Get exceptionTraceId details

Structured Streaming of S3 source

Private PyPI repos on DBR 13+

Re: Private PyPI repos on DBR 13+

Re: OPTIMIZE: Exception thrown in awaitResult: / by zero

Re: Terraform databricks_storage_credential has wrong External ID

Re: Terraform databricks_storage_credential has wrong External ID

Re: Terraform databricks_storage_credential has wrong External ID