Showing results for 
Search instead for 
Did you mean: 
Valued Contributor
since ‎04-11-2022
4 weeks ago

User Stats

  • 29 Posts
  • 5 Solutions
  • 3 Kudos given
  • 10 Kudos received

User Activity

We run `OPTIMIZE` on our tables every 24 hours as follows:spark.sql(f'OPTIMIZE {catalog_name}.{schema_name}.`{table_name}`;') This morning one of our hourly jobs started failing on the call to `OPTIMIZE` with the error:org.apache.spark.SparkException...
We create storage credentials using Terraform. I don't see any way to specify a given External ID (DBR Account ID) when creating the credentials via Terraform or in the web UI console. However, today when I tried creating a new set of credentials usi...
I'm getting the following error:  module.consumer_stage_catalog.databricks_external_location.catalog: Creating... ╷ │ Error: cannot create external location: AWS IAM role does not have READ permissions on url s3://[bucket name]/catalogs. Please conta...
I am trying to setup s3 as a structured streaming source. The bucket receives ~17K files/day and the original load to the bucket was ~54K files. The bucket was first loaded 3 months ago and we haven't started reading from it since. So let's say there...
We use a private PyPI repo (AWS CodeArtifact) to publish custom python libraries. We make the private repo available to DBR 12.2 clusters using an init-script as prescribed here in the Databricks KB.  When we tried to upgrade to 13.2 this stopped wor...
Kudos given to