Databricks Community

jakubk

Hi All,I'm after some guidance on how to identify massive (100000%) spikes in bandwidth usage (and related costs) in the azure databricks provisioned/managed resource group storage account & stop themThese blips are adding 30-50% to our monthly costs...

jakubk · 06-20-2024

I'm using unity catalogI've changed the schema of my table by overwriting it with a newer file df.write \ .format('delta') \ .partitionBy(partitionColumn) \ .mode("overwrite") \ .option("overwriteSchema", "true") \ ...

jakubk · 03-21-2023

I'm looking to migrate onto unity catalog but a number of my data ingestion notebooks throw a securityexception/whitelist errors for numerous spark. functionsIs there some configuration setting I need to enable to whitelist the spark.* methods/functi...

jakubk · 03-01-2023

I'm on unity catalogI'm trying to do a dbt run on a project that works locallybut the databricks dbt workflow task seems to be ignoring the project.yml settings for schemas and catalogs, as well as that defined in the config block of individual model...

jakubk · 09-07-2022

I have some python code which takes parquet files from an adlsv2 location and merges it into delta tables (run as a workflow job on a schedule)I have a try catch wrapper around this so that any files that fail get moved into a failed folder using dbu...

jakubk

@-werners- wrote:That is strange, I think something went wrong in the deployment of the Databricks environment.We have 3 databricks workspaces and do not see such behavior.Any resource deployed on a subscription owned by you is accessible by some ki...

jakubk

@-werners- No, it's the 'databricks workspace' environment that's automatically created when you provision a new databricks workspace in azure. The resource group and storage account are locked down to the databricks service acc only. everyone else i...

jakubk · 10-10-2024

@njcheng @Brammer88 I went back to have another look - i had the same experience where i couldn't see the UC tables in any tool that used the oauth connection type (back in early sept) - but tried it again just now and i can see UC tables again in ac...

jakubk · 06-27-2024

I can't help with the hive metastore only in access - are you sure you're using a databricks endpoint that has unity catalog support? I could see my UC in both access and exceldsnless in excelFrom ODBC > advanced options > paste in your connection st...

jakubk · 06-24-2024

Thanks Daniel, that workedi also had a workaround going with an alter table set tblproperties (optimizewrite = true) which I have set as a default anyway. But just touching the table triggered a metadata refreshi prefer your command tho as that is it...

Databricks Community

User Stats

User Activity

unknown geo redundancy storage events (& costs) in azure databricks resource group

unity catalog information schema columns metadata out of sync with table - cant refresh

Unity Catalog - spark.* functions throwing Py4JSecurityException - org.apache.spark.sql.internal.CatalogImpl.currentCatalog() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl

dbt workflow job limitations - naming the target? where do docs go?

spark.read.parquet() - how to check for file lock before reading? (azure)

Re: unknown geo redundancy storage events (& costs) in azure databricks resource group

Re: unknown geo redundancy storage events (& costs) in azure databricks resource group

Re: How do I setup a connection to get data from unity catalog into MS Access?

Re: How do I setup a connection to get data from unity catalog into MS Access?

Re: unity catalog information schema columns metadata out of sync with table - cant refresh