Databricks Community

jakubk · 06-20-2024

I'm using unity catalogI've changed the schema of my table by overwriting it with a newer file df.write \ .format('delta') \ .partitionBy(partitionColumn) \ .mode("overwrite") \ .option("overwriteSchema", "true") \ ...

jakubk · 03-21-2023

I'm looking to migrate onto unity catalog but a number of my data ingestion notebooks throw a securityexception/whitelist errors for numerous spark. functionsIs there some configuration setting I need to enable to whitelist the spark.* methods/functi...

jakubk · 03-01-2023

I'm on unity catalogI'm trying to do a dbt run on a project that works locallybut the databricks dbt workflow task seems to be ignoring the project.yml settings for schemas and catalogs, as well as that defined in the config block of individual model...

jakubk · 09-07-2022

I have some python code which takes parquet files from an adlsv2 location and merges it into delta tables (run as a workflow job on a schedule)I have a try catch wrapper around this so that any files that fail get moved into a failed folder using dbu...

jakubk · 08-07-2022

I'm using azure databricksI have a custom table valued function which takes a URL as a parameter and outputs a single row table with certain elements from the URL extracted/labelled(i get search activity URLs and when in a specific format I can retri...

jakubk · 10-10-2024

@njcheng @Brammer88 I went back to have another look - i had the same experience where i couldn't see the UC tables in any tool that used the oauth connection type (back in early sept) - but tried it again just now and i can see UC tables again in ac...

jakubk · 06-27-2024

I can't help with the hive metastore only in access - are you sure you're using a databricks endpoint that has unity catalog support? I could see my UC in both access and exceldsnless in excelFrom ODBC > advanced options > paste in your connection st...

jakubk · 06-24-2024

Thanks Daniel, that workedi also had a workaround going with an alter table set tblproperties (optimizewrite = true) which I have set as a default anyway. But just touching the table triggered a metadata refreshi prefer your command tho as that is it...

jakubk · 06-20-2024

I have set it up in Excel where users can pull data from databricks directlyUnfortunately it looks like access doesn't easily support DSNless connections via a URL so you're gonna have to set up a user or system odbc DSN using the simba spark driverh...

jakubk · 04-05-2023

Ok thanks, I suspected so but was hoping there was a way to configure this at the databricks workflow job level for the dbt task

Databricks Community

User Stats

User Activity

unity catalog information schema columns metadata out of sync with table - cant refresh

Unity Catalog - spark.* functions throwing Py4JSecurityException - org.apache.spark.sql.internal.CatalogImpl.currentCatalog() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl

dbt workflow job limitations - naming the target? where do docs go?

spark.read.parquet() - how to check for file lock before reading? (azure)

databricks spark sql Custom table valued function + struct really slow (minutes for a single row)

Re: How do I setup a connection to get data from unity catalog into MS Access?

Re: How do I setup a connection to get data from unity catalog into MS Access?

Re: unity catalog information schema columns metadata out of sync with table - cant refresh

Re: How do I setup a connection to get data from unity catalog into MS Access?

Re: dbt workflow job limitations - naming the target? where do docs go?