Databricks

Pat · 12-16-2022

Hi community,I don't know what is happening TBH. I have a use case where data is written to the location "dbfs:/mnt/...", don't ask me why it's mounted, it's just a side project. I do believe that data is stored in ADLS2.I've been trying to read the ...

Pat · 11-22-2022

How do you automate table creation and management of privilege / grants on securable objects (tables, views, etc.).I had idea to use terraform to manage above, but terraform doesn't work with managed tables - it allows to create table but then there ...

Pat · 09-23-2022

 and  situation?I am currently trying to came up with the way how to deploy Databricks with Terraform in multi-region, multi-tenant environment. I am not talking about simple cases like this (https://docs.databricks.com/data-governance/unity-cata...

Pat · 09-15-2022

I am not sure if I am missing something, but I just created External Table using External Location and I can still access both data through the table and directly access files from the storage: documentation:https://docs.databricks.com/data-governanc...

Pat · 09-15-2022

Error in SQL statement: AnalysisException: SHOW PARTITIONS is not supported in Unity Catalog;What happened with the SHOW PARTITIONS command?Is this information available anywhere in the documentation?If I go to Databricks SQL documentation there is n...

Pat · 12-18-2022

Hi @Aviral Bhardwaj ,I will try to re-produce this. I think that at least one of the files is corrupted, but I would expect different error in that case, not long running job that fails with `Unable to infer schema for Parquet. It must be specified ...

Pat · 12-16-2022

I am still not sure what happened, but I've re-run job on smaller dataset and seems to work, maybe corrupted data ?

Pat · 12-08-2022

I think you should be able to set it once, at the top of your notebook for example. It's spark session parameter, it will be used until you will restart the cluster.

Pat · 12-08-2022

HI @Alex Netelkos ,Please check the dynamic partitions parameter, more details here: https://docs.databricks.com/delta/selective-overwrite.html#dynamic-partition-overwritesthanks,Pat.

Pat · 12-08-2022

I'm very glad to be part of this community. I've learned a lot in the last few months while working on my own and others' challenges :). It's good to have a place where we can share knowledge, we are not left alone on the project!    

Databricks

User Stats

User Activity

Reading data from "dbfs:/mnt/"

Automating DDLs and Privilege management

Error: cannot create metastore data access: No metastore assigned for the current workspace.

Unity Catalog - External Table

SHOW PARTITIONS not supported

Re: Reading data from "dbfs:/mnt/"

Re: Reading data from "dbfs:/mnt/"

Re: What's the easiest way to upsert data into a table? (Azure ADLS Gen2)

Re: What's the easiest way to upsert data into a table? (Azure ADLS Gen2)

Re: Featured Member Interview - December 2022Patrycjusz Sienkiewicz - @Pat Sienkiewicz ⭐️ Community nickname: PatPronouns: He/HimJob Title: Data Enginee...