Databricks Community

eimis_pacheco · 09-29-2023

Hello Community,Today I was in Tips and Tricks - Optimizations webinar and I started being confused, they said:"Don't partition tables <1TB in size and plan carefully when partitioning• Partitions should be >=1GB" Now my confusion is if this recommen...

eimis_pacheco · 09-28-2023

Hi community!I was in a Databricks webinar and one of the participants said "Delta Live Tables seems to have some limitations when using with Unity Catalog. Is the idea to get parity with Hive?" and someone answered "DLT + Unity Catalog combination h...

eimis_pacheco · 09-19-2023

Hi Community I have a doubt. The bronze layer always causes confusion for me. Someone mentioned, "File Format: Store data in Delta Lake format to leverage its performance, ACID transactions, and schema evolution capabilities" for bronze layers.Then, ...

eimis_pacheco · 08-25-2023

Hi dear community,When I used to work in the Hadoop ecosystem with HDS the landing zone was our raw layer, and we used to use AVRO format for the serialization of this raw data (for the schema evolution feature), only assigning names to columns but n...

eimis_pacheco · 07-15-2023

Hi community,I suddenly found myself confused, and this might sound like an obvious answer for some but not for me at least in this moment.I am not getting why companies use databricks SQL and Redshift at the same time?I mean, with databricks platfor...

eimis_pacheco · 10-17-2023

Hi @Retired_mod ,Sorry for asking again but this recommendation of "don't partition tables <1TB" but this recommendation applies when this is written on disk or when the job is actually running in memory too? (sorry if the answer is obvious)Because i...

eimis_pacheco · 10-17-2023

Hi @Retired_mod ,Thank you for your answer. Wondering then why Databricks recommends the usage of both solutions (DLT and Unity Catalog) even with the existence of this imcompatibilities?Best Regards

eimis_pacheco · 09-22-2023

Thank you very much for your answers and insights @Retired_modRegards!

eimis_pacheco · 09-22-2023

Thank you very much for your answer @swethaNandan.Regards!

eimis_pacheco · 09-20-2023

Hi @Retired_mod Just a last question, what would happen if someone decided to change the name of one columns in the source system? For example, if someone renames the column "ID" for "cust_id" in the customer table? how Delta Lake format now will kno...

Databricks Community

User Stats

User Activity

Confused with databricks Tips and Tricks - Optimizations regarding partitining

What are the Delta Live Tables limitations with relation to Unity Catalog?

Is it not needed to preserve the data in its original format anymore with the usage of medallion?

What are the best practices in bronze layer regarding the column data types?

Why companies use databricks SQL and Redshift at the same time?

Re: Confused with databricks Tips and Tricks - Optimizations regarding partitining

Re: What are the Delta Live Tables limitations with relation to Unity Catalog?

Re: Is it not needed to preserve the data in its original format anymore with the usage of medallion

Re: What are the best practices in bronze layer regarding the column data types?

Re: Is it not needed to preserve the data in its original format anymore with the usage of medallion