Databricks Community

Prashanth24 · 08-13-2024

Liquid Clustering is a combination of Partitioning and Zordering. As we know, partitioning will create folders based on the column values and stores the similar value. I believe Liquid Clustering will not create folders like partitioning, so how it w...

Prashanth24 · 08-13-2024

Under Databricks Compute and Worker nodes, we find different types of types as belowStandard_D4ds_v5 => 16 GB Memory, 4 CoresStandard_D8ds_v5 => 32 GB Memory, 8 CoresIn Databricks, each node will have one executor. I have questions below(1) How much ...

Prashanth24 · 08-11-2024

I need to ingest full load with 5 TB of data by applying business transformations and wants to process it in 2-3 hours. Any criteria needs to be considered while selecting min and max node workers for this full load processing.

Prashanth24 · 08-07-2024

I am trying to understand the difference between Liquid clustering and z-ordering. As per my understanding, both stores the clustered information into ZCubes which is of size 100 GB.Liquid Clustering maintains ZCube id in transaction log so when opti...

Prashanth24 · 08-06-2024

I am trying to connect Databricks Notebook using managed identity authentication type from Azure Data Factory. Below are the settings done. Error message is appended at the bottom of this message. With the same settings but with different authenticat...

Prashanth24 · 08-13-2024

Thanks for the information. So for this background processing, core and memory from each node will be allocated OR collectively from cluster will be allocated? Also how much core and memory might get allocated for this work per node or per cluster?

Prashanth24 · 08-11-2024

I gone through the document. Gave good insight information on this. Have few below questions1) Suppose if we receive incremental data with some modifications to existing record and in this case whether existing clustered ZCubes will be re-organized a...

Prashanth24 · 08-11-2024

This is good information. It is been mentioned that LC is replacement of both partitioning and Z-ordering. Partitioning basically places the data in the different folders. LC wont create such folders and place the data. I have below few questions1) S...

Prashanth24 · 08-07-2024

Thanks for the information. In the create_tasks function, do i need to create any object of jobs package to associate the whl list with each task for ex: jobs.TaskDependency is used to add the task dependency in the same function. Similar to this, do...

Prashanth24 · 08-05-2024

Thanks for the information. One more question specific to this task cost - there are different types of tasks like Notebook, Python, Wheel, Condition task etc. There will be some code like reading xml, yaml, json etc which requires normal python code...

Databricks Community

User Stats

User Activity

Databricks Liquid Clustering

Databricks Worker node - Would like to know number of memory in each core

Number of Min and Max nodes count for processing 5 TB data

Difference between Liquid clustering and Z-ordering

Error connecting Databricks Notebook using managed identity from Azure Data Factory

Re: Databricks Worker node - Would like to know number of memory in each core

Re: Difference between Liquid clustering and Z-ordering

Re: Difference between Liquid clustering and Z-ordering

Re: Databricks workflow creation using databricks sdk programming

Re: Databricks workflow each task cost