Databricks Community

KosmaS · 08-05-2024

Hey Everyone,I experience data skewness for: df = (source_df .unionByName(source_df.withColumn("region", lit("Country"))) .groupBy("zip_code", "region", "device_type") .agg(countDistinct("device_id").alias("total_active_unique"), count("device_id").a...

KosmaS · 07-28-2024

To cache/persist an action needs to be triggered. I'm just wondering, will it make any difference if, after persisting some df, I use, for instance, take(5) instead of count()?Will it be a bit more effective, because of sending results from 5 partiti...

KosmaS · 07-19-2024

Hey,I had a stable notebook within the whole job. It contains one action defined as dumping data to s3. Currently, it started generating some issues. Maybe someone can suggest either how to investigate it further or what to try to do with such kinds ...

KosmaS · 08-22-2024

Hey @Retired_mod thanks for the reply. I tried to spend some time on your response.You're suggesting 'double aggregation' and as I'd be guessing it should look more or less this way:df = (source_df .unionByName(source_df.withColumn("region", lit("Cou...

KosmaS · 07-30-2024

Thanks Rishabh-Pandey for your reply!So to clarify by giving some code example. Let's say this is a PROD code: df1 = spark.read..... + transformations df1.persist(StorageLevel.DISK_ONLY) or df1.cache() df2 = df1.transformation... df3 = df1.transf...

Databricks Community

User Stats

User Activity

Skewness / Salting with countDistinct

Efficient caching/persisting

Lost Databricks' dependency in a job.

Re: Skewness / Salting with countDistinct

Re: Efficient caching/persisting