https://community.databricks.com/t5/data-engineering/confuse-about-large-memory-usage-of-cluster/m-p/81838#M36424 <P><SPAN>We set up a demo DLT pipeline with no data involved:</SPAN></P><P>&nbsp;</P><P>&nbsp;</P><LI-CODE lang="python"><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/97035">@Dlt</a>.table( name="demo" ) def sample(): df = spark.sql("SELECT 'silver' as Layer") return df</LI-CODE><P>&nbsp;</P><P><SPAN>However, when we check the metric of the cluster, it looks like 10GB memory has already been used which doesn’t make sense.</SPAN></P><P><SPAN>I noticed that the access mode for the cluster is “shard”. Does this mean the 10GB memory was consumed by other users maybe?</SPAN></P><P><SPAN>If so,&nbsp; do we use the cluster at the same time or do I take over this one after the other user finishes?</SPAN></P> Mon, 05 Aug 2024 08:48:39 GMT guangyi 2024-08-05T08:48:39Z https://community.databricks.com/t5/data-engineering/confuse-about-large-memory-usage-of-cluster/m-p/81838#M36424 <P><SPAN>We set up a demo DLT pipeline with no data involved:</SPAN></P><P>&nbsp;</P><P>&nbsp;</P><LI-CODE lang="python"><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/97035">@Dlt</a>.table( name="demo" ) def sample(): df = spark.sql("SELECT 'silver' as Layer") return df</LI-CODE><P>&nbsp;</P><P><SPAN>However, when we check the metric of the cluster, it looks like 10GB memory has already been used which doesn’t make sense.</SPAN></P><P><SPAN>I noticed that the access mode for the cluster is “shard”. Does this mean the 10GB memory was consumed by other users maybe?</SPAN></P><P><SPAN>If so,&nbsp; do we use the cluster at the same time or do I take over this one after the other user finishes?</SPAN></P> Mon, 05 Aug 2024 08:48:39 GMT https://community.databricks.com/t5/data-engineering/confuse-about-large-memory-usage-of-cluster/m-p/81838#M36424 guangyi 2024-08-05T08:48:39Z