Databricks Community

Thank you @Retired_mod  for the prompt reply. This clears the things and also distinguishes between spark-scala and pyspark. Appreciate your explanation. Will apply this and also share any findings based on this which will help the community!

Thanks @Witold ! Thought so. I decided to go with an init script where I install my dependencies rather than installing libraries. For future reference, this is what it looks like:job_clusters: - job_cluster_key: job_cluster new_cluster: ...

I would keep both write operations separate, i.e. they should write in own tables/partitions. In later stages (e.g. silver), you can easily merge them.

Can you show us your code?

Hi @Retired_mod,Thanks for the quick response. I have checked my region is on Public Preview for DLT, North Europe. But I am not able to see the "Serverless" checkbox.Am I missing some detail?Thanks!

 Implementing conditional logic in Databricks using Terraform involves setting up tasks and condition checks between them. Here's how you can structure your Terraform code to achieve the desired workflow:Step 1: Define Databricks Notebooks as TasksAs...

Just overtake Databricks sparksession.from pyspark.sql import SparkSession spark = SparkSession.getActiveSession()