Databricks Community

In your scenario, if the data loaded on day 2 also includes all the data from day 1, you can still apply a "remove duplicates" logic. For instance, you could compute a hashdiff by hashing all the columns and use this to exclude rows you've already se...

Just maybe - there is no outbound connection on DEV from Cluster VNET to URL you are trying to get ? You can spin ALl purpose cluster and try testing connection with %sh magic command

Hello, if you want, you can deploy specific jobs only in the development environment. Since you have only two environments, a straightforward approach is to modify your jobs YAML definition as follows:resources: jobs: # Define the jobs to be de...

Is there a way to export a notebook though terraform? If so please provide examples?With terraform-provider-databricks.exe we are able to export all the notebooks from workspace but not a single notebook. Any suggestions ? 

UPDATE: tried adding writeStream.start() like error suggested + as per other posts and ended up with following error/code: @dlt.table(   name="ETD_Bz",  temporary=False)def Bronze():    return (spark.readStream                 .format("delta")       ...

Hi @Retired_mod  , The point is that in the aim of our project, we need the timestamp attribute to be as they are from the source system. So basically our aim would be to have the attribute without the timezone conversion. I did the below tests so fa...

Thanks @User16502773013 @jacovangelder That is something interesting to know that Lakehouse Federation does not support UPSERT(merge into...)!!@jacovangelder  I think with above approach(link shared by you) only support "append" and "overwrite", but ...