Getting Different results when I am running the Global Templary table (Transform Query)

Pradeep_Namani — Mon, 25 Nov 2024 10:08:29 GMT

We are running a logic in an Azure Databricks notebook using Python with Spark. Initially, we read data from ADLS and load it into a global temporary table to perform data quality checks. We then recreate the same temporary table. Afterward, we use these temporary tables to write Spark SQL for transformations, execute the query, and load the results into another global temporary table. Finally, the result set is written back to ADLS.

Problem: When reading the transformed data, one of the column values is inconsistent. Each time we query the data, the column alternates between returning null and the another some value.

RUN1: Written Premium column consist of Null Value

RUN2: Written Premium column consist of Value

Please help me on this, Thanks in Advance

Re: Getting Different results when I am running the Global Templary table (Transform Query)

Pradeep_Namani — Mon, 25 Nov 2024 10:30:19 GMT

The script is too large to paste in here, so please get in touch with me to to obtain it.

topic Re: Getting Different results when I am running the Global Templary table (Transform Query) in Data Engineering

Getting Different results when I am running the Global Templary table (Transform Query)

Re: Getting Different results when I am running the Global Templary table (Transform Query)