We are running a logic in an Azure Databricks notebook using Python with Spark. Initially, we read data from ADLS and load it into a global temporary table to perform data quality checks. We then recreate the same temporary table. Afterward, we use these temporary tables to write Spark SQL for transformations, execute the query, and load the results into another global temporary table. Finally, the result set is written back to ADLS.
Problem: When reading the transformed data, one of the column values is inconsistent. Each time we query the data, the column alternates between returning null and the another some value.
RUN1: Written Premium column consist of Null Value
RUN2: Written Premium column consist of Value
Please help me on this, Thanks in Advance
