Databricks Community

SantiNath_Dey

We are extracting data from SQL Server/Oracle using ADF and storing it in Parquet format. When reading the files in Databricks using spark.read.parquet, decimal values are getting truncated—for example, 1245.1111111189979 becomes 1245.111111118. This...

SantiNath_Dey

Hi,As part of our Databricks modernization program for the customer, we have received a request from the client regarding a Catalog template that they would like to populate to capture and articulate their business benefits.I wanted to check if we ha...

SantiNath_Dey

Currently implementing a Data Quality framework using the DQX framework with a metadata-driven architecture. The solution incorporates various data quality checks such as null checks, duplicate detection, date validation, and numeric validations.Coul...

SantiNath_Dey · 03-11-2026

We are implementing an incremental load for semi-structured data [ complex nested Json file ]using Auto Loader. To handle schema drifts—such as new fields, changes in column order, or data type and precision modifications (e.g., Decimal and Integer)—...

SantiNath_Dey · 03-09-2026

Hi ,We have multiple complex JSON files and need to flatten them, especially handling array data types. Whenever an array is present, we need to create a new child table and establish a relationship between the master and child tables. This will foll...

SantiNath_Dey

If fields are added in the JSON file from the source, the nested field names are captured in the rescue column. However, we do not get any indication of operation types such as addition, deletion, or data type changes. Attribute type changes and attr...

SantiNath_Dey

Thank you for your response.

SantiNath_Dey

Thanks for quick turn around , I will check and get back you..

SantiNath_Dey · 03-12-2026

Thank for quick response . let me implement the code .

SantiNath_Dey · 03-11-2026

Thanks for the quick response! I'm going to dive in and start building the implementation now.

Databricks Community

User Stats

User Activity

Decimal Precision Loss While Reading Parquet Files in Databricks

Availability of Catalog Template for Databricks Modernization Program

Implementing DQ Checks (Null, Duplicate, Date, Numeric) Using DQX

Workflow Notification: Pass/Failed –Schema Evolution/Rescue Mode Triggered for complex json file

Complex Json file Flatten Dynamically

Re: Workflow Notification: Pass/Failed –Schema Evolution/Rescue Mode Triggered for complex json fil

Re: Availability of Catalog Template for Databricks Modernization Program

Re: Implementing DQ Checks (Null, Duplicate, Date, Numeric) Using DQX

Re: Workflow Notification: Pass/Failed –Schema Evolution/Rescue Mode Triggered for complex json fil

Re: Complex Json file Flatten Dynamically