Databricks Community

shan-databricks

We are using Lakehouse Federation to connect to a SQL Server database and are currently looking for a CI/CD deployment solution using DABs. Could you please advise on the available options for deploying from the Dev environment to the Stage environme...

shan-databricks

I have five data ingestion jobs using Lakeflow Connect from SQL Server to Databricks, along with a Spark Declarative Pipeline to load data from bronze to silver. I need guidance on promoting these pipelines and jobs from Dev to Stage and Prod, and ho...

shan-databricks

I am using SQL Server, Postgres, and MongoDB as data sources, connecting through Spark and JDBC connector. I would like to store the credentials and connection details in Databricks, pass them as job parameters, and need guidance on possible approach...

shan-databricks · 06-16-2026

Facing an issue with the MongoDB Void/Null type in Databricks, which requires explicit casting or conversion to an array or struct of strings. Looking for guidance on how to handle this data type when reading from MongoDB and writing to a Delta table...

shan-databricks · 06-04-2026

I am trying to read a MongoDB collection using spark.read.format("mongodb"). However, when I attempt to display the collection, I receive the error: "UC command is not supported without recommendation." Please help resolve this issue.

shan-databricks · 01-26-2026

Thank you for your response. I will try the integration and options and will reach out if I need further assistance. Shanmugam

shan-databricks · 07-23-2025

I have already analysed the issue and, yes, the schema doesn't match one of the rows, and it moved the complete file into badRecords and I have seen the behavior and that's fine, and thanks for the response.

shan-databricks · 07-23-2025

I am using the same in my code, but instead of moving only bad data to badRecordsPath, it is moving complete file into badRecordsPath, which has good data as well in the same file.

shan-databricks · 07-22-2025

I am using the Autoloader features spark.readStream and writeStream, and have used the option(badRecordsPath), so when I use either Permissive or Drop malformed or FailFast, I get an exception, like if 'badRecordsPath' is specified mode is not allowe...

shan-databricks · 07-22-2025

I am using the Autoloader features spark.readStream and writeStream, and have used the option(badRecordsPath), so when I use either Permissive or Dropmalformed or FailFast, I get an exception, like if 'badRecordsPath' is specified mode is not allowed...

Databricks Community

User Stats

User Activity

How to Deploy Lakehouse Federation Using DABs from the Dev Environment to the Stage Environment

How to promote Lakeflow Connect and Spark Declarative Pipeline to a higher environment

How to store credentials in Databricks and assign them to job parameters

Issue with MongoDB Void Null Type in Databricks

MongoDB Spark Connector UC command is not supported without recommendation

Re: What are the prerequisites for connecting Confluent Kafka with Databricks?

Re: Databricks Autoloader BadRecords path Issue

Re: Databricks Autoloader BadRecords path Issue

Re: Databricks Autoloader BadRecords path Issue

Re: Databricks Autoloader BadRecords path Issue