Databricks Community

Here are few DQ packages for DLT or LDP that you can try.1. Databricks Labs DQXPurpose-built for Spark and Databricks.Rule-based checks on DataFrames (batch & streaming).Supports quarantine and profiling.Lightweight and easy to integrate.2. Great Exp...

@hasnat_unifeye  I think you setup recently the Salesforce connector. Can you please have a look at the issue described above? Thanks. 

@ShivMukesh I have used UCX to migrate to Unity catalog. It is a great tool. But it also involves lot of workarounds specially in group migration and table migration. In group migration it renames the old workspace group and assigns the same permissi...

Option-1 using Oracle's Bulk extraction utility BICC. It can directly export the extracted data files (typically CSV) to Oracle cloud storage destination and then you could use ADF to get it copied over to ADLS.

It shouldn't. You have append only stream and SkipChangeCommit will ignore any modification that were applied to already existing files 

Hi @Han_bbb ,Could you provide more details? Is it workspace in your work or your private one? Which cloud provider? 

Found an answer!Basically pip was somehow installed the dependencies from the git repo first and was not following the given order so in order to solve this, I added the libraries for conda to install.``` conda_env = { "channels": [ "defa...

Reference Link - https://docs.databricks.com/aws/en/volumes/volume-files#upload-files-to-a-volume

Yes, for your setup, Databricks Auto Loader will create a separate event queue for each independent stream running with the cloudFiles.useManagedFileEvents = true option.As you are running - 1 stream per table, 1 unique directory per stream and 1 uni...

I think there is no other way. In any case, here is how I usually configure my (all-purpose and jobs compute) clusters to access secrets via environment variables so that you don't have to update all references if some similar issue arises again. The...

This issue is related to how Delta Lake’s structured streaming interacts with schema evolution and options like startingVersion and schemaTrackingLocation. The behavior you've observed has been noted by other users, and can be subtle due to how check...

There is indeed a better and more integrated way to make Delta Lake tables accessible in Redshift without manually generating manifests and dynamically creating external tables or partitions. Some important points and options: Databricks Delta Lake ...

The most effective way to display joined data from Azure Databricks tables (like Employee and Salary) in a React JS UI involves exposing your Databricks data through an API and then consuming that API in your frontend. Flask can work, but there are b...