Databricks Community

Hi Everyone,I'd like to start a discussion about the best practices for copying data between environments. Here's a typical setup:Environment Setup:The same region and metastore (Unity Catalog) is used across environments.Each environment has a singl...

Hi All,In my organization, we use Databricks Connect and VS Code for data engineering purposes. This setup is working great, especially for:- Debugging- Unit tests- GitHub Copilot- Reusable modules and custom librariesIn my view, the developer experi...

Hi,As a formal requirement in my project I need to keep original, raw (mainly CSVs and XMLs) files on the lake. Later on they are being ingested into Delta format backed medallion stages, bronze, silver, gold etc.From the audit, operations and discov...

Hi Databricks Community,If only possible I would like to use Shared Jobs Cluster on external orchestrator like Azure Data Factory (ADF) or Synapse Workspace.The main reasons for using Shared Job cluster are:reduction of start-up time (<1min vs 5 min ...

When trying to use `dbutils.fs.cp` in the #databricks-connect #databricks-connect context to upload files to Azure Datalake Gen2 I get a malformed URI errorI have used the code provided here:https://learn.microsoft.com/en-gb/azure/databricks/dev-tool...