Databricks Community

I think you need to link the git account with databricks by passing the access token which is generated in githubFollow the document for reference: https://docs.databricks.com/en/repos/get-access-tokens-from-git-provider.htmlNote : While creating the...

Hi @Gilg, When multiple pipelines are simultaneously accessing the same directory path and utilizing Autoloader in continuous mode, it is crucial to consider the management of file locks and data consistency carefully.    Let's delve into the specifi...

I recommend you read this article (Managed vs External tables) and answer the following questions:do I require direct access to the data outside of Azure Databricks clusters or Databricks SQL warehouses?If yes, then External is your only optionIn rel...

Thanks for the reply @Kaniz_Fatma . I have some questions related to you answer.Checkpoint Location:Does deleteing checkpoint folder (or only files?) mean that next run of AutoLoader will load all files from provided source locations? So it will dupl...

Thanks for the answer!It seems that using shared access mode adds overhead.  The nodes/driver are not stressed at all (cpu/ram/network).We use UC only.The clusters seems configured correctly (using the same cluster in single user mode changes perform...

Hi @essura, Setting up a Docker image for your dbt execution is a great approach. Let’s dive into the details. Prebuilt Docker Images: dbt Core and all adapter plugins maintained by dbt Labs are available as Docker images. These images are distr...

Hi @zero234, It seems you’re encountering an issue with your Delta Live Tables (DLT) pipeline where you’re trying to create two streaming tables from different sources with distinct schemas. Let’s dive into this! DLT is a powerful feature in Data...

job_cluster_key string [ 1 .. 100 ] characters ^[\w\-\_]+$ If job_cluster_key, this task is executed reusing the cluster specified in job.settings.job_clusters.Create a new job | Jobs API | REST API reference | Databricks on AWS

df.select("field1","field2","field3").distinct()do you mean get distinct rows for selected column?

replying to my above questionwe cannot use inferschema on streaming table we need to externally specify schema can anyone please suggest a way to write data in nested form to streaming table and if this is possible?

Keep your DLT code separate from your comparison code, and run your comparison code once your DLT data has been ingested.

Hi @Avinash_Narala There is no one-click solution to refactor all table names notebooks with UC's three level namespaces. At least, manual updating table names is required during the migration process.One option is you can you search feature. Search ...