Databricks Community

I'm creating this query with parameters in SQL Editor in Databricks and added it to the SQL Dashboard.Do we need to create Widget while creating parameters in SQL Editor? When i tried creating widget in SQL editor, Im getting syntax error near Widget...

Hi @Clara ,I don't think so. But you can build history such history tables by yourself. Desing ETL process that will extract data from system tables and store them in your own data tables. 

Thanks @Alberto_Umana . I will try that.

Hi @ashraf1395, The term "rate" refers to a special source in Apache Spark's Structured Streaming that generates data at a specified rate. This source is primarily used for testing and benchmarking purposes. When you use spark.readStream.format("rate...

 1. Instead of using many small partitions (e.g., country or region), opt for larger partitions, such as continent or time-based partitions (e.g., weekly or monthly). This will reduce the number of partitions and improve performance.2. Write data to ...

Hi @erigaud, Here’s how you can achieve this:   Define Variables in the Bundle Configuration: You can define custom variables in your databricks.yml file to hold the catalog names for different environments. For example:   variables:    catalog_name...

@devagya wrote:I'm working on this project which involves moving data from Infor to Databricks.Infor is somewhat of an enterprise solution. I could not find much resources on this. I could not even find any free trial option on their site.If anyone h...

Hi @schluca ,I’ve encountered an issue where an error occurred when trying to reference a table after deleting and recreating the source table for a Shallow Clone, and then performing the Shallow Clone again. As a solution, try deleting the destinati...

This will be extremely beneficial. We have certain use cases where we do not leverage Spark in our pipelines and lose the lineage. I would prefer to set an extra parameter when writing a table to specify the lineage. 

Hi @vinitkhandelwal, Looks like the token could be missing required permissions for the operation. Please refer to: You can clone public remote repositories without Git credentials (a personal access token and a username). To modify a public remote r...

I'm having a similar problem, my aim is from an Azure DataBricks notebook to invoke an AzureDataDactory pipeline I created an Access Connector for Azure Databricks to which I gave Data Factory Contributor permissions. Using these lines pythonfrom azu...