- 4190 Views
- 1 replies
- 0 kudos
Adding a message to azure service bus
I am trying to send a message to a service bus in azure. But I get following error:ServiceBusError: Handler failed: DefaultAzureCredential failed to retrieve a token from the included credentials.This is the line that fails: credential = DefaultAzure...
- 4190 Views
- 1 replies
- 0 kudos
- 0 kudos
@Sega2 - Can you explicitly useClientSecretCredential and try
- 0 kudos
- 2768 Views
- 1 replies
- 0 kudos
Spot instances usage in Azure Databricks
Hi all,as per the below article -https://community.databricks.com/t5/technical-blog/optimize-costs-for-your-data-and-ai-workloads-with-azure-and-aws/ba-p/662411. it is possible to choose the number of spot instances using 'availability' parameter. Bu...
- 2768 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @noorbasha534, Thanks for your question! 1. 'availability' Parameter: The 'availability' parameter in Azure Databricks controls whether the compute uses on-demand or spot instances. The values for this parameter are: ON_DEMAND_AZURE: This value...
- 0 kudos
- 2530 Views
- 11 replies
- 3 kudos
Accessing delta tables using API outside azure (Workiva)
I need to access delta tables with API outside azure using in a reporting tool workiva with using the connector. Can someone able to provide the details on how I can achieve it
- 2530 Views
- 11 replies
- 3 kudos
- 3 kudos
I need to retrieve delta tables using an API outside of Azure without requiring a connection using a reporting tool named Workiva. I would appreciate it if someone could tell me exactly how to do that.
- 3 kudos
- 37859 Views
- 4 replies
- 2 kudos
Understanding Used Memory in Databricks Cluster
Hello, I wonder if anyone could give me any insights regarding used memory and how could I change my code to "release" some memory as the code runs. I am using a Databricks Notebook.Basically, what we need to do is perform a query, create a spark sql...
- 37859 Views
- 4 replies
- 2 kudos
- 2 kudos
I have exactly the same kind of problem.I really do not understand why my driver goes out of memory meanwhile I do not cache anything in Spark.Since I don't cache anything, I expect references to objects that are not used anymore to be freed.Even a s...
- 2 kudos
- 3012 Views
- 1 replies
- 2 kudos
Brickflow - An opinionated python framework to help build and deploy Databricks workflows at scale
Hello, Databricks Community, I’m thrilled to introduce “Brickflow,” our innovative solution designed to streamline the development and deployment of workflows at scale, leveraging straightforward Python coding. This tool has already achieved a signif...
- 3012 Views
- 1 replies
- 2 kudos
- 2 kudos
Brickflow is indeed an awesome tool to work w/ Databricks!
- 2 kudos
- 641 Views
- 2 replies
- 0 kudos
Query related to Z ordering
I have join on two large tables.If i apply Z ordering on 3 columns for both the tables ,I am joining two tables on the basis on the same 3 columns used for Z ordering ,Will i get any benefit of Z ordering on performance when i use Joins here .So as p...
- 641 Views
- 2 replies
- 0 kudos
- 0 kudos
I have asked this query because w.r.t Predicate query push down to storage .So here as part of joins . Data needs to be loaded into memory first and then join is performed . So How does Z order help here if it can't skip the data being fetched from s...
- 0 kudos
- 3889 Views
- 1 replies
- 0 kudos
Resolved! getting short of error while mount to storage account.
while mount to the storage account using this below code dbutils.fs.mount( source=f"wasbs://{cointainer_name}@{storage_name}.blob.core.windows.net", mount_point=f"/mnt/{cointainer_name}", extra_configs={f"fs.azure.account.key.{storage_name}.blob.c...
- 3889 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @kumarPatra_07 , Greetings! From the above code which you have shared, I could see that you are using a WASBS driver to mount the storage and as of now, WASB is already deprecated. Reference document : https://learn.microsoft.com/en-us/azure/dat...
- 0 kudos
- 3346 Views
- 1 replies
- 0 kudos
Log data from reports in PowerBI
Where to find log data from PowerBI? I need to find what tables are being used in my PowerBI reports that are pointing to Databricks. I tried system.access.audit but I'm not finding new data when I refresh my report
- 3346 Views
- 1 replies
- 0 kudos
- 0 kudos
@ksenija To enable ODBC logging in PowerBI, go to C:\Program Files\Microsoft Power BI Desktop\bin\ODBC Drivers\Simba Spark ODBC Driver folder and create or edit the file microsoft.sparkodbc.ini and update as below: [Driver] LogLevel=6 LogPath=<...
- 0 kudos
- 13912 Views
- 5 replies
- 2 kudos
Resolved! Databricks job keep getting failed due to GC issue
There is a job that running successful but it's for more than a month we are experiencing long run which gets failed. In the stdout log file(attached), there are numerous following messages:[GC (Allocation Failure) [PSYoungGen:...] and [Full GC ...
- 13912 Views
- 5 replies
- 2 kudos
- 2 kudos
Thanks a lot @shahabm for your prompt response, appreciate it. I'll try to debug in this direction.Thanks again!
- 2 kudos
- 3975 Views
- 4 replies
- 2 kudos
Resolved! Timezone in silver tables
Hello,What is the best practice in Databricks for storing DateTime data in silver layer tables, considering the source data is in AEST and we store it in UTC by default?Thanks
- 3975 Views
- 4 replies
- 2 kudos
- 2 kudos
@Freshman wrote:Hello,What is the best practice in Databricks for storing DateTime data in silver layer tables, considering the source data is in AEST and we store it in UTC by default?ThanksThe best practice for storing DateTime data in the Silver l...
- 2 kudos
- 1236 Views
- 1 replies
- 1 kudos
Resolved! if-else statement in DAB YAML file
Hi.Is it possible to use a "better" way to override the "git_branch" key's value on the right file (which is the resource-yaml-file)?Or a different way, like an "if-else" statement. I'd like to have it all in the resource-yaml-file instead of overrid...
- 1236 Views
- 1 replies
- 1 kudos
- 1 kudos
I think your approach is correct using targets and ${var.git_branch} variables. At the moment there is no if-else statement, but will investigate internally if there is a better way.
- 1 kudos
- 599 Views
- 2 replies
- 0 kudos
Read task level parameters in python
I am creating Databricks jobs and tasks using a python package. I have defined a task level parameter and would like to reference it in my script using sys.argv. How can I do that ?
- 599 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks, but the link works for notebook. I have a python package run as python wheel and am wondering how to access the parameters. When I run the job its not able to understand the task level parameters in sys.argv
- 0 kudos
- 1285 Views
- 2 replies
- 4 kudos
DLT multiple source table to single silver table generating unexpected result
Hi,I´ve been trying this all day long. I'm build a POC of a pipeline that would be used on my everyday ETL.I have two initial tables, vendas and produtos, and they are as the following:vendas_rawvenda_idproduto_iddata_vendaquantidadevalor_totaldth_in...
- 1285 Views
- 2 replies
- 4 kudos
- 4 kudos
When dealing with Change Data Capture (CDC) in Delta Live Tables, it's crucial to handle out-of-order data correctly. You can use the APPLY CHANGES API to manage this. The APPLY CHANGES API ensures that the most recent data is used by specifying a co...
- 4 kudos
- 1176 Views
- 4 replies
- 1 kudos
How to use Widgets with SQL Endpoint in Databricks?
I' trying to use widgets with SQL Endpoints but I'm encountering an error, whereas they work seamlessly with Databricks Interactive Cluster. While query parameters can substitute widgets in SQL endpoints, but I specifically require dropdown and multi...
- 1176 Views
- 4 replies
- 1 kudos
- 1 kudos
Got it, let me test, i think there is no specific way to do it, but if you add the option All as the first on the list it should select it
- 1 kudos
- 1041 Views
- 3 replies
- 1 kudos
Resolved! Issue with filter_by in Databricks SQL Query History API (/api/2.0/sql/history/queries)
Hi everyone,I'm trying to use the filter_by parameter in a GET request to /api/2.0/sql/history/queries, but I keep getting a 400 Bad Request error. When I use max_results, it works fine, but adding filter_by causes the request to fail.Example value f...
- 1041 Views
- 3 replies
- 1 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 466 | |
| 349 | |
| 287 |
