- 1817 Views
- 2 replies
- 2 kudos
Resolved! Job Cluster Reuse
Hi, I have a job where a job cluster is reused twice for task A and task C. Between A and C, task B runs for 4 hours on a different interactive cluster. The issue here is that the job cluster doesn't terminate as soon as Task A is completed and sits ...
- 1817 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @SparkMan ,This is expected behavior with Databricks job cluster reuse unless you change your job/task configuration. Look at following documentation entry:So with your flow you have something like this:Task A (job cluster) → Task B (interactive c...
- 2 kudos
- 920 Views
- 1 replies
- 1 kudos
Resolved! Regarding Accelerators
Are there any databricks accelerators to convert the c# and qlikview code to pyspark ? We are using the Open source AI tools to convert now but wondering is there any better way to do the same?Thanks in advance
- 920 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @nkrish ,Unfortunately, I don't think so. Available accelerators you can find here:Databricks Solution Accelerators for Data & AI | DatabricksBut I haven't heard anything about accelerator for c# and qlikview specifically.
- 1 kudos
- 1323 Views
- 1 replies
- 0 kudos
Resolved! DLT Gold aggregation with apply_change
I am building a Gold table using Delta Live Tables (DLT). The Gold table contains aggregated data derived from a Silver table. Aggregation happens monthly. However, the requirement is Only the current (year, month) should be recalculated. Previous mo...
- 1323 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @deepu1 , Assuming that @dlt.table refers to a Materialized View (MV), you are correct that this is the standard way to create aggregated tables in the Gold layer. A Materialized View is essentially a table that stores the results of a specific qu...
- 0 kudos
- 2109 Views
- 5 replies
- 3 kudos
Resolved! How to use/install a driver in Spark Declarative Pipelines (ETL)?
Salutations,I'm using SDP for an ETL that extracts data from HANA and put it in the Unity Catalog. I defined a Policy with the needed driver:But I get this error:An error occurred while calling o1013.load. : java.lang.ClassNotFoundException: com.sap....
- 2109 Views
- 5 replies
- 3 kudos
- 3 kudos
At this time, Databricks does not offer native connectors for SAP HANA. You can find the complete list of managed connectors currently available in Databricks here. We generally recommend beginning with SAP’s own commercial tools, prioritizing SAP Bu...
- 3 kudos
- 14994 Views
- 5 replies
- 1 kudos
UDF not working inside transform() & lambda (SQL)
Below is toy example of what I'm trying to achieve, but don't understand why it fails. Can anyone explain why, and suggest a fix or not overly bloated workaround?%sqlcreate or replace function status_map(status int)returns stringreturn map(10, "STATU...
- 14994 Views
- 5 replies
- 1 kudos
- 1 kudos
Scoped variables in a transform() are not accessible by UDFs. However, you can workaround this using explode():# equivalent of: select transform(arr, e -> status_map(e.v1)) from s1 select collect_list(status_map(status_id)) from explode((select trans...
- 1 kudos
- 1220 Views
- 3 replies
- 4 kudos
Resolved! write both logging error Pyspark and Python exceptions
Hello guyz, Happy new year and best wishes for all of us. I am catching both Pyspark and Python exceptions but i want to write this logging error inside a delta table when i logging. Someone knows the best practise for this ? Thanks Cordially,
- 1220 Views
- 3 replies
- 4 kudos
- 1285 Views
- 1 replies
- 0 kudos
Resolved! Few queries on Autoloader
How to retrieve filename and file path from the trigger and consume in Databricks Notebook dynamicallyIf the same file is being modified with no change in name but in data then will this trigger work? If not what is the walkaround?In landing we are g...
- 1285 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Digvijay_11 ,1. You can use metadata column for that purpose File metadata column - Azure Databricks | Microsoft Learn2. With the default setting (cloudFiles.allowOverwrites = false), files are processed exactly once. When a file is appended to o...
- 0 kudos
- 1446 Views
- 2 replies
- 3 kudos
Resolved! Serverless Compute Spark Version Flexibility?
Hi there, I'm wondering what determines the Serverless Compute spark version? Is it based on the current DBR LTS? And is there a way to modify the spark version for serverless compute?For example, when I check the spark version for our serverless com...
- 1446 Views
- 2 replies
- 3 kudos
- 3 kudos
Serverless compute always run on the latest runtime version. You cannot choose it like in standard compute.
- 3 kudos
- 2616 Views
- 4 replies
- 10 kudos
Resolved! Testing Spark Declarative Pipeline in Docker Container > PySparkRuntimeError
Hi there, I see via an announcement last year that Spark Declarative Pipeline (previously DLT) was getting open sourced into Apache Spark, and I see that this recently is true as of Apache 4.1:Spark Declarative Pipelines Programming Guide I'm trying ...
- 2616 Views
- 4 replies
- 10 kudos
- 10 kudos
Hi @ChristianRRL ,In addition to @osingh 's answers, check out this old but good blog post about how to structure the pipelines's code to enable dev and test cycle: https://www.databricks.com/blog/applying-software-development-devops-best-practices-d...
- 10 kudos
- 1199 Views
- 2 replies
- 0 kudos
Resolved! daabricks workflow design
Hello Team,I have use-case in which i want to trigger another dlt pipeline if 1 table got succeded in my parent dlt pipeline. I dont want to create pipeline to pipeline dependency. Is there any way to create table to pipeline dependency?Thank youAnis...
- 1199 Views
- 2 replies
- 0 kudos
- 0 kudos
@Anish_2 - TUT is the solution. in TUT, instead of the parent pipeline "pushing" a notification, the child job is "pulled" into action by a metadata change.Set it up as below.Create a Databricks Job and add a Pipeline task pointing to your Secondary ...
- 0 kudos
- 1628 Views
- 2 replies
- 3 kudos
Resolved! Cluster crashes occasionally but not all of the time
We have a small cluster (Standard D2ads v6) with 8 gigs of ram and 2 cores. This is an all-purpose cluster and for some reason, the client demands to use this one for our ETL process. The ETL process is simple, the client drops parquet files in the b...
- 1628 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @NotCuriosAtAll ,You have undersized cluster for your workload. This error is typical on driver node with that high cpu consumption. You can check below article (and related solution):Job run fails with error message “Could not reach driver of clu...
- 3 kudos
- 2831 Views
- 4 replies
- 5 kudos
Resolved! DBR 17.3.3 introduced unexpected DEBUG logs from ThreadMonitor – how to disable?
After upgrading from DBR 17.3.2 to DBR 17.3.3, we started seeing a flood of DEBUG logs like this in job outputs:```DEBUG:ThreadMonitor:Logging python thread stack frames for MainThread and py4j threads: DEBUG:ThreadMonitor:Logging Thread-8 (run) stac...
- 2831 Views
- 4 replies
- 5 kudos
- 5 kudos
@iyashk-DBWe are currently using DBR version 17.3 LTS, and the issue is still occurring.Do you know when the fix is expected to be applied? We need this information to decide whether we should wait for the fix or proceed with the workaround you propo...
- 5 kudos
- 1865 Views
- 4 replies
- 3 kudos
Mongo Db connector - Connection timeout when trying to connect to AWS Document DB
I am on Databricks Run Time LTE 14.3 Spark 3.5.0 Scala 2.12 and mongodb-spark-connector_2.12:10.2.0. Trying to connect to Document DB using the connector and all I get is a connection timeout. I tried using PyMongo, which works as expected and I can ...
- 1865 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @rijin-thomas - Can you please allow the CIDR block for databricks account VPC from aws document db sg ( Executor connectivity stated by@bianca_unifeye ) .
- 3 kudos
- 649 Views
- 1 replies
- 1 kudos
Resolved! Tab navigation between fields in dashboards is random
Tab navigation between fields in published dashboards seem very random.I have a dashboard with multiple text input fields (mapped to query paramters / filters). I expect to move logically between them when pressing tab (keyboard navigation), but I mo...
- 649 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @tvdh! You can share this as product feedback so it’s visible to the Databricks product team and can be tracked and prioritized.
- 1 kudos
- 5624 Views
- 2 replies
- 1 kudos
Resolved! Databricks APP OBO User Authorization
Hi All,We are using on-behalf of user authorization method for our app and the x-forwarded-access-token is expiring after sometime and we have to redeploy our app to rectify the issue. I am not sure what is the issue or how we can keep the token aliv...
- 5624 Views
- 2 replies
- 1 kudos
- 1 kudos
I am confronted with a similar error. I am also using obo user auth and have implemented accessing the token via st.context.headers.get('x-forwarded-access-token') for every query and do not save it in a cache. Still, after 1 hour, i am hit with the...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
