- 1699 Views
- 3 replies
- 5 kudos
Resolved! Is Auto Loader open source now in Apache 4.1 SDP?
With Spark Declarative Pipelines (SDP) being open source now, does this mean that the Databricks Auto Loader functionality is also open source? Is it called something else? If not, how does the open-source version handle incremental data processing a...
- 1699 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @ChristianRRL ,No, autoloader is propriety to Databricks. It's not open sourced. Open source version of SDP uses spark structured streaming for incremental processing. Keep in mind that Auto Loader is basically just Spark streaming under the hood ...
- 5 kudos
- 1024 Views
- 1 replies
- 0 kudos
Resolved! Databricks free Edition to Azure Connectivity
Hello EveryoneAs a beginner in databricks, I have a question. Can we connect Databricks Free Edition to connect Azure Blob/ Gen 2 storage? I would like to create external tables on files on Azure and Delta lake tables on top of it.Your help is apprec...
- 1024 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Bkr-dbricks ,According to following topic Free Edition doesn't support external locations.Solved: If use databricks free version not free trail can ... - Databricks Community - 127421
- 0 kudos
- 1313 Views
- 1 replies
- 1 kudos
HTML Formatting Issue in Databricks Alerts
Hello everyone,I have recently encountered an issue with HTML formatting in custom templates for Databricks Alerts. Previously, the formatting worked correctly, but now the alerts display raw HTML instead of properly rendered content.For example, an ...
- 1313 Views
- 1 replies
- 1 kudos
- 1 kudos
hi @kivaniutenko thanks for reaching out. Databricks alerts still support basic HTML in email templates, but HTML will render correctly only for email destinations and only with simple, allowed tags.​​ Quick things to try Make sure you are using Ale...
- 1 kudos
- 1724 Views
- 2 replies
- 2 kudos
Resolved! Job Cluster Reuse
Hi, I have a job where a job cluster is reused twice for task A and task C. Between A and C, task B runs for 4 hours on a different interactive cluster. The issue here is that the job cluster doesn't terminate as soon as Task A is completed and sits ...
- 1724 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @SparkMan ,This is expected behavior with Databricks job cluster reuse unless you change your job/task configuration. Look at following documentation entry:So with your flow you have something like this:Task A (job cluster) → Task B (interactive c...
- 2 kudos
- 870 Views
- 1 replies
- 1 kudos
Resolved! Regarding Accelerators
Are there any databricks accelerators to convert the c# and qlikview code to pyspark ? We are using the Open source AI tools to convert now but wondering is there any better way to do the same?Thanks in advance
- 870 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @nkrish ,Unfortunately, I don't think so. Available accelerators you can find here:Databricks Solution Accelerators for Data & AI | DatabricksBut I haven't heard anything about accelerator for c# and qlikview specifically.
- 1 kudos
- 1251 Views
- 1 replies
- 0 kudos
Resolved! DLT Gold aggregation with apply_change
I am building a Gold table using Delta Live Tables (DLT). The Gold table contains aggregated data derived from a Silver table. Aggregation happens monthly. However, the requirement is Only the current (year, month) should be recalculated. Previous mo...
- 1251 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @deepu1 , Assuming that @dlt.table refers to a Materialized View (MV), you are correct that this is the standard way to create aggregated tables in the Gold layer. A Materialized View is essentially a table that stores the results of a specific qu...
- 0 kudos
- 1899 Views
- 5 replies
- 3 kudos
Resolved! How to use/install a driver in Spark Declarative Pipelines (ETL)?
Salutations,I'm using SDP for an ETL that extracts data from HANA and put it in the Unity Catalog. I defined a Policy with the needed driver:But I get this error:An error occurred while calling o1013.load. : java.lang.ClassNotFoundException: com.sap....
- 1899 Views
- 5 replies
- 3 kudos
- 3 kudos
At this time, Databricks does not offer native connectors for SAP HANA. You can find the complete list of managed connectors currently available in Databricks here. We generally recommend beginning with SAP’s own commercial tools, prioritizing SAP Bu...
- 3 kudos
- 14809 Views
- 5 replies
- 1 kudos
UDF not working inside transform() & lambda (SQL)
Below is toy example of what I'm trying to achieve, but don't understand why it fails. Can anyone explain why, and suggest a fix or not overly bloated workaround?%sqlcreate or replace function status_map(status int)returns stringreturn map(10, "STATU...
- 14809 Views
- 5 replies
- 1 kudos
- 1 kudos
Scoped variables in a transform() are not accessible by UDFs. However, you can workaround this using explode():# equivalent of: select transform(arr, e -> status_map(e.v1)) from s1 select collect_list(status_map(status_id)) from explode((select trans...
- 1 kudos
- 1150 Views
- 3 replies
- 4 kudos
Resolved! write both logging error Pyspark and Python exceptions
Hello guyz, Happy new year and best wishes for all of us. I am catching both Pyspark and Python exceptions but i want to write this logging error inside a delta table when i logging. Someone knows the best practise for this ? Thanks Cordially,
- 1150 Views
- 3 replies
- 4 kudos
- 1214 Views
- 1 replies
- 0 kudos
Resolved! Few queries on Autoloader
How to retrieve filename and file path from the trigger and consume in Databricks Notebook dynamicallyIf the same file is being modified with no change in name but in data then will this trigger work? If not what is the walkaround?In landing we are g...
- 1214 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Digvijay_11 ,1. You can use metadata column for that purpose File metadata column - Azure Databricks | Microsoft Learn2. With the default setting (cloudFiles.allowOverwrites = false), files are processed exactly once. When a file is appended to o...
- 0 kudos
- 1312 Views
- 2 replies
- 3 kudos
Resolved! Serverless Compute Spark Version Flexibility?
Hi there, I'm wondering what determines the Serverless Compute spark version? Is it based on the current DBR LTS? And is there a way to modify the spark version for serverless compute?For example, when I check the spark version for our serverless com...
- 1312 Views
- 2 replies
- 3 kudos
- 3 kudos
Serverless compute always run on the latest runtime version. You cannot choose it like in standard compute.
- 3 kudos
- 2431 Views
- 4 replies
- 10 kudos
Resolved! Testing Spark Declarative Pipeline in Docker Container > PySparkRuntimeError
Hi there, I see via an announcement last year that Spark Declarative Pipeline (previously DLT) was getting open sourced into Apache Spark, and I see that this recently is true as of Apache 4.1:Spark Declarative Pipelines Programming Guide I'm trying ...
- 2431 Views
- 4 replies
- 10 kudos
- 10 kudos
Hi @ChristianRRL ,In addition to @osingh 's answers, check out this old but good blog post about how to structure the pipelines's code to enable dev and test cycle: https://www.databricks.com/blog/applying-software-development-devops-best-practices-d...
- 10 kudos
- 1106 Views
- 2 replies
- 0 kudos
Resolved! daabricks workflow design
Hello Team,I have use-case in which i want to trigger another dlt pipeline if 1 table got succeded in my parent dlt pipeline. I dont want to create pipeline to pipeline dependency. Is there any way to create table to pipeline dependency?Thank youAnis...
- 1106 Views
- 2 replies
- 0 kudos
- 0 kudos
@Anish_2 - TUT is the solution. in TUT, instead of the parent pipeline "pushing" a notification, the child job is "pulled" into action by a metadata change.Set it up as below.Create a Databricks Job and add a Pipeline task pointing to your Secondary ...
- 0 kudos
- 1529 Views
- 2 replies
- 3 kudos
Resolved! Cluster crashes occasionally but not all of the time
We have a small cluster (Standard D2ads v6) with 8 gigs of ram and 2 cores. This is an all-purpose cluster and for some reason, the client demands to use this one for our ETL process. The ETL process is simple, the client drops parquet files in the b...
- 1529 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @NotCuriosAtAll ,You have undersized cluster for your workload. This error is typical on driver node with that high cpu consumption. You can check below article (and related solution):Job run fails with error message “Could not reach driver of clu...
- 3 kudos
- 2730 Views
- 4 replies
- 5 kudos
Resolved! DBR 17.3.3 introduced unexpected DEBUG logs from ThreadMonitor – how to disable?
After upgrading from DBR 17.3.2 to DBR 17.3.3, we started seeing a flood of DEBUG logs like this in job outputs:```DEBUG:ThreadMonitor:Logging python thread stack frames for MainThread and py4j threads: DEBUG:ThreadMonitor:Logging Thread-8 (run) stac...
- 2730 Views
- 4 replies
- 5 kudos
- 5 kudos
@iyashk-DBWe are currently using DBR version 17.3 LTS, and the issue is still occurring.Do you know when the fix is expected to be applied? We need this information to decide whether we should wait for the fix or proceed with the workaround you propo...
- 5 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
