- 4650 Views
- 11 replies
- 3 kudos
Resolved! Unreliable file events on Azure Storage (SFTP) for job trigger
Hi allI got a job trigger by a file event on the external location.The location and jobs triggers are working fine when uploading file via Azure Portal.I need SFTP trigger, so I went into the event grid, found subscription for the storage account on ...
- 4650 Views
- 11 replies
- 3 kudos
- 3 kudos
UpdateAppears that even uploading via UI does not trigger it any more. It did trigger weeks ago.I have just uploaded a file in UI and saw this message in the storage queue:{"topic":"/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Storage/st...
- 3 kudos
- 857 Views
- 1 replies
- 0 kudos
Databricks Lakebridge: Azure SQL DB to Databricks (Error in Data and Schema Validation)
Hi community experts,I am getting error while Data and Schema Validation with the Reconciler. As attached here screenshots. Please help resolve this issue.Output:
- 857 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @shubham007 ,As stated in another thread. I think this error could be related to misconfiguration on your side. Lakebridge is trying to find following table in your SQL Server instance -> None.SalesLT.customerBut look at which database reconciliat...
- 0 kudos
- 2076 Views
- 1 replies
- 2 kudos
Databrick Genie
I have curated silver and gold tables in Advana that feed downstream applications. Other organizations also create tables for their own use. Can Databricks Genie query across tables from different pipelines within the same organization and across mul...
- 2076 Views
- 1 replies
- 2 kudos
- 2 kudos
Can you explain the landscape a bit more? The term "pipelines" means something specific in Databricks. You mention "across multiple organizations." What does that mean? Are you guys using Unity Catalog, are all the tables/data in Unity? Please elab...
- 2 kudos
- 2045 Views
- 4 replies
- 4 kudos
Resolved! Does the free version of Databricks not support external storage data sources?
1、Can the data I use with the free version of Databricks on Azure only be stored on Azure, AWS, and Google Cloud Storage?2、Assuming the network is connected, can the paid version be used to access other publicly stored data (i.e., custom storage spac...
- 2045 Views
- 4 replies
- 4 kudos
- 4 kudos
Not sure if this is a cheeky way to get around bringing files in: https://community.databricks.com/t5/data-engineering/connect-to-azure-data-lake-storage-using-databricks-free-edition/m-p/127900#M48116 but I answered a similar thing on a different po...
- 4 kudos
- 1334 Views
- 1 replies
- 1 kudos
Identify updated rows during incremental refresh in DLT Materialized Views
Hello, every time that I run a delta live table materialized view in serverless , I get a log of "COMPLETE RECOMPUTE". I realised I was using current_timestamp as a column in my MV to identify rows which got updated in the last refresh. But that make...
- 1334 Views
- 1 replies
- 1 kudos
- 1 kudos
@ManoramTaparia The issue is that current_timestamp() makes your MV non-deterministic, forcing complete recomputes. Here's how to fix it:Solution: Use the Source Table's Change TrackingOption 1: Leverage Source Table's Timestamp Column@Dlt.table(name...
- 1 kudos
- 1990 Views
- 5 replies
- 2 kudos
- 1990 Views
- 5 replies
- 2 kudos
- 2 kudos
Hello @yinan Good day!!Databricks, being a cloud-based platform, does not have direct built-in support for reading data from a truly air-gapped (completely offline, no network connectivity) Cloudera Distribution for Hadoop (CDH) environment. In such...
- 2 kudos
- 996 Views
- 2 replies
- 0 kudos
Using Databricks to transform cloudera lakehouse on-prem without bringing the data to cloud
I am looking for a solution to connect databricks to cloudera lakehouse hosted on-prem and transform the data using databricks without bringing the data to databricks delta tables or cloud storage. once the transformation is done the data need to be ...
- 996 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello, What is your data volume? You can connect using jdbc/odbc but this process will be slower if the data volume is too high.Another way of connecting is if your cloudera storage is in HDFS then you can also connect through HDFS API as well.
- 0 kudos
- 2172 Views
- 4 replies
- 2 kudos
How can I structure pipeline-specific job params separately in Databricks Asset Bundle.
Hi all, I am working with databricks asset bundle and want to separate environment-specific job params (for example, for "env" and "dev") for each pipeline within my bundle. I need each pipeline to have its own job params values for different environ...
- 2172 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi azam-io, were you able to solve your problem? Are you trying to have different parameters depending on the environment, or a different parameter value? I think the targets would allow to specify different parameters per environment / target. As fo...
- 2 kudos
- 3172 Views
- 2 replies
- 1 kudos
Resolved! assets bundle
Hello Guys,I am working on assets bundle. So i want to make it generic for all team like ( analytics, data engineering), Someone could you share a best practice for this purpose ? Cordially,
- 3172 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi seefoods, Were you able to achieve that generic asset bundle setup? I've been working on something, potentially, similar, and I'd be happy to discuss it, hoping to share experiences. While what I have works for a few teams, it is focused on declar...
- 1 kudos
- 2067 Views
- 4 replies
- 0 kudos
Git integration inconsistencies between git folders and job git
It's a little confusing and limiting that the git integration support is inconsistent between the two options available.Sparse checkout is only supported when using a workspace Git folder, and checking out by commit hash is only supported when using ...
- 2067 Views
- 4 replies
- 0 kudos
- 3558 Views
- 6 replies
- 7 kudos
Resolved! Getting data from the Spark query profiler
When you navigate to Compute > Select Cluster > Spark UI > JDBC/ODBC There you can see grids of Session stats and SQL stats. Is there any way to get this data in a query so that I can do some analysis? Thanks
- 3558 Views
- 6 replies
- 7 kudos
- 7 kudos
Hi @IONA ,As @Louis_Frolio correctly suggested there no native way to get stats from JDBC/ODBC Spark UI.1. You can try to use query history system table, but it has limited number of metrics %sql SELECT * FROM system.query.history 2. You can use /a...
- 7 kudos
- 35945 Views
- 7 replies
- 1 kudos
Resolved! Could not reach driver of cluster
Hi, Rencently, I am seeing issue Could not reach driver of cluster <some_id> with my structure streaming job when migrating to unity catalog and found this when checking the traceback:Traceback (most recent call last):File "/databricks/python_shell/...
- 35945 Views
- 7 replies
- 1 kudos
- 1 kudos
It seems like a temporary connectivity or cluster initialization glitch. So if anyone else runs into this, try re-running the job before diving into deeper troubleshooting - it might just work!Hope this helps someone save time.
- 1 kudos
- 1325 Views
- 1 replies
- 1 kudos
Resolved! Can schemaHints dynamically handle nested json structures? (Part 2)
Hi there, I'd like to follow up on a prior post:https://community.databricks.com/t5/data-engineering/can-schemahints-dynamically-handle-nested-json-structures/m-p/130209/highlight/true#M48731Basically I'm wondering what's the best way to set *both* d...
- 1325 Views
- 1 replies
- 1 kudos
- 1 kudos
I am not aware on schemahints supporting wildcards for now. It would be awesome to have though, I agree.So I think you are stuck with what is already proposed in your previous post, or exploding the json or other transformations.
- 1 kudos
- 1192 Views
- 1 replies
- 1 kudos
Could not reach driver of cluster
I am running a pipeline job in Databricks and it failed with the following message:Run failed with error message Could not reach driver of cluster 5824-145411-p65jt7uo. This message is not very descriptive, and I am not able to identify the root ca...
- 1192 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @minhhung0507 ,Typically this error could appear when there's a high load on the driver node. Another reason could be related to high garbage collection on driver node as well as high memory and cpu which leads to throttling, and prevents the driv...
- 1 kudos
- 13100 Views
- 7 replies
- 6 kudos
Resolved! SFTP Autoloader
Hello, Don't know if it is possible, but I am wondering if it is possible to ingest files from a SFTP server using autoloader ? Or do I have to first copy the files to my dbfs and then use autoloader on that location ? Thank you !
- 13100 Views
- 7 replies
- 6 kudos
- 6 kudos
Hi @erigaud We haven't heard from you since the last response from, @BriceBuso and I was checking back to see if her suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to others. Al...
- 6 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
