- 3688 Views
- 11 replies
- 4 kudos
Lakehouse federation for SQL server: database name with spaces
We're currently using lakehouse federation for various sources (Snowflake, SQL Server); usually succesful. However we've encountered a case where one of the databases on the SQL Server has spaces in its name, e.g. 'My Database Name'. We've tried vari...
- 3688 Views
- 11 replies
- 4 kudos
- 4 kudos
Hello all,We have a good amount of tables from an external ERP system that are being replicated to an existing dwh in an Azure SQL Server database.We have set up a foreign connection for this database and we can connect to the server and database. Sa...
- 4 kudos
- 102 Views
- 5 replies
- 2 kudos
Can't Migrate Auto Loader To File Events
Hi there,I am trying to migrate my auto loader job to use file events, but it's failing with this error:com.databricks.sql.util.UnexpectedHttpStatus: Failed to list objects. There are problems on the location that need to be resolved. Details: Failed...
- 102 Views
- 5 replies
- 2 kudos
- 2 kudos
I'm so sorry for this. Turns out I've been assigning roles to the wrong service account. I recently got access to the Storage Credential in Databricks and noticed the different service account.These roles were enough:Storage Blob Data Contributor (st...
- 2 kudos
- 236 Views
- 3 replies
- 0 kudos
Lakeflow Connect Snapshot Restarts After Gateway Restart
Hi everyoneWe recently started using Lakeflow Connect to ingest data from our on-prem SQL Server environment, and we’ve run into an issue related to gateway restarts.From our understanding, the gateway begins by performing full snapshots of each tabl...
- 236 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @DavidOldelius , If the initial snapshot has been fully ingested, then in case of ingestion gateway restart the connector will start from where it left and not from the beginning. Do you observe the behaviour you described during or after the snap...
- 0 kudos
- 2254 Views
- 4 replies
- 0 kudos
How to pass parameters to a notebook via URL?
I have a Notebook written in Python, and a Notebook's Dashboard that shows its result.The Notebook has a parameters, processed with a text widget: dbutils.widgets.text("p_test", "", "Enter Test Value") p_test = dbutils.widgets.get("p_test") When I en...
- 2254 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi, were you able to figure this out? This is important for dashboard embedding as well.https://<account>.cloud.databricks.com/embed/dashboardsv3/<dashboard>?o=<id>&p_param1=101&p_param_2=500&p_status=activethis is not passing the parameters as expec...
- 0 kudos
- 135 Views
- 4 replies
- 2 kudos
Resolved! Netsuite Data Connector Not Available
I see that the Azure Databricks Data Connector for Netsuite is in Public Preview. Unfortunately I am unable to see it in my instance. I have gone into Preview and selected it as enabled, have downloaded the JAR file from Netsuite and have setup the i...
- 135 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @rwhitepwt, From what I can see, having the NetSuite connector in Public Preview doesn’t automatically guarantee that the tile appears in every workspace. In addition to enabling the preview and creating the UC connection + uploading the SuiteAnal...
- 2 kudos
- 42640 Views
- 4 replies
- 4 kudos
Resolved! How to get the Job ID and Run ID and save into a database
We are having Databricks Job running with main class and JAR file in it. Our JAR file code base is in Scala. Now, when our job starts running, we need to log Job ID and Run ID into the database for future purpose. How can we achieve this?
- 42640 Views
- 4 replies
- 4 kudos
- 4 kudos
i came across a similar requirement and got it through named parameters. Wrote a community blog about it
- 4 kudos
- 463 Views
- 11 replies
- 6 kudos
Resolved! OrderBy is not sorting the results
Hi,I am currently using Lakeflow SDP ,firstly I am creating 2 views and then joining them and creating materialized view and using order by in the materialized view create function , but the results are not sorted does order by not work on materializ...
- 463 Views
- 11 replies
- 6 kudos
- 52 Views
- 0 replies
- 0 kudos
Connect Databrick's cluster with Artifactory
Hello,I'm trying to connect databricks with an own JFrog Artifactory. The objective is to download both PIP/JAR dependencies from it instead of connecting to maven-central/PyPi. Im struggling with JAR's. My aproximation to solve the problem is:1. Cre...
- 52 Views
- 0 replies
- 0 kudos
- 210 Views
- 3 replies
- 0 kudos
Structured streaming error- NON_TIME_WINDOW_NOT_SUPPORTED_IN_STREAMING
Hi,I was using window function row_number(),min,sum in the code, then the Lakeflow SDP pipeline was failing with the error: NON_TIME_WINDOW_NOT_SUPPORTED_IN_STREAMING - Window function is not supported on streaming dataframeswhat is the recommended a...
- 210 Views
- 3 replies
- 0 kudos
- 0 kudos
@Louis_Frolio suppose if I use foreachbatch I might end up with duplicates as the state is not maintainedcan you please share more information on max_by
- 0 kudos
- 2024 Views
- 5 replies
- 2 kudos
Exploring Postman Alternatives for API Testing in VSCode?
Has anyone here explored Postman alternatives within VSCode? I’ve seen mentions of Thunder Client and Apidog. Would love to know if they offer a smoother integration or better functionality.
- 2024 Views
- 5 replies
- 2 kudos
- 80 Views
- 2 replies
- 1 kudos
Is it required to run Lakeflow Connect on Serverless?
As the subject states, my question is:Is it required to run the Ingestion Pipeline in Lakeflow Connect on Serverless compute? Cause I try to define my own cluster in the DAB, but it raises an error:`Error: cannot create pipeline: You cannot provide c...
- 80 Views
- 2 replies
- 1 kudos
- 1 kudos
Yes — Lakeflow Connect ingestion pipelines always run on Serverless compute. Databricks overrides your compute config and switches back to serverless,because the ingestion connector requires it.
- 1 kudos
- 743 Views
- 2 replies
- 1 kudos
Support for JDBC writes from serverless compute
Hi team,Are there any plans in place to support JDBC writes using serverless compute.
- 743 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello! And what is the standard way to write to a external database through databricks? general purpose compute?Thanks very much.
- 1 kudos
- 216 Views
- 2 replies
- 3 kudos
Resolved! Questions on Auto Loader auto Listing Logic
Hi everyone,I’m investigating some performance patterns in our Auto Loader (S3) pipelines and would like to clarify the internal listing logic.Context: We run a batch job every hour using Auto Loader. Recently, after March 10th, we noticed our execut...
- 216 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @JIWON , 1. There is no such option; 2. Assuming that the job is triggered every hour, the spikes every 8-hours can be explained by this: To ensure eventual completeness of data in auto mode, Auto Loader automatically triggers a full directory lis...
- 3 kudos
- 4415 Views
- 4 replies
- 10 kudos
How do you define PyPi libraries on job level in Asset Bundles?
Hello,Reading the documentation, it does not state it is possible to define libraries on job level instead of on task level. It feels really counter-intuitive putting libraries on task level in Databricks workflows provisioned by Asset Bundles. Is th...
- 4415 Views
- 4 replies
- 10 kudos
- 10 kudos
Thanks @Witold ! Thought so. I decided to go with an init script where I install my dependencies rather than installing libraries. For future reference, this is what it looks like:job_clusters: - job_cluster_key: job_cluster new_cluster: ...
- 10 kudos
- 163 Views
- 4 replies
- 0 kudos
Kill/Cancel a Notebook Cell Running Too Long on an All-purpose Cluster
Hi everyone, I’m facing an issue when running a notebook on a Databricks All-purpose cluster. Some of my cells/pipelines run for a very long time, and I want to automatically cancel/kill them when they exceed a certain time limit.I tried setting spar...
- 163 Views
- 4 replies
- 0 kudos
- 0 kudos
@zenwanderer Have you looked into Query Watchdog? For Classic All-Purpose clusters this might be your best bet. https://docs.databricks.com/aws/en/compute/troubleshooting/query-watchdog
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
