- 1995 Views
- 3 replies
- 0 kudos
Ingest data from SQL Server
I've been working on data ingestion from SQL Server to UC using lakeflow connect. Lakeflow connect actually made the work easier when everything is right. I am trying to incorporate this with DAB and this would work fine with schema and table tags fo...
- 1995 Views
- 3 replies
- 0 kudos
- 0 kudos
I’ve been using the notebook style as there are some transformations that need to be done on destination_table names as the source_table names has special characters. My requirement is: for example, I have a schema with 5 tables, 10 individual tables...
- 0 kudos
- 897 Views
- 3 replies
- 0 kudos
Databricks Row filter causing duplicates in Delta Table while writing
Hello,I have a databricks external table to which I have applied a row filter, After applying the row filter I see that the Delta merge functionality is not working as expected even with simple records ( not nested) and I have already tried execution...
- 897 Views
- 3 replies
- 0 kudos
- 0 kudos
What is the workaround to achieve the same functionality of upsert that we get with Merge?
- 0 kudos
- 1550 Views
- 2 replies
- 3 kudos
Run failed with error message Failed to resolve references: Parameter values exceed the size limit
Hi community,My team and I are facing an issue with the Parameter Values (see the title of this discussion) being passed through each task of a job. Unfortunately, this causes our job run to fail.Do you have any suggestions on how to handle parameter...
- 1550 Views
- 2 replies
- 3 kudos
- 3 kudos
We're dealing with this issue on our project in following way:- we have defined config JSON file (could also be YAML - doesn't matter)- and now let's say that you have param that has really long value - for the sake of example let's consider paramete...
- 3 kudos
- 2220 Views
- 3 replies
- 0 kudos
Azure Databricks – Lakehouse Federation MySQL Connection Fails but Works in Notebook
I am trying to set up a Lakehouse Federation connection to an Azure MySQL database. When I connect from a Databricks notebook using Python - mysql.connector, same cluster, it works fine.But when I set up the Lakehouse Federation connection and test i...
- 2220 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @asrivas , I’ve been trying to simulate this on my side and in my case I was able to complete the connection, but I believe in your case the issue comes from the MySQL setting --require_secure_transport=ON. In the notebook it works because the dri...
- 0 kudos
- 4085 Views
- 7 replies
- 10 kudos
Resolved! How to overwritten job parameter task inside a job task
Hi community,How to overwritten the job parameter inside the job task? Because seems that the job parameter has a higher priority than a task parameter although it is overwritten
- 4085 Views
- 7 replies
- 10 kudos
- 10 kudos
Hi Pilsner,Thanks for your response, the issue is that I need to know it before. In this case, we need to set inside a notebook for example the task values. I want to be able to set it at task value. I think it is not provided from Databricks.
- 10 kudos
- 1385 Views
- 1 replies
- 0 kudos
DLT Pipeline from Streaming Table
HiI have a bronze table with Product_id, *, start_at, end_At which is a streaming and SCD Type 2 Table, which means any change in product_attributes would insert a new row with end_at as null. So when we take this table with end_at as null , the tabl...
- 1385 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @km1837 ,Instead of trying to implement a stream table on a stream table, for your use case I think using Materialized View on next child table would be the best choice.For e.g.: @dlt.table(name="workspace.silver.current_product") def sample_trips...
- 0 kudos
- 3093 Views
- 9 replies
- 3 kudos
Resolved! issue updating DLT pipeline configurations using databricks sdk
I am updating dlt pipeline configs with job id , run id and run_datetime of the job , so that i can access these values inside dlt pipeline. below is the code i am using to do that. # Databricks notebook sourceimport sysimport loggingfrom databricks....
- 3093 Views
- 9 replies
- 3 kudos
- 1027 Views
- 2 replies
- 1 kudos
Can the default cluster Serverless of Databricks install Scala packages
Can the default cluster Serverless of Databricks install Scala packagesI need to use the spark-sftp package, but it seems that serverless is different from purpose compute, and I can only install python packages?There is another question. I can use p...
- 1027 Views
- 2 replies
- 1 kudos
- 1 kudos
no scala, you can't even run scala notebooks.about the sftp: the serverless compute is way more limited than general purpose clusters.what folder can't be found? dbfs or s3?
- 1 kudos
- 4276 Views
- 8 replies
- 0 kudos
Intermittent error: "Command failed because warehouse <<warehouse id>> was stopped."
The error "Command failed because warehouse <<warehouse id>> was stopped." has started popping up during deployment runs. Some times the error correlates with serverless warehouse cluster count reducing to zero while a query is running, sometimes it ...
- 4276 Views
- 8 replies
- 0 kudos
- 0 kudos
Hey all, having the same issue here. Just doing some development work connected to a serverless SQL warehouse from dbt. Suddenly getting the error "Command failed because warehouse <warehouse_id> was stopped."Nothing's changed between those runs  
- 0 kudos
- 3896 Views
- 5 replies
- 3 kudos
AutoLoader options includeExistingFiles and modifiedAfter not working
I'm using this code to read data from an ADLS Gen2 location. There are txt files present in sub-folders in the container. df_stream = spark.readStream \ .format("cloudFiles") \ .option("cloudFiles.format", "text") \ .optio...
- 3896 Views
- 5 replies
- 3 kudos
- 3 kudos
Root cause:includeExistingFiles is only evaluated the first time the stream is started with a fresh checkpoint. If the stream is restarted or the checkpoint folder is reused, changing this option will have no effect on subsequent runs—old files previ...
- 3 kudos
- 3681 Views
- 5 replies
- 5 kudos
Resolved! Extracting PDFs and using AI queries | best practices
Problem i am solving:Upload PDF → available in /Volumes/<catalog>/<schema>/<volume>/.Extract text with pdfplumber (or OCR if scanned).Store in Delta table for governance.Parse intelligently using:ai_query() with Databricks LLMs for flexible JSON outp...
- 3681 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi @ManojkMohan ,Maybe you're using wrong endpoint name. Try with databricks-meta-llama-3-3-70b-instructIn your case you're trying to call an API with following name: databricks-meta-llama-3-70b-instruct which I guess has small typo
- 5 kudos
- 1309 Views
- 3 replies
- 5 kudos
Resolved! Unable to Delete Failed Databricks Job VMs in Azure
My Job Compute had trouble on starting the cluster, acusing "Unexpected failure while waiting for the cluster (xxxx) to be ready: Cluster 'xxxx' is unhealthy"After multiple retries, a new error message appeared:"Operation could not be completed as it...
- 1309 Views
- 3 replies
- 5 kudos
- 5 kudos
UPDATE: Before opening the Support Ticket, the machines suddently disappeared. I deleted the Jobs definitions with its JobClusters definitions, and maybe that solved it, or after some hours the machines were cleaned. Not sure what cleaned it.Also I n...
- 5 kudos
- 5774 Views
- 3 replies
- 1 kudos
Resolved! Unable to unpack or read rar file
Hi everyone,I'm encountering an issue with the following code when trying to unpack or read a RAR file in Databricks: with rarfile.RarFile(s3_path) as rf: for file_info in rf.infolist(): with rf.open(file_info) as file: file_c...
- 5774 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Walter_C,I am also using this unrar utility but the problem it is a proprietary software and i am working for a client and this license could cause issues. What is the alternative to unrar so that we eliminate the risk of any legal compliance.
- 1 kudos
- 2527 Views
- 5 replies
- 1 kudos
Resolved! How to design Airship Integration with Azure Databricks
Hello,I have to push data from Airship and persists it to Delta tables. I think We can used SFTP , May someone please help me how to design the inbound part , it using SFTP on Airship end to push file on ADLS Gen2.networking and security consideratio...
- 2527 Views
- 5 replies
- 1 kudos
- 1 kudos
Inbound Flow DesignEnable SFTP on the ADLS Gen2 (or Azure Blob Storage) account;Generate and register an SSH public/private key pair with Airship, enter your SFTP endpoint credentials (username, host, port, key) in Airship’s settings to authenticate ...
- 1 kudos
- 3646 Views
- 10 replies
- 5 kudos
Resolved! CONTAINER_LAUNCH_FAILURE
Hello everyone!I need some help, unable to get cluster up and running. I did try creating classic compute but fails, is there any limit to use databricks community edition? Error here: { "reason": { "code": "CONTAINER_LAUNCH_FAILURE", "type...
- 3646 Views
- 10 replies
- 5 kudos
- 5 kudos
To all legacy community edition is working fine if you use dbr <= 15.4 for both general and ML modes.I think legacy community still far more better than free edition. I was selecting >15.4 DBR Thank you.
- 5 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
