- 1845 Views
- 1 replies
- 1 kudos
DLT started by SERVICE_UPGRADE
HelloI'm developing a dlt pipeline, configured in continuous mode.I'm still in dev mode, so I stop my pipeline when i'm not working on it.My problem is that the pipeline is frequently started by SERVICE_UPGRADE.example of message:'Update xxxxx starte...
- 1845 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @alex mo Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 10524 Views
- 9 replies
- 14 kudos
Resolved! Pipeline workflow dude
Hi! I have a problem. I'm using an autoloader to ingest data from raw to a Delta Lake, but when my pipeline starts, I want to apply the pipeline only to the new data. The autoloader ingests data into the Delta Lake, but now, how can I distinguish the...
- 10524 Views
- 9 replies
- 14 kudos
- 14 kudos
Hi @Alejandro Piury Pinzón We haven't heard from you since the last response from @Tyler Retzlaff , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be he...
- 14 kudos
- 20160 Views
- 7 replies
- 8 kudos
Resolved! How to manipulate files in an external location?
According to the documentation, the usage of external locations is preferred over the use of mount points.Unfortunately the basic funtionality to manipulate files seems to be missing.This is my scenario:create a download folder in an external locatio...
- 20160 Views
- 7 replies
- 8 kudos
- 8 kudos
The main problem was related to the network configuration of the storage account: Databricks did not have access. Quite strange that it did manage to create folders...Currently dbutils.fs functionality is working.For the zipfile manipulation: that on...
- 8 kudos
- 3869 Views
- 3 replies
- 1 kudos
Resolved! Autoload files in wide table format, but store it unpivot in Streaming Table
Hey, I get wide table format in csv file. Where each sensor have its own column. I want to store it in Delta Live Streaming Table. But since it is inefficient to process it and storage space, due to varying frequency and sensor amount. I want to tran...
- 3869 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Simen Småriset,Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
- 1 kudos
- 8244 Views
- 2 replies
- 2 kudos
In the VSCode Databricks Extension how can one specify notebook parameters to a pass to a workflow job?
I have successfully used the VSCode extension for Databricks to run a notebook on a cluster from my IDE. However in order to test effectively without changing the source, I need a way to pass parameters to the workflow job.I have tried various ways ...
- 8244 Views
- 2 replies
- 2 kudos
- 2 kudos
@matthew kaplan I am not using widgets but what works is running it by pressing F5 in the python file you want to run.
- 2 kudos
- 3986 Views
- 3 replies
- 5 kudos
Issue with connection our Alation instance to Azure Databricks using the Unity Catalog OCF connector
When we try and do the above I am able to get the list of schemas. But when I select one to injest we are then getting issue due to it trying to access system.lineage.table_lineage. When I look in the System catalog I can only see a schema called inf...
- 3986 Views
- 3 replies
- 5 kudos
- 5 kudos
We are working with an agency that is a Databricks partner if that is what you mean.
- 5 kudos
- 3726 Views
- 3 replies
- 4 kudos
Resolved! Kept outputs
Hi everyone,I was wondering if perhaps someone of you could tell me which kinds of outputs are kept in a notebook after the cluster to which it is attached is terminated... Actually, I am asking it especially because I lost some visualization that I ...
- 3726 Views
- 3 replies
- 4 kudos
- 2602 Views
- 1 replies
- 2 kudos
Update query from Azure databricks to Oracle table
I need to update a single row on a on-prem Oracle table via jdbc connection.Please note, I don't want to append, just have to update a row, is it possible ?
- 2602 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Tony S Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 2879 Views
- 1 replies
- 0 kudos
- 2879 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sourav Nandy Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 9817 Views
- 1 replies
- 2 kudos
getting error while runction show function
I was using String indexer, while fitting, transforming I didn't get any erro. but While runnign show function I am getting error, I mention the error beloworg.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 45.0 failed...
- 9817 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Bojja Giri Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 31328 Views
- 2 replies
- 2 kudos
Resolved! Am I being charged for Starter Warehouse Pro?
When I go to add data, I see that the Starter Warehouse Pro cluster spun up after the first use and has been there for a long time. It does not show in my clusters and I can't find a way to shut it down. Am I being charged for this? If so, how do I s...
- 31328 Views
- 2 replies
- 2 kudos
- 17300 Views
- 4 replies
- 3 kudos
Databricks CI/CD Azure DevOps
Hi All. I have a scenario where there are few .sql scripts present in my repo. Is there any way we can execute those SQLs on Databricks via Azure DevOps CI/CD pipeline?Please help.
- 17300 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Divyansh Jain Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 3 kudos
- 4338 Views
- 1 replies
- 3 kudos
how to alter a column into an IDENTITY column
Here's me use case: I'm migrating out of an old DWH, into Databricks. When moving dimension tables into Databricks, I'd like old SKs (surrogate keys) to be maintained, while creating the SKs column as an IDENTITY column, so new dimension values get a...
- 4338 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Ronen Levi Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 3 kudos
- 2029 Views
- 1 replies
- 0 kudos
How can I know if databricks auto-detected to use tuneFileSizesForRewrites?
We are having some issues with merge performance, so I went and read a bit in the documentation, I found this section:https://docs.databricks.com/delta/tune-file-size.html#autotune-file-size-based-on-workload"Databricks recommends setting the table p...
- 2029 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Ron Serruya Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 2221 Views
- 1 replies
- 1 kudos
ReadStream & writeStream at gold layer level
Hello, I have seen in many places readStream and writeStream in gold layer, Is it correct to use readStream and writeStream for gold layer ? knowing that a gold table is no not valid for streaming.is there some logic when to use readStream/ writeStr...
- 2221 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Ibrahim ISSOUANI Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 585 | |
| 349 | |
| 287 |
