- 6010 Views
- 9 replies
- 14 kudos
Resolved! Pipeline workflow dude
Hi! I have a problem. I'm using an autoloader to ingest data from raw to a Delta Lake, but when my pipeline starts, I want to apply the pipeline only to the new data. The autoloader ingests data into the Delta Lake, but now, how can I distinguish the...
- 6010 Views
- 9 replies
- 14 kudos
- 14 kudos
Hi @Alejandro Piury Pinzón We haven't heard from you since the last response from @Tyler Retzlaff , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be he...
- 14 kudos
- 12651 Views
- 7 replies
- 8 kudos
Resolved! How to manipulate files in an external location?
According to the documentation, the usage of external locations is preferred over the use of mount points.Unfortunately the basic funtionality to manipulate files seems to be missing.This is my scenario:create a download folder in an external locatio...
- 12651 Views
- 7 replies
- 8 kudos
- 8 kudos
The main problem was related to the network configuration of the storage account: Databricks did not have access. Quite strange that it did manage to create folders...Currently dbutils.fs functionality is working.For the zipfile manipulation: that on...
- 8 kudos
- 1921 Views
- 3 replies
- 1 kudos
Resolved! Autoload files in wide table format, but store it unpivot in Streaming Table
Hey, I get wide table format in csv file. Where each sensor have its own column. I want to store it in Delta Live Streaming Table. But since it is inefficient to process it and storage space, due to varying frequency and sensor amount. I want to tran...
- 1921 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Simen Småriset,Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
- 1 kudos
- 5988 Views
- 2 replies
- 2 kudos
In the VSCode Databricks Extension how can one specify notebook parameters to a pass to a workflow job?
I have successfully used the VSCode extension for Databricks to run a notebook on a cluster from my IDE. However in order to test effectively without changing the source, I need a way to pass parameters to the workflow job.I have tried various ways ...
- 5988 Views
- 2 replies
- 2 kudos
- 2 kudos
@matthew kaplan I am not using widgets but what works is running it by pressing F5 in the python file you want to run.
- 2 kudos
- 2120 Views
- 3 replies
- 5 kudos
Issue with connection our Alation instance to Azure Databricks using the Unity Catalog OCF connector
When we try and do the above I am able to get the list of schemas. But when I select one to injest we are then getting issue due to it trying to access system.lineage.table_lineage. When I look in the System catalog I can only see a schema called inf...
- 2120 Views
- 3 replies
- 5 kudos
- 5 kudos
We are working with an agency that is a Databricks partner if that is what you mean.
- 5 kudos
- 2095 Views
- 3 replies
- 4 kudos
Resolved! Kept outputs
Hi everyone,I was wondering if perhaps someone of you could tell me which kinds of outputs are kept in a notebook after the cluster to which it is attached is terminated... Actually, I am asking it especially because I lost some visualization that I ...
- 2095 Views
- 3 replies
- 4 kudos
- 1685 Views
- 1 replies
- 2 kudos
Update query from Azure databricks to Oracle table
I need to update a single row on a on-prem Oracle table via jdbc connection.Please note, I don't want to append, just have to update a row, is it possible ?
- 1685 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Tony S Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 1981 Views
- 1 replies
- 0 kudos
- 1981 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sourav Nandy Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 8914 Views
- 1 replies
- 2 kudos
getting error while runction show function
I was using String indexer, while fitting, transforming I didn't get any erro. but While runnign show function I am getting error, I mention the error beloworg.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 45.0 failed...
- 8914 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Bojja Giri Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 28882 Views
- 2 replies
- 2 kudos
Resolved! Am I being charged for Starter Warehouse Pro?
When I go to add data, I see that the Starter Warehouse Pro cluster spun up after the first use and has been there for a long time. It does not show in my clusters and I can't find a way to shut it down. Am I being charged for this? If so, how do I s...
- 28882 Views
- 2 replies
- 2 kudos
- 14196 Views
- 4 replies
- 3 kudos
Databricks CI/CD Azure DevOps
Hi All. I have a scenario where there are few .sql scripts present in my repo. Is there any way we can execute those SQLs on Databricks via Azure DevOps CI/CD pipeline?Please help.
- 14196 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Divyansh Jain Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 3 kudos
- 2553 Views
- 1 replies
- 3 kudos
how to alter a column into an IDENTITY column
Here's me use case: I'm migrating out of an old DWH, into Databricks. When moving dimension tables into Databricks, I'd like old SKs (surrogate keys) to be maintained, while creating the SKs column as an IDENTITY column, so new dimension values get a...
- 2553 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Ronen Levi Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 3 kudos
- 1117 Views
- 1 replies
- 0 kudos
How can I know if databricks auto-detected to use tuneFileSizesForRewrites?
We are having some issues with merge performance, so I went and read a bit in the documentation, I found this section:https://docs.databricks.com/delta/tune-file-size.html#autotune-file-size-based-on-workload"Databricks recommends setting the table p...
- 1117 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Ron Serruya Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 1036 Views
- 1 replies
- 1 kudos
ReadStream & writeStream at gold layer level
Hello, I have seen in many places readStream and writeStream in gold layer, Is it correct to use readStream and writeStream for gold layer ? knowing that a gold table is no not valid for streaming.is there some logic when to use readStream/ writeStr...
- 1036 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Ibrahim ISSOUANI Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 1348 Views
- 1 replies
- 0 kudos
Failed to import `Ray` on jupyter notebook.
While working on my school's Linux server, I encountered an issue while attempting to install and import Ray in my Jupyter Notebook. I successfully installed the package ray==2.4.0, but encountered an error when trying to import it, specifically stat...
- 1348 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Jaeseon Song Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos