- 2273 Views
- 2 replies
- 1 kudos
HOW TO SKIP A STEP FROM AN EXISTING WORKFLOW
Hi, can any one guide me in below scenario:Suppose we have created a workflow with 10 steps and our batches are running properly but due to some business requirement/testing purpose we have to skip step 4 and run the rest of job from step 5 onwords t...
- 2273 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @17abhishek ,You can try if/else conditions between the tasks with a Job parameter of "IsActiveTask" with a True or False.But it would be great if databricks team just add ability to simply disable some task from UI.
- 1 kudos
- 1552 Views
- 3 replies
- 0 kudos
DBR 16.0 spark read Azure Blob file Failed.
ala:570) at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:354) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.lang.NullPointerException at java.base/java.lang.Class.forName0(Native Method) at java.b...
- 1552 Views
- 3 replies
- 0 kudos
- 17844 Views
- 7 replies
- 1 kudos
FileNotFoundError: [Errno 2] No such file or directory: when I try to unzip .tar or .zip files it gives me this error
Hello, how are you? I have a small problem. I need to unzip some .zip, tar files. and gz inside these may have multiple files trying to unzip the .zip files i got this errorFileNotFoundError: [Errno 2] No such file or directory: but the files are in ...
- 17844 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @Alfredo Vallejos​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...
- 1 kudos
- 15855 Views
- 7 replies
- 15 kudos
Resolved! Failed to launch pipeline cluster
Hi, there. I encountered an issue when I was trying to create my delta live table pipeline. The error is "DataPlaneException: Failed to launch pipeline cluster 1202-031220-urn0toj0: Could not launch cluster due to cloud provider failures. azure_error...
- 15855 Views
- 7 replies
- 15 kudos
- 15 kudos
you can create the pool instance in the databricks under compute/pool and assign the value in the json of the DLT pipeline. With this, we will control on pool min workers and max workers and the reuse of the pools available by other pipelines. "node_...
- 15 kudos
- 10421 Views
- 4 replies
- 0 kudos
Exploring the Use of Databricks as a Transactional Database
Hey everyone, I’m currently working on a project where my team is thinking about using Databricks as a transactional database for our backend application. We're familiar with Databricks for analytics and big data processing, but we're not sure if it’...
- 10421 Views
- 4 replies
- 0 kudos
- 0 kudos
I have a similar situation in my data quality check process. During this stage, I frequently find errors or potential issues that can stop the pipeline. Each of these errors requires manual intervention, which might involve making edits or supplying ...
- 0 kudos
- 5330 Views
- 1 replies
- 0 kudos
Databricks:null error message: Cannot resolve hostname: Caused by: UnknownHostException
Hello,We are suddenly getting following error message while running any code from Databricks which is accessing Blob storage.We checked our App registration key and it's not expired.If we run to "dbutils.fs.mount" and we are able to get some info and...
- 5330 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @pora , just checking if this is still an issue, otherwise where is help still required? Could you also please elaborate on the setup and requirement.
- 0 kudos
- 5642 Views
- 1 replies
- 0 kudos
Understanding and loading SQL Server Temp Tables from Databricks
Hi everyone...Came across this question in Stackoverflow and wanted to try my hand in trying it. Unfortunately I have not been able to fix it...https://stackoverflow.com/questions/78953930/create-and-load-sql-server-temp-table-table-or-table-from-dat...
- 5642 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ranged_coop , thanks for your question!Just checking if you were able to make progress, how far you were able to get and if still needing assistance ?
- 0 kudos
- 4318 Views
- 2 replies
- 0 kudos
Failure to locate configuration file when using spark-submit task
Hello,We are trying to run a job with a spark-sumit task on cluster mode, this spark submit task requires a configuration file application.conf that we provide using --files flag option in the spark-submit parameters and put an alias on it using \#ap...
- 4318 Views
- 2 replies
- 0 kudos
- 0 kudos
@Tham99 would it be possible to share the java.io.FileNotFoundException stacktrace? And refernece from the Driver log about the file localization process.
- 0 kudos
- 6332 Views
- 3 replies
- 0 kudos
Databricks Autoloader Schema Evolution throws StateSchemaNotCompatible exception
I am trying to use Databricks Autoloader for a very simple use case:Reading JSONs from S3 and loading them into a delta table, with schema inference and evolution.This is my code:self.spark \ .readStream \ .format("cloudFiles") \ .o...
- 6332 Views
- 3 replies
- 0 kudos
- 0 kudos
@robertkoss I have the exact same problem... have you found a solution ?
- 0 kudos
- 16880 Views
- 3 replies
- 4 kudos
Resolved! Unable to read data from ADLS using databricks serverless sql pool
I have a data bricks workspace and an Azure data lake storage account.Both are present in the same Vnet.Unity catalog is enabled in the worksapce.I have created some tables in unity catalog.I am able to query the data from the tables when I use the a...
- 16880 Views
- 3 replies
- 4 kudos
- 4 kudos
I'm having the same issue and tried to follow the document shared above, but quite not sure what I'm missing, as I can't make it work. Can someone please help me here? TIA.
- 4 kudos
- 9976 Views
- 4 replies
- 3 kudos
Data contract implementation best practices
Hi all.We've written some .yml files for our data products in a UC-enabled workspace (dev and prod). We've constructed a directory identical to the one containing the scripts which ultimately creates these products and put them there, initially for g...
- 9976 Views
- 4 replies
- 3 kudos
- 3 kudos
Thank you for your follow-up question. Yes, if it helps, this would be a good starting point/demo: import yaml import pytest # Load the data contract with open('data_contract.yml', 'r') as file: data_contract = yaml.safe_load(file) # Example da...
- 3 kudos
- 4099 Views
- 5 replies
- 1 kudos
Resolved! Delta Log Files in GCS Not Deleting Automatically Despite Configuration
Hello Databricks Community,I am experiencing an issue with Delta Lake where the _delta_log files are not being deleted automatically in GCS bucket, even though I have set the table properties to enable this behavior. Here is the configuration I used:...
- 4099 Views
- 5 replies
- 1 kudos
- 1 kudos
Glad it helps, and agree to monitoring this behaviour closely. Should you need further assistance, please don't hesitate to reach out.
- 1 kudos
- 2741 Views
- 1 replies
- 0 kudos
Not able to create external location in unity catalog
You do not have the CREATE EXTERNAL LOCATION privilege for this credential. Contact your metastore administrator to grant you the privilege to this credential. -- My user id is having access to Create external location then also i am getting the sam...
- 2741 Views
- 1 replies
- 0 kudos
- 0 kudos
If you go to the specific storage credential you are trying to use to create this External Location, under permissions does it actually show you have All privileges or the CREATE EXTERNAL LOCATION permission?
- 0 kudos
- 2629 Views
- 2 replies
- 0 kudos
Implement SCD Type 2 in Bronze Layer of DLT Pipeline with Structured Streaming
Hi everyone,I am implementing SCD Type 2 in the Bronze layer of a Delta Live Table (DLT) pipeline using Structured Streaming. I am curious about the necessity of having a table or view before loading data into the Bronze table. Without this, it seems...
- 2629 Views
- 2 replies
- 0 kudos
- 0 kudos
Optimizing SCD Type 2: Ensure that the column used for sequencing is a sortable data type.Handle out-of-sequence records by specifying a column in the source data that represents the proper ordering of the source data.1Use the track_history_except_co...
- 0 kudos
- 848 Views
- 1 replies
- 0 kudos
Serverless compute for file notification mode
I am creating a table that ingests data from aws s3 using the 'file notification mode'. With a single user cluster, it works. I would like to use Serverless compute, but I get an error about authentication. Is it possible to do this, or are there alt...
- 848 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Isa1, Using Serverless compute with Auto Loader in file notification mode can indeed present authentication challenges. Based on the context provided, here are some insights and alternatives: Authentication Issues with Serverless Compute:Server...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 567 | |
| 349 | |
| 287 |
