- 3184 Views
- 3 replies
- 0 kudos
How to stop a continuous pipeline which is set to RETRY on FAILURE and failing for some reason
I have created a pipeline which is continuous and set to RETRY on FAILURE. For some reason it keeps failing and retrying. Is there any way I can stop it. Hitting Stop button throws an error.
- 3184 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @szymon_dybczak I already tried to remove it via REST API but got same error as in the pipeline logs. Eventually, I had to remove workspace to get rid of it.
- 0 kudos
- 3581 Views
- 4 replies
- 1 kudos
How to get catalog and schema from system query table
Hi,We are querying the system.query table to parse query history. If the table in the query is not fully qualified with its catalog and schema, how can we derive the catalog and schema?Thanks,Jen
- 3581 Views
- 4 replies
- 1 kudos
- 1 kudos
There is no straightforward method to get this data. Run the query to check the defaults:SELECT current_catalog() AS default_catalog, current_schema() AS default_schema;Catalog and schema may be changed in the query, so if you have query text you...
- 1 kudos
- 6374 Views
- 9 replies
- 1 kudos
Java heap issue, GC allocation failure while writing data from mysql to adls
Hi Team,I am reading 60 million -80million data from mysql server and writing into ADLS in parquet format but i am getting java heap issue, GC allocation failure and out of memory issue.below are my cluster configuration Driver - 56GB Ram, 16 coreWo...
- 6374 Views
- 9 replies
- 1 kudos
- 5063 Views
- 3 replies
- 0 kudos
FileAlreadyExistsException when restarting a structured stream with checkpoint (DBR 14.3)
Structured streams with a checkpoint location, which have been running fine for months, can no longer be restarted properly. When restarting they fail with a FileAlreadyExistsException.I reproduced the issue in the attached pdf. Has anyone else exper...
- 5063 Views
- 3 replies
- 0 kudos
- 0 kudos
Apparently the issue was caused by a change in Azure, which was hotfixed cfr https://learn.microsoft.com/en-us/answers/questions/2007592/how-to-fix-the-specified-path-already-exists-issue?comment=question#newest-question-comment.Checkpoints work fine...
- 0 kudos
- 2012 Views
- 1 replies
- 2 kudos
Liquid clustering within partitions
As the tables are already partitioned. Is it possible to have liquid clustering within a partition or recreating the table is the only option?
- 2012 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @sashikanth ,No, it's not possible to have liquid clustering within a partition. According to documentation:"You can enable liquid clustering on an existing table or during table creation. Clustering is not compatible with partitioning or ZORDER, ...
- 2 kudos
- 1875 Views
- 3 replies
- 4 kudos
Databricks job trigger in specific times
Hello,I have a Databricks notebook that processes data and generates a list of JSON objects called "list_json". Each JSON object contains an item called "time_to_send" (in UTC datetime format). I want to find the best way to send these JSON messages ...
- 1875 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi everyone,Thank you for your responses to my question.@szymon_dybczak, if I understood correctly, your suggestion is based on running the Databricks job in continuous mode. However, this might incur significant costs if the cluster is running every...
- 4 kudos
- 1186 Views
- 1 replies
- 0 kudos
Different GCP Service Account for cluster (compute) creation?
I have a Databricks workspace that is attached to a GCP Service Account from a project named "random-production-data". I want to create a cluster (compute) on Databricks that uses a different Service Account from another project for isolation purpose...
- 1186 Views
- 1 replies
- 0 kudos
- 0 kudos
@LeoRickli wrote:I have a Databricks workspace that is attached to a GCP Service Account from a project named "random-production-data". I want to create a cluster (compute) on Databricks that uses a different Service Account from another project for ...
- 0 kudos
- 1216 Views
- 1 replies
- 0 kudos
Databricks Workflow File Based Trigger
Hi All,Is there any way to define multiple paths in file arrival trigger setting for Databricks job?For a single path it's working fine.
- 1216 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @monojmckvie ,You can specify only 1 path as per documentation:https://docs.databricks.com/en/jobs/file-arrival-triggers.html
- 0 kudos
- 6479 Views
- 6 replies
- 0 kudos
Getting SparkConnectGrpcException: (java.io.EOFException) error when using foreachBatch
Hello, I am trying to write a simple upsert statement following the steps in the tutorials. here is what my code looks like:from pyspark.sql import functions as Fdef upsert_source_one(self df_source = spark.readStream.format("delta").table(self.so...
- 6479 Views
- 6 replies
- 0 kudos
- 0 kudos
Here is the full message Exception has occurred: SparkConnectGrpcException (java.io.IOException) Connection reset by peer grpc._channel._MultiThreadedRendezvous: _MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.INTERNAL deta...
- 0 kudos
- 3137 Views
- 2 replies
- 0 kudos
spark.sql not supporting kwargs as documented
This documentation https://api-docs.databricks.com/python/pyspark/latest/pyspark.sql/api/pyspark.sql.SparkSession.sql.html#pyspark.sql.SparkSession.sql claims that spark.sql() should be able to take kwargs, such that the following should work:display...
- 3137 Views
- 2 replies
- 0 kudos
- 3360 Views
- 2 replies
- 2 kudos
foreachBatch
With parameterized SQL queries in Structured Streaming's foreachBatch, there's no longer a need to create temp views for the MERGE command.
- 3360 Views
- 2 replies
- 2 kudos
- 2 kudos
Note that this functionality broke somewhere between DBR 13.3 and 15, so best bet is 15.4 LTSSolved: Parameterized spark.sql() not working - Databricks Community - 56510
- 2 kudos
- 9436 Views
- 5 replies
- 10 kudos
Is there any way to save the notebook in the "Results Only" view?
Hi! I'm looking for a solution to save a notebook in HTML format that has the "Results Only" view (without the executed code). Is there any possibility to do that?Thank you
- 9436 Views
- 5 replies
- 10 kudos
- 10 kudos
Use option "+New dashboard" in the top menu (picture icon). Add results there (use display() in code to show data), and then you can export the dashboard to HTML.
- 10 kudos
- 7584 Views
- 7 replies
- 4 kudos
Resolved! Databricks Asset Bundles complex variable for cluster configuration substitute
Using this page of the DAB docs, I tried to substitute cluster configuration by a variable. That way, I want to predefine different job cluster configurations. Doing exactly what is used in the docs yields this error:Error: failed to load [...]/speci...
- 7584 Views
- 7 replies
- 4 kudos
- 1955 Views
- 1 replies
- 1 kudos
Logging in Databricks
Hi All,I am trying to create a log using python logging package.Is this allowed in databricks and any sample working code that you can share?Thank you for your guidance.Best regards,NC
- 1955 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @NC ,I recommend below video that guides you how to use logging in databricks:Introduction to Logging and Quality Control in Databricks (youtube.com)
- 1 kudos
- 2034 Views
- 1 replies
- 1 kudos
Seeking Advice on Data Lakehouse Architecture with Databricks
I'm currently designing a data lakehouse architecture using Databricks and have a few questions. What are the best practices for efficiently ingesting both batch and streaming data into Delta Lake? Any recommended tools or approaches?
- 2034 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @joshbuttler,I think the best way is to use auto loader, which provides a highly efficient way to incrementally process new data, while also guaranteeing each file is processed exactly once.It supports ingestion in a batch mode (Trigger.Available...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
