- 1356 Views
- 4 replies
- 1 kudos
Resolved! Why is spark creating 5 jobs and 200 tasks?
I am trying to read 1000 small csv files each 30 kb size which are stored in databricks volume. Below is the query i am doing:df=spark.read.csv.options(header=true).load('/path')df.collect() Why is it creating 5 jobs? Why 1-3 jobs have 200 tasks,4 ha...
- 1356 Views
- 4 replies
- 1 kudos
- 1 kudos
@mordex - yes, Spark caps the parallelism for file listing at 200 tasks, regardless of whether you have 1,000 or 10,000 files. it is controlled by spark.sql.sources.parallelPartitionDiscovery.parallelism. Run below command to get value of it. spark.c...
- 1 kudos
- 908 Views
- 2 replies
- 0 kudos
Declarative Pipeline Re-Deployment and existing managed tables exception
Hi,I am facing a issue regarding re deployment of declarative pipeline using asset bundle. On first deployment, I am able to run the pipeline successfully. On execution, pipeline, as expected create tables. However, when I try to re-deploy the pipeli...
- 908 Views
- 2 replies
- 0 kudos
- 0 kudos
Managed tables are “owned” by a DLT pipeline. Re-deploying a pipeline that references the same managed tables will fail unless you either:Drop the existing tables firstUse external tables that are not owned by DLTUse a separate development schema/pip...
- 0 kudos
- 25480 Views
- 5 replies
- 6 kudos
What is the difference between OPTIMIZE and Auto Optimize?
I see that Delta Lake has an OPTIMIZE command and also table properties for Auto Optimize. What are the differences between these and when should I use one over the other?
- 25480 Views
- 5 replies
- 6 kudos
- 6 kudos
Auto Optimize = automatically reduces small files during writes. Best for ongoing ETL.OPTIMIZE = manual compaction + Z-ORDER for improving performance on existing data.They are complementary, not competing. Most teams use Auto Optimize for daily inge...
- 6 kudos
- 2992 Views
- 5 replies
- 7 kudos
Resolved! Row tracking in Delta tables
What exactly is row tracking and why should we use it for our delta tables? Could you explain with an example how it works internally and is it mandatory to use?
- 2992 Views
- 5 replies
- 7 kudos
- 7 kudos
Row tracking gives each Delta row a stable internal ID, so Delta can track inserts/updates/deletes across table versions—even when files are rewritten or compacted.Suppose we have a Delta table:id value 1A2BWhen row tracking is enabled, Delta Lake st...
- 7 kudos
- 1528 Views
- 1 replies
- 1 kudos
Resolved! mongodb connector duplicate writes
Hi everyone,Has anyone run into this issue? I’m using the MongoDB Spark Connector on Databricks to expose data from Delta Lake to MongoDB. My workflow is:overwrite the collection (very fast),then create the indexes.Occasionally, I’m seeing duplicates...
- 1528 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Aziz,What you’re seeing is an expected behaviour when combining Spark retries with non-idempotent writes.Spark’s write path is task-based and fault-tolerant. If a task fails part-way through writing to MongoDB, Spark will retry that task.From Spar...
- 1 kudos
- 2542 Views
- 3 replies
- 2 kudos
AI
At Chevron we actively use Databricks to provide answers to business users. It was extremely interesting to see the use LakeHouseIQ initiatives as it can expedite how fast our users can receive their answers/reports. Is there any documentation that I...
- 2542 Views
- 3 replies
- 2 kudos
- 2 kudos
Guys, this thread was created in 2023. And the user who created it was last seen in 2023. I think there’s no point in resurrecting this thread
- 2 kudos
- 1948 Views
- 4 replies
- 1 kudos
"ai_parse_document()" is not a full OCR engine ? It's not extracting text from high quality image
I used "ai_parse_document()" to parse a PNG file that contains cat images and text. From the image, I wanted to extract all the cat names, but the response returned nothing. It seems that "ai_parse_document()" does not support rich image extraction....
- 1948 Views
- 4 replies
- 1 kudos
- 1 kudos
@szymon_dybczak - yes, as it relies on AI models, there are chances of missing few cases due to non-deterministic nature of it. I have used it with vast number of PDFs in anger and it has worked pretty well in all those cases. Have not tried with PNG...
- 1 kudos
- 17386 Views
- 5 replies
- 8 kudos
Resolved! Monitoring Azure Databricks in an Azure Log Analytics Workspace
Does anyone have experience with the mspnp/spark-monitoring library ?Is this best practice, or are there better ways to monitor a Databricks Cluster?
- 17386 Views
- 5 replies
- 8 kudos
- 8 kudos
Interesting that Microsoft deleted this project. Was there any announcement as to when, why, and what to do now?
- 8 kudos
- 4230 Views
- 4 replies
- 1 kudos
Resolved! Issue with Logging Spark Events to LogAnalytics after Upgrading to Databricks 11.3 LTS
We have recently been in the process of upgrading our Databricks clusters to version 11.3 LTS. As part of this upgrade, we have been working on integrating the logging of Spark events to LogAnalytics using the repository available at https://github.c...
- 4230 Views
- 4 replies
- 1 kudos
- 1 kudos
Anyone knows why was this repository deleted?https://github.com/mspnp/spark-monitoring
- 1 kudos
- 11643 Views
- 5 replies
- 5 kudos
Resolved! What are the options for "spark_conf.spark.databricks.cluster.profile"?
Hey guys, I'm trying to find what are the options we can pass to spark_conf.spark.databricks.cluster.profileI know looking around that some of the available configs are singleNode and serverless, but there are others?Where is the documentation of it?...
- 11643 Views
- 5 replies
- 5 kudos
- 5 kudos
Looking internally, I was able to find the following: For single node mode: the config should be set to 'singleNode' For standard mode: the config should NOT be set to 'singleNode' For serverless mode: the config should be set to 'serverless' So, in ...
- 5 kudos
- 2740 Views
- 3 replies
- 3 kudos
Resolved! Databricks AutoLoader IncrementalListing mode changes
Hi everyone!I wan investigating how Databricks AutoLoader IncrementalListing mode changes will impact my current autoloader streams. Currently all of them are set to cloudFiles.useIncrementalListing: auto. So I wanted to check if any of streams is ac...
- 2740 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @deng_dev ,When cloudFiles.useIncrementalListing is set to auto, Auto Loader automatically detects whether a given directory is applicable for incremental listing by checking and comparing file paths of previously completed directory listings.To e...
- 3 kudos
- 8442 Views
- 2 replies
- 2 kudos
Resolved! Establishing a Connection between ADLS Gen2, Databricks and ADF In Microsoft Azure
Hello,May, Someone please help me with establishing connection between ADLS Gen2, Databricks and ADF, full steps if possibble. Do I need to route through key-vault, this is i am first time doing in production,.May somebody please share detailed step ...
- 8442 Views
- 2 replies
- 2 kudos
- 2 kudos
For a production environment (ADF as orchestrator, ADLS Gen2 as storage, Databricks for PySpark transformations), follow Microsoft-recommended best practices:Databricks → ADLS Gen2: Use Unity Catalog with Azure Managed Identity (via Access Connector)...
- 2 kudos
- 9154 Views
- 7 replies
- 5 kudos
Resolved! ADE 2.1 Unable to run Classroom-Setup-3.1
%run ../Includes/Classroom-Setup-3.1After running the above command, getting this error message, attached error screenshot.AnalysisException: You are trying to read a Delta table `spark_catalog`.`dbacademy_sumit_s_jha_hk_ey_com_adewd_3_1`.`date_looku...
- 9154 Views
- 7 replies
- 5 kudos
- 994 Views
- 2 replies
- 1 kudos
Resolved! Streamed DLT Pipeline using a lookup table
Hi, I need to join three streams/streamed data sets in a DLT pipeline. I am reading from a Kinesis data stream a sequence of events per group key. The logically first of the events per group contains a marker which determines whether that group is re...
- 994 Views
- 2 replies
- 1 kudos
- 1 kudos
hi @mark_ott , thank you for your help. I have a follow up question regarding data completeness and out of order processing. I have decided to go with the delta table option since super low latency is not an issue and since this option has (seemingly...
- 1 kudos
- 1391 Views
- 1 replies
- 0 kudos
Resolved! Delta live tables upsert logic without apply changes or autocdc logic
i want to create delta live tables which should be streaming and i want to use the manual upsert logic without using the apply changes api or autocdc api . how can i do it
- 1391 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @hidden , Creating Streaming Delta Live Tables with Manual Upsert Logic Let’s dig in… this question comes up a lot when folks want upsert behavior in DLT but aren’t using APPLY CHANGES or Auto-CDC. The short version: DLT doesn’t let you drop...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
