- 1235 Views
- 1 replies
- 0 kudos
Best option for parallel processing
I faced some challenges in my projects related to parallel processing in Databricks. In many cases, the issue was not the volume of data itself, but the overall execution time. I was processing a relatively small number of objects, but each object re...
- 1235 Views
- 1 replies
- 0 kudos
- 0 kudos
The Driver was the bottleneck in the Thread Pool approach. By moving to Serverless Workflows, you can shift the orchestration weight to the Databricks Control Plane.Eliminate Driver Saturation: Serverless compute for Workflows natively handles task d...
- 0 kudos
- 2234 Views
- 4 replies
- 1 kudos
Resolved! Ingest data from REST endpoint into Databricks
Hello,I'm looking for the best option to retrieve between 1-1.5TB of data per day from a REST API into Databricks.Thank you,Rodrigo Escamilla
- 2234 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi Rodrigo,One simple approach I’ve used is calling the REST API directly from a Databricks notebook using standard Python libraries—no extra setup or tools required.The idea is to keep it minimal: generate the API signature, call the endpoint, and l...
- 1 kudos
- 2151 Views
- 5 replies
- 2 kudos
Resolved! Lakeflow Connect - SQL Server - Database Setup step keeps failing
Hello,I am trying to ingest data from an Azure SQL Database using lakeflow connect.- I'm using a service principle for authentication (created the login and user in the DB am trying to ingest)- The utility script was executed by a DB owner=== Install...
- 2151 Views
- 5 replies
- 2 kudos
- 2 kudos
We figured out the issue finally! We checked the database sql audit logs and noticed that there was a particular query that was taking too long (4min) for the ingestion user. This was causing a timeout. This query is very simple and takes usually a c...
- 2 kudos
- 626 Views
- 1 replies
- 1 kudos
Resolved! Unable to see lakeflow designer option in my free edition databricks account
I am unable to see the lakeflow designer option in my databricks account. Even the previews option is not there ... Please let me know how can access that
- 626 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @ashutoshacharya, Right now, Lakeflow Designer is in Public Preview, and it isn’t fully rolled out to Databricks Free Edition yet, which is why you don’t see it in the UI or under Previews. On full (paid or trial) workspaces, a workspace admin can...
- 1 kudos
- 459 Views
- 1 replies
- 0 kudos
What the maximum size to read using dbutils.fs.head
Hi,What the maximum size to read using dbutils.fs.head()?is there a limit? because AI says 10MB and I couldn't find useful info in documentations, while I tried in the actual one and it was only limited by the driver memory.Thanks in advance.
- 459 Views
- 1 replies
- 0 kudos
- 0 kudos
dbutils.fs.head() itself does not have a documented hard cap like 10 MB. From the official dbutils reference, the signature is: dbutils.fs.head(file: String, max_bytes: int = 65536): String “Returns up to the specified maximum number of bytes in t...
- 0 kudos
- 646 Views
- 2 replies
- 1 kudos
Resolved! Data in Unity Catalog that can't be previewed
This is a small deficiency, but a fix would be nice to have.For a long time now, the Sample Data previewer in the Unity Catalog explorer has been unable to show tables that contain a certain kind of column. Instead of showing sample rows of the tabl...
- 646 Views
- 2 replies
- 1 kudos
- 1 kudos
Yes, my vector space is commonly of dimension 4000 or 8000.I don't write any dense vectors to table; can't speak to what happens previewing that type.Thanks for taking up the issue!
- 1 kudos
- 1907 Views
- 6 replies
- 7 kudos
Resolved! Managed Delta table: time travel blocked after automatic VACUUM
Hi,On a managed Delta table I get:SELECT * FROM abc VERSION AS OF 25;Error:DELTA_UNSUPPORTED_TIME_TRAVEL_BEYOND_DELETED_FILE_RETENTION_DURATION Cannot time travel beyond delta.deletedFileRetentionDuration (168 HOURS).Audit logs show VACUUM START/END...
- 1907 Views
- 6 replies
- 7 kudos
- 7 kudos
VACUUM will never delete files on the latest version even if Version 10 was not accessed or modified as it represents the current state of the table. VACUUM targets files that are not referenced by the recent version. It identifies files that were re...
- 7 kudos
- 328 Views
- 1 replies
- 0 kudos
Supporting File unrecognition in DLT Pipeline.
We have a dlt pipeline which creates some same table, which are created based on some transformation and those transformation are kept inside a function in a seperate file. and those file were used using import function.we are deploying those changes...
- 328 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Muralidharan_A, To your question about whether retry_on_failure does more than a manual refresh, the answer is yes! retry_on_failure (along with pipelines.numUpdateRetryAttempts and pipelines.maxFlowRetryAttempts) performs classified, timed retri...
- 0 kudos
- 582 Views
- 2 replies
- 1 kudos
Resolved! Jobs & Pipelines: is it possible for "Run parameters" to display a value generated in code?
Hi. I'm testing out the "Run parameters" you see in Jobs & Pipelines. As far as I know, this value is set manually by "Job parameters" on the right side bar. Can I set the value within code though? Like if I want something dynamically generated depen...
- 582 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @397973, Interesting question and I did not know the answer. So, I ran the test you described on my own workspace. Sharing what I found in case it saves you time. The short answer is that the task values won't populate the Run parameters column. V...
- 1 kudos
- 656 Views
- 4 replies
- 0 kudos
Driver memory utilization grows continuously during job
I have a batch job that runs thousands of Deep Clone commands, it uses a ForEach task to run multiple Deep Clones in parallel. It was taking a very long time and I realized that the Driver was the main culprit since it was using up all of its memory ...
- 656 Views
- 4 replies
- 0 kudos
- 0 kudos
You’re seeing (a monotonic / stair‑step climb in driver RAM over thousands of DEEP CLONE statements) is a very common pattern when the driver is not “holding data”, but holding metadata, query artifacts, and per‑command state that accumulates faster ...
- 0 kudos
- 834 Views
- 4 replies
- 1 kudos
Resolved! Unable to View Tables While Setting Up PostgreSQL CDC via Lakeflow Connect
Dear Experts,I have a requirement to implement PostgreSQL CDC using Databricks Lakeflow Connect. While setting up the tables, I am unable to see the list of available tables, even though the connection settings appear to be correct.Could you please s...
- 834 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @harisrinivasay, @szymon_dybczak is correct. You must enter the database name. Lakeflow Connect can only connect to and query that database, and list the schemas and tables if you provide the correct name. If the name is incorrect or if you don’t ...
- 1 kudos
- 360 Views
- 0 replies
- 0 kudos
Stop Refreshing. Start Querying.
How Databricks Metric Views Are Replacing Power BI Import Models — and What Your Team Needs to Do About It.IntroductionPower BI Import models work — until scheduled refreshes, size limits, and governance sprawl become too big to ignore. Databricks Un...
- 360 Views
- 0 replies
- 0 kudos
- 774 Views
- 4 replies
- 0 kudos
Resolved! Sharepoint Connector Site Limitation
Hey All!We are trying out the Beta connector for SharePoint and found that the connector will not work at the root-level site. Is there a reason for this limitation. It is unfortunately a hard blocker for us to use the native connector. MUST_START...
- 774 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi Scott, Just asking our product team the quesiton. By the root level site do you mean content that is stored on the root level site? Or do you mean everything across your root tennant. ie you want to ingest all files across your tennant in a single...
- 0 kudos
- 1859 Views
- 2 replies
- 4 kudos
Resolved! How Deep clone works
Hi,For DR purposes, we have setup Deep clone using delta share. Each time the deep clone job runs, it executes the query create or replace table {schema}.{table} deep clone {delta_share}.{schema}.{table} The 1st time the job ran, it took few hours to...
- 1859 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @DineshOjha, Deep clone is incremental, not a full re-copy every time, even when you use CREATE OR REPLACE TABLE … DEEP CLONE … against a Delta Sharing table. On the first DEEP CLONE, Databricks must read the entire source table (via Delta Sharing...
- 4 kudos
- 790 Views
- 2 replies
- 1 kudos
Resolved! Environment-Specific Schemas in SQL Files
Hi Databricks Community,We are looking for the recommended best practice for managing environment-specific configurations (catalogs and schemas) within our SQL files when using Databricks Asset Bundles.Our Setup:We use Databricks Asset Bundles to pac...
- 790 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you LR, we tried this approach and it worked but for this we would need to define the variable at multiple places, soch as the databricks.yml, the jobs file and the main code.So we decided to put all these variables in a single config file whic...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
