- 7874 Views
- 7 replies
- 3 kudos
Limiting concurrent tasks in a job
I have a job with > 10 tasks in it that interacts with an external system outside of databricks. At the moment that external system cannot handle more than 3 of the tasks executing concurrently. How can I limit the number of tasks that concurrently...
- 7874 Views
- 7 replies
- 3 kudos
- 3 kudos
You do something like:E1 E4E2 Z E5 Z ...E3 E6So Z does not actually do anything it's just a funnel that waits for the 3 tasks at a time to complete ...
- 3 kudos
- 578 Views
- 3 replies
- 1 kudos
Resolved! Lakeflow Connect - Postgres connector
Hi, I was wondering when is the ETA of the LF Connector for PostgreSQL (even in public/private preview)?
- 578 Views
- 3 replies
- 1 kudos
- 1 kudos
Ask your workspace administrator if they disabled access to it. Louis
- 1 kudos
- 105 Views
- 4 replies
- 2 kudos
Resolved! Error trying to edit Job Cluster via Databricks CLI
Good Day all,After having issues with Cloud resources allocated to Lakeflow jobs and Gateways I am trying to apply a policy to the cluster that is allocated to the Job. I am very new to a lot of the databricks platform and the administration so all h...
- 105 Views
- 4 replies
- 2 kudos
- 2 kudos
Thank you so much Louis,This has resolved all of our issues! Really appreciate the help.
- 2 kudos
- 68 Views
- 1 replies
- 0 kudos
Inquiry on GraphFrame Library Upgrade Timeline for Databricks Runtime for Machine Learning
Thanks for the Databricks community and maintaining such a valuable platform.I would like to inquire if there is a planned timeline for upgrading the GraphFrame library. We’ve noticed that the latest release on GitHub is v0.9.3, while the Databricks ...
- 68 Views
- 1 replies
- 0 kudos
- 0 kudos
For PySpark.
- 0 kudos
- 20 Views
- 0 replies
- 1 kudos
dbt Cloud + Databricks SQL Warehouse with microbatching (48h lookback) — intermittent failures
Hey everyone,I’m currently running hourly dbt Cloud job (27 models with 8 threads) on a Databricks SQL Warehouse using the dbt microbatch approach, with a 48-hour lookback window.But I’m running into some recurring issues:Jobs failing intermittentlyO...
- 20 Views
- 0 replies
- 1 kudos
- 123 Views
- 2 replies
- 0 kudos
Ingesting Data from Event Hubs via Kafka API with Serverless Compute
Hi!I'm currently working on ingesting log data from Azure Event Hubs into Databricks. Initially, I was using a managed Databricks workspace, which couldn't access Event Hubs over a private endpoint. To resolve this, our DevOps team provisioned a VNet...
- 123 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @mark_ott , Thanks for your response.I followed below mentioned documentation to configure private connectivity with Azure resources and was able to ingest logs using serverless compute. Having NCC set up is the key here.https://learn.microsoft.co...
- 0 kudos
- 119 Views
- 2 replies
- 0 kudos
Unable to create workspace
Hi Team,we have challenge creating workspace in data bricks account created on top of aws.below are the details:Databricks account name : saichaitanya.vaddadhi@healthfirsttech.com's LakehouseAWS Account id : 720016114009databricks id: 1ee8765f-b472-4...
- 119 Views
- 2 replies
- 0 kudos
- 0 kudos
@hf-databricks there's a quickstart guide for creating a workspace with AWS: https://docs.databricks.com/aws/en/admin/workspace/quick-start There's a list of requirements:There's more options for creating workspaces. Above, I just listed the recommen...
- 0 kudos
- 2118 Views
- 3 replies
- 1 kudos
Workspace allows dbf path to install in Databricks 16.4 LTS
Feature: Library installation using requirements.txt on DB Runtime 16.4 LTSAffected Areas: Workspace isolation, Library ManagementSteps to Reproduce:Upload a wheel file to dbfPut the requirements.txt file in the Workspace and put dbfs path in require...
- 2118 Views
- 3 replies
- 1 kudos
- 1 kudos
I would like to know if the workspace isolation is a priority, and only Databricks 14.3 and lower allow installation via DBFS.Why should the requirements.txt allow you to install libraries or packages via dbfs path?Could someone please explain why th...
- 1 kudos
- 48 Views
- 0 replies
- 0 kudos
On Prem MS sql to Azure Databricks
Hi allI need to ingest data from on prem MS sql tables using Databricks to Azure Cloud. For the ingest, previously I used notebooks, jdbc connectors, read sql tables and write in unity catalog tables. Now, I want to experiment Databricks connectors f...
- 48 Views
- 0 replies
- 0 kudos
- 4037 Views
- 6 replies
- 3 kudos
Resolved! What are powerfull data quality tools/libraries to build data quality framework in Databricks ?
Dear Community Experts,I need your expert advice and suggestions on development of data quality framework. What are powerfull data quality tools or libraries are good to go for development of data quality framework in Databricks ? Please guide team.R...
- 4037 Views
- 6 replies
- 3 kudos
- 3 kudos
Consider our open-source data quality tool, DataOps Data Quality TestGen. Our goal is to help data teams automatically generate 80% of the data tests they need with just a few clicks, while offering a nice UI for collaborating on the remaining 20% th...
- 3 kudos
- 2909 Views
- 1 replies
- 1 kudos
how to use azure one lake in aws databricks unity catalog
i'm trying to connect azure one lake in aws databricks unity catalog but i'm not able to storage credential, since it's currently allowing s3 location only but in hive catalog i'm able to connect to one lake but not in unity.
- 2909 Views
- 1 replies
- 1 kudos
- 1 kudos
Azure OneLake cannot be directly connected or credentialed in AWS Databricks Unity Catalog at this time, because AWS Databricks Unity Catalog supports only storage credentials for S3 and a select few options (like Cloudflare R2), rather than Azure-ba...
- 1 kudos
- 3150 Views
- 1 replies
- 0 kudos
GitLab Integration
Hello I'm struggling with Gitlab integration in databricks.I've got jobs that run on a daily basis, pointing directly to .py files in my repo. In order to do so, my gitlab account is linked to databricks with a PAT expiring within a month.But every o...
- 3150 Views
- 1 replies
- 0 kudos
- 0 kudos
The error you are experiencing—"UNAUTHENTICATED: Invalid Git provider Personal Access Token credentials for repository URL"—is a common pain point when integrating GitLab repos with Databricks using Personal Access Tokens (PATs), especially for sched...
- 0 kudos
- 3171 Views
- 2 replies
- 0 kudos
Writing to Delta Table and retrieving back the IDs doesn't work
Hi.I have a workflow in which I write few rows into a Delta Table with auto-generated IDs. Then, I need to retrieve them back just after they're written into the table to collect those generated IDs, so I read the table and I use two columns (one is ...
- 3171 Views
- 2 replies
- 0 kudos
- 0 kudos
Your workflow issue—writing to a Delta Table, immediately reading back using a join on client_id and timestamp, but sometimes missing rows—suggests a subtle problem, likely related to timing, consistency, or column precision between your input DataFr...
- 0 kudos
- 3247 Views
- 1 replies
- 0 kudos
Slow performance in REFRESH MATERIALIZED VIEW over CTAS
Hello guys, i have some materialized views created in my databricks workspace and after 1 change in one of them, it became 3x slower (9 minutes to 30 minutes). After some debugging i found that the bottleneck process in the execution plan is one call...
- 3247 Views
- 1 replies
- 0 kudos
- 0 kudos
Materialized views in Databricks, especially those maintained via DLT (Delta Live Tables) pipelines, often have different execution patterns and optimization strategies compared to running the same SQL in a standard serverless notebook. This can lead...
- 0 kudos
- 3045 Views
- 1 replies
- 0 kudos
Best practice for unified cloud cost attribution (Databricks + Azure)?
Hi! I’m working on a FinOps initiative to improve cloud cost visibility and attribution across departments and projects in our data platform. We do tagging production workflows on department level and can get a decent view in Azure Cost Analysis by f...
- 3045 Views
- 1 replies
- 0 kudos
- 0 kudos
To achieve unified cloud cost visibility and attribution for Azure and Databricks (including SQL Serverless Warehouses), consider the following best practices and solutions: Tagging Databricks SQL Warehouses for Attribution Creating a separate SQL Wa...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 482 | |
| 349 | |
| 287 |
