- 1854 Views
- 2 replies
- 2 kudos
Resolved! I can't use my own .whl package in Databricks app with databricks asset bundles
I am building a databricks app using databricks asset bundles. I need to use a helpers packages that i built as an artifact and using in other resources outside the app. The only way to use it is to have the built package inside the app source code f...
- 1854 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Oumeima , One potential way around this is to upload the wheel file into a Unity Catalog volume or workspace file. For the volume route, reference it directly in your app’s requirements.txt using an absolute /Volumes/<catalog>/<schema>/<volume>/....
- 2 kudos
- 32 Views
- 2 replies
- 0 kudos
Databricks CLI binding storage credential to a workspace
In the documentation from Databricks it says to run the below for binding a storage credential to a workspace (after already completing step 1 to update the `isolation-mode` to be `ISOLATED`): databricks workspace-bindings update-bindings storage-cre...
- 32 Views
- 2 replies
- 0 kudos
- 0 kudos
This appears to be a documentation inconsistency. The CLI implementation seems to: 1. Require binding_type to be explicitly specified (contradicting the docs) 2. Require it to be placed within each workspace object, not as a top-level parameter ...
- 0 kudos
- 22 Views
- 1 replies
- 0 kudos
org.apache.spark.SparkRuntimeException: [UDF_USER_CODE_ERROR.GENERIC]
Hi EveryoneI am writing a small function, with spark read from a csv and spark write into a table. I could execute this function within the notebook. But, when I register the same function as a unity catalog function and calling it from Playground, i...
- 22 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @GiriSreerangam, You cannot use a Unity Catalog user-defined function (UDF) in Databricks to perform Spark read from a CSV and write to a table. Unity Catalog Python UDFs execute in a secure, isolated environment without access to the file system ...
- 0 kudos
- 7877 Views
- 7 replies
- 3 kudos
Limiting concurrent tasks in a job
I have a job with > 10 tasks in it that interacts with an external system outside of databricks. At the moment that external system cannot handle more than 3 of the tasks executing concurrently. How can I limit the number of tasks that concurrently...
- 7877 Views
- 7 replies
- 3 kudos
- 3 kudos
You do something like:E1 E4E2 Z E5 Z ...E3 E6So Z does not actually do anything it's just a funnel that waits for the 3 tasks at a time to complete ...
- 3 kudos
- 579 Views
- 3 replies
- 2 kudos
Resolved! Lakeflow Connect - Postgres connector
Hi, I was wondering when is the ETA of the LF Connector for PostgreSQL (even in public/private preview)?
- 579 Views
- 3 replies
- 2 kudos
- 2 kudos
Ask your workspace administrator if they disabled access to it. Louis
- 2 kudos
- 107 Views
- 4 replies
- 2 kudos
Resolved! Error trying to edit Job Cluster via Databricks CLI
Good Day all,After having issues with Cloud resources allocated to Lakeflow jobs and Gateways I am trying to apply a policy to the cluster that is allocated to the Job. I am very new to a lot of the databricks platform and the administration so all h...
- 107 Views
- 4 replies
- 2 kudos
- 2 kudos
Thank you so much Louis,This has resolved all of our issues! Really appreciate the help.
- 2 kudos
- 68 Views
- 1 replies
- 0 kudos
Inquiry on GraphFrame Library Upgrade Timeline for Databricks Runtime for Machine Learning
Thanks for the Databricks community and maintaining such a valuable platform.I would like to inquire if there is a planned timeline for upgrading the GraphFrame library. We’ve noticed that the latest release on GitHub is v0.9.3, while the Databricks ...
- 68 Views
- 1 replies
- 0 kudos
- 0 kudos
For PySpark.
- 0 kudos
- 21 Views
- 0 replies
- 1 kudos
dbt Cloud + Databricks SQL Warehouse with microbatching (48h lookback) — intermittent failures
Hey everyone,I’m currently running hourly dbt Cloud job (27 models with 8 threads) on a Databricks SQL Warehouse using the dbt microbatch approach, with a 48-hour lookback window.But I’m running into some recurring issues:Jobs failing intermittentlyO...
- 21 Views
- 0 replies
- 1 kudos
- 128 Views
- 2 replies
- 0 kudos
Ingesting Data from Event Hubs via Kafka API with Serverless Compute
Hi!I'm currently working on ingesting log data from Azure Event Hubs into Databricks. Initially, I was using a managed Databricks workspace, which couldn't access Event Hubs over a private endpoint. To resolve this, our DevOps team provisioned a VNet...
- 128 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @mark_ott , Thanks for your response.I followed below mentioned documentation to configure private connectivity with Azure resources and was able to ingest logs using serverless compute. Having NCC set up is the key here.https://learn.microsoft.co...
- 0 kudos
- 120 Views
- 2 replies
- 0 kudos
Unable to create workspace
Hi Team,we have challenge creating workspace in data bricks account created on top of aws.below are the details:Databricks account name : saichaitanya.vaddadhi@healthfirsttech.com's LakehouseAWS Account id : 720016114009databricks id: 1ee8765f-b472-4...
- 120 Views
- 2 replies
- 0 kudos
- 0 kudos
@hf-databricks there's a quickstart guide for creating a workspace with AWS: https://docs.databricks.com/aws/en/admin/workspace/quick-start There's a list of requirements:There's more options for creating workspaces. Above, I just listed the recommen...
- 0 kudos
- 2119 Views
- 3 replies
- 1 kudos
Workspace allows dbf path to install in Databricks 16.4 LTS
Feature: Library installation using requirements.txt on DB Runtime 16.4 LTSAffected Areas: Workspace isolation, Library ManagementSteps to Reproduce:Upload a wheel file to dbfPut the requirements.txt file in the Workspace and put dbfs path in require...
- 2119 Views
- 3 replies
- 1 kudos
- 1 kudos
I would like to know if the workspace isolation is a priority, and only Databricks 14.3 and lower allow installation via DBFS.Why should the requirements.txt allow you to install libraries or packages via dbfs path?Could someone please explain why th...
- 1 kudos
- 53 Views
- 0 replies
- 0 kudos
On Prem MS sql to Azure Databricks
Hi allI need to ingest data from on prem MS sql tables using Databricks to Azure Cloud. For the ingest, previously I used notebooks, jdbc connectors, read sql tables and write in unity catalog tables. Now, I want to experiment Databricks connectors f...
- 53 Views
- 0 replies
- 0 kudos
- 4039 Views
- 6 replies
- 3 kudos
Resolved! What are powerfull data quality tools/libraries to build data quality framework in Databricks ?
Dear Community Experts,I need your expert advice and suggestions on development of data quality framework. What are powerfull data quality tools or libraries are good to go for development of data quality framework in Databricks ? Please guide team.R...
- 4039 Views
- 6 replies
- 3 kudos
- 3 kudos
Consider our open-source data quality tool, DataOps Data Quality TestGen. Our goal is to help data teams automatically generate 80% of the data tests they need with just a few clicks, while offering a nice UI for collaborating on the remaining 20% th...
- 3 kudos
- 2910 Views
- 1 replies
- 1 kudos
how to use azure one lake in aws databricks unity catalog
i'm trying to connect azure one lake in aws databricks unity catalog but i'm not able to storage credential, since it's currently allowing s3 location only but in hive catalog i'm able to connect to one lake but not in unity.
- 2910 Views
- 1 replies
- 1 kudos
- 1 kudos
Azure OneLake cannot be directly connected or credentialed in AWS Databricks Unity Catalog at this time, because AWS Databricks Unity Catalog supports only storage credentials for S3 and a select few options (like Cloudflare R2), rather than Azure-ba...
- 1 kudos
- 3151 Views
- 1 replies
- 0 kudos
GitLab Integration
Hello I'm struggling with Gitlab integration in databricks.I've got jobs that run on a daily basis, pointing directly to .py files in my repo. In order to do so, my gitlab account is linked to databricks with a PAT expiring within a month.But every o...
- 3151 Views
- 1 replies
- 0 kudos
- 0 kudos
The error you are experiencing—"UNAUTHENTICATED: Invalid Git provider Personal Access Token credentials for repository URL"—is a common pain point when integrating GitLab repos with Databricks using Personal Access Tokens (PATs), especially for sched...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 482 | |
| 349 | |
| 287 |
