- 33 Views
- 2 replies
- 1 kudos
Resolved! org.apache.spark.SparkRuntimeException: [UDF_USER_CODE_ERROR.GENERIC]
Hi EveryoneI am writing a small function, with spark read from a csv and spark write into a table. I could execute this function within the notebook. But, when I register the same function as a unity catalog function and calling it from Playground, i...
- 33 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @GiriSreerangam, You cannot use a Unity Catalog user-defined function (UDF) in Databricks to perform Spark read from a CSV and write to a table. Unity Catalog Python UDFs execute in a secure, isolated environment without access to the file system ...
- 1 kudos
- 5 Views
- 0 replies
- 0 kudos
Pull workspace url and workspace name using databricks-sdk / programaticaly in notebook
1. How could I pull workspace url (https://adb-XXXXX.XX.....net) 2. How could I get workspace name visible in top right corner.I know that easies solution is dbutils.notebook.entry_point.... browserHostName but unfortunetly it is not working in job c...
- 5 Views
- 0 replies
- 0 kudos
- 8 Views
- 0 replies
- 0 kudos
Global Parameter at the Pipeline level in Lakeflow Job
Hi ,any work around or Databricks can enable global parameters feature at the pipeline level in the lakeflow job.Currently I am working on migrating adf pipeline schedule set up to lakeflow job.
- 8 Views
- 0 replies
- 0 kudos
- 19 Views
- 0 replies
- 0 kudos
Drop Delta Log seems not to be working
I have a delta table where I set the following propertylogRetentionDuration: "interval 1 days"I was doing some table operations and see in the _delta_log folder files such as00000000000000000000.json 00000000000000000001.json 00000000000000000002.js...
- 19 Views
- 0 replies
- 0 kudos
- 25 Views
- 1 replies
- 1 kudos
dbt Cloud + Databricks SQL Warehouse with microbatching (48h lookback) — intermittent failures
Hey everyone,I’m currently running hourly dbt Cloud job (27 models with 8 threads) on a Databricks SQL Warehouse using the dbt microbatch approach, with a 48-hour lookback window.But I’m running into some recurring issues:Jobs failing intermittentlyO...
- 25 Views
- 1 replies
- 1 kudos
- 1 kudos
Here are few options you can try and see if it resolves your issue.1. SQL Warehouse TuningUse Serverless SQL Warehouse with Photon for faster spin-up and query execution. [docs.getdbt.com]Size Appropriately: Start with Medium or Large, and enable au...
- 1 kudos
- 71 Views
- 2 replies
- 0 kudos
Inquiry on GraphFrame Library Upgrade Timeline for Databricks Runtime for Machine Learning
Thanks for the Databricks community and maintaining such a valuable platform.I would like to inquire if there is a planned timeline for upgrading the GraphFrame library. We’ve noticed that the latest release on GitHub is v0.9.3, while the Databricks ...
- 71 Views
- 2 replies
- 0 kudos
- 0 kudos
I don't see any dates for it. But you can try this work around.If you need access to the latest GraphFrames featuresManual Installation: You can manually install the GraphFrames v0.9.3 JAR in your cluster.Download from GraphFrames GitHub ReleasesUplo...
- 0 kudos
- 1866 Views
- 2 replies
- 2 kudos
Resolved! I can't use my own .whl package in Databricks app with databricks asset bundles
I am building a databricks app using databricks asset bundles. I need to use a helpers packages that i built as an artifact and using in other resources outside the app. The only way to use it is to have the built package inside the app source code f...
- 1866 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Oumeima , One potential way around this is to upload the wheel file into a Unity Catalog volume or workspace file. For the volume route, reference it directly in your app’s requirements.txt using an absolute /Volumes/<catalog>/<schema>/<volume>/....
- 2 kudos
- 35 Views
- 2 replies
- 0 kudos
Databricks CLI binding storage credential to a workspace
In the documentation from Databricks it says to run the below for binding a storage credential to a workspace (after already completing step 1 to update the `isolation-mode` to be `ISOLATED`): databricks workspace-bindings update-bindings storage-cre...
- 35 Views
- 2 replies
- 0 kudos
- 0 kudos
This appears to be a documentation inconsistency. The CLI implementation seems to: 1. Require binding_type to be explicitly specified (contradicting the docs) 2. Require it to be placed within each workspace object, not as a top-level parameter ...
- 0 kudos
- 7885 Views
- 7 replies
- 3 kudos
Limiting concurrent tasks in a job
I have a job with > 10 tasks in it that interacts with an external system outside of databricks. At the moment that external system cannot handle more than 3 of the tasks executing concurrently. How can I limit the number of tasks that concurrently...
- 7885 Views
- 7 replies
- 3 kudos
- 3 kudos
You do something like:E1 E4E2 Z E5 Z ...E3 E6So Z does not actually do anything it's just a funnel that waits for the 3 tasks at a time to complete ...
- 3 kudos
- 584 Views
- 3 replies
- 2 kudos
Resolved! Lakeflow Connect - Postgres connector
Hi, I was wondering when is the ETA of the LF Connector for PostgreSQL (even in public/private preview)?
- 584 Views
- 3 replies
- 2 kudos
- 2 kudos
Ask your workspace administrator if they disabled access to it. Louis
- 2 kudos
- 107 Views
- 4 replies
- 2 kudos
Resolved! Error trying to edit Job Cluster via Databricks CLI
Good Day all,After having issues with Cloud resources allocated to Lakeflow jobs and Gateways I am trying to apply a policy to the cluster that is allocated to the Job. I am very new to a lot of the databricks platform and the administration so all h...
- 107 Views
- 4 replies
- 2 kudos
- 2 kudos
Thank you so much Louis,This has resolved all of our issues! Really appreciate the help.
- 2 kudos
- 129 Views
- 2 replies
- 0 kudos
Ingesting Data from Event Hubs via Kafka API with Serverless Compute
Hi!I'm currently working on ingesting log data from Azure Event Hubs into Databricks. Initially, I was using a managed Databricks workspace, which couldn't access Event Hubs over a private endpoint. To resolve this, our DevOps team provisioned a VNet...
- 129 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @mark_ott , Thanks for your response.I followed below mentioned documentation to configure private connectivity with Azure resources and was able to ingest logs using serverless compute. Having NCC set up is the key here.https://learn.microsoft.co...
- 0 kudos
- 120 Views
- 2 replies
- 0 kudos
Unable to create workspace
Hi Team,we have challenge creating workspace in data bricks account created on top of aws.below are the details:Databricks account name : saichaitanya.vaddadhi@healthfirsttech.com's LakehouseAWS Account id : 720016114009databricks id: 1ee8765f-b472-4...
- 120 Views
- 2 replies
- 0 kudos
- 0 kudos
@hf-databricks there's a quickstart guide for creating a workspace with AWS: https://docs.databricks.com/aws/en/admin/workspace/quick-start There's a list of requirements:There's more options for creating workspaces. Above, I just listed the recommen...
- 0 kudos
- 2124 Views
- 3 replies
- 1 kudos
Workspace allows dbf path to install in Databricks 16.4 LTS
Feature: Library installation using requirements.txt on DB Runtime 16.4 LTSAffected Areas: Workspace isolation, Library ManagementSteps to Reproduce:Upload a wheel file to dbfPut the requirements.txt file in the Workspace and put dbfs path in require...
- 2124 Views
- 3 replies
- 1 kudos
- 1 kudos
I would like to know if the workspace isolation is a priority, and only Databricks 14.3 and lower allow installation via DBFS.Why should the requirements.txt allow you to install libraries or packages via dbfs path?Could someone please explain why th...
- 1 kudos
- 58 Views
- 0 replies
- 0 kudos
On Prem MS sql to Azure Databricks
Hi allI need to ingest data from on prem MS sql tables using Databricks to Azure Cloud. For the ingest, previously I used notebooks, jdbc connectors, read sql tables and write in unity catalog tables. Now, I want to experiment Databricks connectors f...
- 58 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 482 | |
| 349 | |
| 287 |
