- 4573 Views
- 4 replies
- 2 kudos
Convert EBCDIC (Binary) file format to ASCII
Hi Team,How can we convert EBCDIC (Binary) file format to ASCII in databricks? Do we have any libraries in Databricks?
- 4573 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi Phani1 Were you able to do that successfully ? Can you share the details and steps please. Thanks.
- 2 kudos
- 14 Views
- 1 replies
- 0 kudos
Simple integration to push data from third-party into a client's Databricks instance
Hi there, we have an industry data platform with multiple customers using it. We provide each customer with their own data every night via .csv. Some of our customers use Databricks, and import their data from us into it.We would like to offer a more...
- 14 Views
- 1 replies
- 0 kudos
- 0 kudos
You could use external volumes with a Cloudflare R2 bucket as an intermediary - you write the nightly data files to R2 (using S3-compatible API), and your customers create external volumes in their Databricks workspace pointing to their designated R2...
- 0 kudos
- 14 Views
- 0 replies
- 0 kudos
Location of spark.scheduler.allocation.file
In DBR 164.LTS, I am trying to add the following Spark config: spark.scheduler.allocation.file: file:/Workspace/init/fairscheduler.xmlBut the all purpose cluster is throwing this error Spark error: Driver down cause: com.databricks.backend.daemon.dri...
- 14 Views
- 0 replies
- 0 kudos
- 20 Views
- 0 replies
- 0 kudos
Reading empty json file in serverless gives error
I ran a databricks notebook to do incremental loads from files in raw layer to bronze layer tables. Today, I encountered a case where the delta file was empty. I tried running it manually on the serverless compute and encountered an error.df = spark....
- 20 Views
- 0 replies
- 0 kudos
- 50 Views
- 2 replies
- 1 kudos
Resolved! org.apache.spark.SparkRuntimeException: [UDF_USER_CODE_ERROR.GENERIC]
Hi EveryoneI am writing a small function, with spark read from a csv and spark write into a table. I could execute this function within the notebook. But, when I register the same function as a unity catalog function and calling it from Playground, i...
- 50 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @GiriSreerangam, You cannot use a Unity Catalog user-defined function (UDF) in Databricks to perform Spark read from a CSV and write to a table. Unity Catalog Python UDFs execute in a secure, isolated environment without access to the file system ...
- 1 kudos
- 12 Views
- 0 replies
- 0 kudos
Pull workspace url and workspace name using databricks-sdk / programaticaly in notebook
1. How could I pull workspace url (https://adb-XXXXX.XX.....net) 2. How could I get workspace name visible in top right corner.I know that easies solution is dbutils.notebook.entry_point.... browserHostName but unfortunetly it is not working in job c...
- 12 Views
- 0 replies
- 0 kudos
- 17 Views
- 0 replies
- 0 kudos
Global Parameter at the Pipeline level in Lakeflow Job
Hi ,any work around or Databricks can enable global parameters feature at the pipeline level in the lakeflow job.Currently I am working on migrating adf pipeline schedule set up to lakeflow job.
- 17 Views
- 0 replies
- 0 kudos
- 25 Views
- 0 replies
- 0 kudos
Drop Delta Log seems not to be working
I have a delta table where I set the following propertylogRetentionDuration: "interval 1 days"I was doing some table operations and see in the _delta_log folder files such as00000000000000000000.json 00000000000000000001.json 00000000000000000002.js...
- 25 Views
- 0 replies
- 0 kudos
- 30 Views
- 1 replies
- 1 kudos
dbt Cloud + Databricks SQL Warehouse with microbatching (48h lookback) — intermittent failures
Hey everyone,I’m currently running hourly dbt Cloud job (27 models with 8 threads) on a Databricks SQL Warehouse using the dbt microbatch approach, with a 48-hour lookback window.But I’m running into some recurring issues:Jobs failing intermittentlyO...
- 30 Views
- 1 replies
- 1 kudos
- 1 kudos
Here are few options you can try and see if it resolves your issue.1. SQL Warehouse TuningUse Serverless SQL Warehouse with Photon for faster spin-up and query execution. [docs.getdbt.com]Size Appropriately: Start with Medium or Large, and enable au...
- 1 kudos
- 82 Views
- 2 replies
- 0 kudos
Inquiry on GraphFrame Library Upgrade Timeline for Databricks Runtime for Machine Learning
Thanks for the Databricks community and maintaining such a valuable platform.I would like to inquire if there is a planned timeline for upgrading the GraphFrame library. We’ve noticed that the latest release on GitHub is v0.9.3, while the Databricks ...
- 82 Views
- 2 replies
- 0 kudos
- 0 kudos
I don't see any dates for it. But you can try this work around.If you need access to the latest GraphFrames featuresManual Installation: You can manually install the GraphFrames v0.9.3 JAR in your cluster.Download from GraphFrames GitHub ReleasesUplo...
- 0 kudos
- 1869 Views
- 2 replies
- 2 kudos
Resolved! I can't use my own .whl package in Databricks app with databricks asset bundles
I am building a databricks app using databricks asset bundles. I need to use a helpers packages that i built as an artifact and using in other resources outside the app. The only way to use it is to have the built package inside the app source code f...
- 1869 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Oumeima , One potential way around this is to upload the wheel file into a Unity Catalog volume or workspace file. For the volume route, reference it directly in your app’s requirements.txt using an absolute /Volumes/<catalog>/<schema>/<volume>/....
- 2 kudos
- 45 Views
- 2 replies
- 0 kudos
Databricks CLI binding storage credential to a workspace
In the documentation from Databricks it says to run the below for binding a storage credential to a workspace (after already completing step 1 to update the `isolation-mode` to be `ISOLATED`): databricks workspace-bindings update-bindings storage-cre...
- 45 Views
- 2 replies
- 0 kudos
- 0 kudos
This appears to be a documentation inconsistency. The CLI implementation seems to: 1. Require binding_type to be explicitly specified (contradicting the docs) 2. Require it to be placed within each workspace object, not as a top-level parameter ...
- 0 kudos
- 7895 Views
- 7 replies
- 3 kudos
Limiting concurrent tasks in a job
I have a job with > 10 tasks in it that interacts with an external system outside of databricks. At the moment that external system cannot handle more than 3 of the tasks executing concurrently. How can I limit the number of tasks that concurrently...
- 7895 Views
- 7 replies
- 3 kudos
- 3 kudos
You do something like:E1 E4E2 Z E5 Z ...E3 E6So Z does not actually do anything it's just a funnel that waits for the 3 tasks at a time to complete ...
- 3 kudos
- 586 Views
- 3 replies
- 2 kudos
Resolved! Lakeflow Connect - Postgres connector
Hi, I was wondering when is the ETA of the LF Connector for PostgreSQL (even in public/private preview)?
- 586 Views
- 3 replies
- 2 kudos
- 2 kudos
Ask your workspace administrator if they disabled access to it. Louis
- 2 kudos
- 114 Views
- 4 replies
- 2 kudos
Resolved! Error trying to edit Job Cluster via Databricks CLI
Good Day all,After having issues with Cloud resources allocated to Lakeflow jobs and Gateways I am trying to apply a policy to the cluster that is allocated to the Job. I am very new to a lot of the databricks platform and the administration so all h...
- 114 Views
- 4 replies
- 2 kudos
- 2 kudos
Thank you so much Louis,This has resolved all of our issues! Really appreciate the help.
- 2 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 482 | |
| 349 | |
| 287 |
