- 42851 Views
- 26 replies
- 14 kudos
Resolved! How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?
Databricks Repos best-practices recommend using the Repos REST API to update a repo via your git provider. The REST API requires authentication, which can be done one of two ways:A user / personal access tokenA service principal access tokenUsing a u...
- 42851 Views
- 26 replies
- 14 kudos
- 14 kudos
For anyone coming here in the future, this should explain it: https://docs.databricks.com/aws/en/repos/ci-cd-techniques-with-repos#authorize-a-service-principal-to-access-git-foldersBasically:1. Go to your service in settings -> identity and access -...
- 14 kudos
- 7268 Views
- 5 replies
- 3 kudos
Resolved! Generating Personal Access Token to service principle databricks cli
Hi I am having issues generating personal access token to my service principle.I followed the steps from here my `~/.databrickscfg` has the following```[my-profile-name]host = <account-console-url>account_id = <account-id>azure_tenant_id = <azure-ser...
- 7268 Views
- 5 replies
- 3 kudos
- 3 kudos
I want something similar, to use a service principal token instead of a PAT, have you ever done this?https://community.databricks.com/t5/administration-architecture/use-a-service-principal-token-instead-of-personal-access-token/m-p/91629
- 3 kudos
- 241 Views
- 1 replies
- 0 kudos
Facing error which submitting lab assessment of Delivery Specialization: UC Upgrade
Hi,I attended the lab assessment of Delivery Specialization: UC Upgrade and while submitting my answers the grade is not getting updated successfully due to which the status is being showed as failed, although all the checkpoints got validated succes...
- 241 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ushnish_18, Can you share the user ID or email you used to access the lab so the team can take a look? Alternatively, you can send all the details, including screenshots, here for the team to investigate in detail and guide you appropriately. Tha...
- 0 kudos
- 351 Views
- 2 replies
- 0 kudos
how to pass secrets keys using a spark_python_task
Hello community,I was searching a way to pass secrets to spark_python_task. Using a notebook file is easy, it's only to use dbutils.secrets.get(...) but how to do the same thing using a spark_python_task set using serveless compute?Kind regards,
- 351 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @jeremy98. To securely access secrets in a spark_python_task with serverless compute in Databricks... first create a secret scope and add secrets (refer to this article). Then, pass secrets by injecting them into environment variables in the job c...
- 0 kudos
- 10728 Views
- 7 replies
- 3 kudos
TesseractNotFoundError
TesseractNotFoundError: tesseract is not installed or it's not in your PATH. See README file for more information. in databricks
- 10728 Views
- 7 replies
- 3 kudos
- 864 Views
- 1 replies
- 0 kudos
Acess to s3 in aws
My Problem:My Databricks workspace (Serverless Compute) is in AWS Account A, but my S3 bucket is in AWS Account B.Works in Shared Compute because i am manually setting access_key and secret_key.Does NOT work in Serverless Compute
- 864 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @HarryRichard08 ,I came across a similar thread of yours. Were you able to find a resolution for this?
- 0 kudos
- 1706 Views
- 10 replies
- 5 kudos
Performance issue with Spark SQL when working with data from Unity Catalog
We're encountering a performance issue with Spark SQL when working with data from Unity Catalog. Specifically, when I use Spark to read data from a Unity Catalog partition created by DLT and then create a view, the executor retrieval is very slow. ...
- 1706 Views
- 10 replies
- 5 kudos
- 5 kudos
you can read the physical parquet files with spark.read.parquet().The trick is to know which files are the current ones.
- 5 kudos
- 516 Views
- 3 replies
- 2 kudos
Spark DataFrame Checkpoint
Good morning,I am having a difficulty when trying to checkpoint a PySpark DataFrame.The DataFrame is not involved in a DLT pipeline so I am using the df.checkpoint(eager=True) command, to truncate the logical plan of df and materialize it as files wi...
- 516 Views
- 3 replies
- 2 kudos
- 1253 Views
- 1 replies
- 0 kudos
Generating Multiple Excels for SQL Query
Hello,I am getting "OSError: Errno 95: Operation not supported for the code below. I have 'openpyxl 3.1.5' installed on the cluster and have imported all required modules. I am sure this is something small, but I can't put my finger on it why this is...
- 1253 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @libpekin, is filepath pointing to a DBFS path? Because writing directly to DBFS paths using to_excel() is not supported due to DBFS limitations with certain file operations, especially random writes.As a workaround, first save the Excel file to a...
- 0 kudos
- 840 Views
- 0 replies
- 0 kudos
Hydra configuration and job parameters of DABs
Hello Community,I'm trying to create a job pipeline in Databricks that runs a spark_python_task, which executes a Python script configured with Hydra. The script's configuration file defines parameters, such as id.How can I pass this parameter at the...
- 840 Views
- 0 replies
- 0 kudos
- 526 Views
- 1 replies
- 1 kudos
Visualizations ignore Timezones
Databricks inline visualizations (bar charts / line charts) etc. ignore Timezones and always display the UTC Time on the X-Axis.
- 526 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @nielsehlers! As a workaround, you can use the from_utc_timestamp() function to convert UTC timestamps to your desired time zone before visualising:SELECT from_utc_timestamp(column_name, 'Asia/Kolkata’) AS alias_name FROM table_name;
- 1 kudos
- 687 Views
- 2 replies
- 0 kudos
Lineage not visible for table created in DLT
Hello,I've been struggling for two days with missing lineage information for the silver layer table, and I'm unsure what I'm doing incorrectly.I have a DLT pipeline with DPM public preview enabled. Data is ingested from an S3 bucket into the bronze t...
- 687 Views
- 2 replies
- 0 kudos
- 0 kudos
I'm having the same exact issue as the poster above.I'm using a newly created DLT pipeline as of April 1st 2025 (Unity catalog enabled, Serverless). I get lineage for all tables and views involved in my pipeline except for the streaming table.The st...
- 0 kudos
- 757 Views
- 4 replies
- 0 kudos
Disabled billing system schema accidentally and now I cannot enable it again
We accidentally disabled billing system table schema and now getting this error on re-enabling it:Error: billing system schema can only be enabled by Databricks.How can we resolve the same. We have not purchased any of the support contracts and hence...
- 757 Views
- 4 replies
- 0 kudos
- 419 Views
- 3 replies
- 0 kudos
databricks_error_message:REQUEST_LIMIT_EXCEEDED:
A databricks job failed unexpectedly with below error. There ware only 5 jobs running at that time and no major operations. what could be the root cause and how can we avoid this in futureCluster '0331-xxxxxx-zs8i8pcn' was terminated. Reason: INIT_SC...
- 419 Views
- 3 replies
- 0 kudos
- 0 kudos
are you generating any databricks tokens in this process? if yes there is a limit of 600.
- 0 kudos
- 454 Views
- 1 replies
- 0 kudos
Resolved! dbutls.fs.cp() fails in Runtime 16.3 Beta, when using abfss://
Hello,I am not sure if this is the right place to post this, however, I reporting what seems to me, a breaking issue with 16.3 Beta Runtime, when performing dbutils.fs.cp() operations between abfss://This is not a permissions issue — let's get that o...
- 454 Views
- 1 replies
- 0 kudos
- 0 kudos
To close off my loop: The ClassCastException has been resolved in 16.3
- 0 kudos
| User | Count |
|---|---|
| 1612 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
