- 13963 Views
- 3 replies
- 1 kudos
AWS Secrets Works In One Cluster But Not Another
Why can I use boto3 to go to secrets manager to retrieve a secret with a personal cluster but I get an error with a shared cluster?NoCredentialsError: Unable to locate credentials
- 13963 Views
- 3 replies
- 1 kudos
- 1 kudos
Hey @dbdude, I am facing the same error. Did you find a solution to access the AWS credentials on a Shared Cluster?This article describes a way of storing credentials in a Unity Catalog Volume to fetch by the Shared Cluster:https://medium.com/@amluci...
- 1 kudos
- 1669 Views
- 1 replies
- 0 kudos
ODBC PowerBI 2 commands in one query
Hello everyone,I'm trying to use the ODBC DirectQuery option in PowerBI, but I keep getting an error about another command. The SQL query works while using the SQL Editor. Do I need to change the setup of my ODBC connector?DECLARE dateFrom DATE = DA...
- 1669 Views
- 1 replies
- 0 kudos
- 13150 Views
- 3 replies
- 3 kudos
Resolved! Invalid configuration value detected for fs.azure.account.key with com.crealytics:spark-excel
I have setup my Databricks notebook to use Service Principal to access ADLS using below configuration.service_credential = dbutils.secrets.get(scope="<scope>",key="<service-credential-key>") spark.conf.set("fs.azure.account.auth.type.<storage-accou...
- 13150 Views
- 3 replies
- 3 kudos
- 3 kudos
Below is the implementation of same code in scala:spark.sparkContext.hadoopConfiguration.set("fs.azure.account.key.<accountName>.dfs.core.windows.net",<accountKey>)
- 3 kudos
- 1036 Views
- 1 replies
- 0 kudos
schedule job termination
Hi i want to terminate my databricks job daily at 11.59am, how can i achieve this in databricks
- 1036 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @prats33 You can use databricks cluster API for terminate your cluster at any specific time, create notebook for API and schedule it as databricks workflow job on job cluster at 11:59.
- 0 kudos
- 3206 Views
- 2 replies
- 0 kudos
Can we use Managed Identity to create mount point for ADLS Gen2
Hi,We would like to use Azure Managed Identity to create mount point to read/write data from/to ADLS Gen2?We are also using following code snippet to use MSI authentication to read data from ADLS Gen2 but it is giving error,storage_account_name = "<<...
- 3206 Views
- 2 replies
- 0 kudos
- 0 kudos
It seems that using User Assigned Managed Identity to read/write from ADLS Gen2 inside a notebook is not directly supported at the moment.
- 0 kudos
- 1132 Views
- 1 replies
- 0 kudos
Workflow UI broken after creating job via the api
Hi all,I'm in the progress of migrating from Databricks Azure to Databricks AWS.One part of this is migrating all our workflows which I wanted to via the /api/2.1/jobs/create api with the workflow passed via the json body. I have successfully created...
- 1132 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, many thanks for your question, as per the error message showed it was mentioning a possible timeout or network issue. As first step have you tried to open the page on another browser or using incognito mode?Also have you tried using different ...
- 0 kudos
- 1575 Views
- 1 replies
- 0 kudos
Service Principal Managed by Databricks
I have done the below steps1. Created a databricks managed service principal2. Created a Oauth Secret3. Gave all necessary permissions to the service principalI'm trying to use this Service principal in Azure Devops to automate CI/CD. but it fails as...
- 1575 Views
- 1 replies
- 0 kudos
- 0 kudos
Have you follow the steps available for service principal for CI/CD available here: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-sp
- 0 kudos
- 2101 Views
- 1 replies
- 0 kudos
Can on-demand clusters be shared across multiple jobs using cluster pool with max capacity ?
I have a cluster pool with max capacity. I run multiple jobs against that cluster pool.Can on-demand clusters, created within this cluster pool, be shared across multiple different jobs, at the same time?The reason I'm asking is I can see a downgrade...
- 2101 Views
- 1 replies
- 0 kudos
- 2122 Views
- 0 replies
- 0 kudos
Reading from one Postgres table result in several Scan JDBCRelation operations
Hello,I am working on a Spark job where I'm reading several tables from PostgreSQL into DataFrames as follows: df = (spark.read .format("postgresql") .option("query", query) .option("host", database_host) .option("port...
- 2122 Views
- 0 replies
- 0 kudos
- 1284 Views
- 0 replies
- 0 kudos
Strange Error with custom module in delta live table pipeline
The chunk of code in questionsys.path.append( spark.conf.get("util_path", "/Workspace/Repos/Production/loch-ness/utils/") ) from broker_utils import extract_day_with_suffix, proper_case_address_udf, proper_case_last_name_first_udf, proper_case_ud...
- 1284 Views
- 0 replies
- 0 kudos
- 2560 Views
- 0 replies
- 0 kudos
Job Cluster's CPU utilization goes higher than 100% few times during the workload run
I have Data Engineering Pipeline workload that run on Databricks.Job cluster has following configuration :- Worker i3.4xlarge with 122 GB memory and 16 coresDriver i3.4xlarge with 122 GB memory and 16 cores ,Min Worker -4 and Max Worker 8 We noticed...
- 2560 Views
- 0 replies
- 0 kudos
- 10179 Views
- 3 replies
- 1 kudos
Value of SQL variable in IF statement using Spark SQL
Hi there,I am new to Spark SQL and would like to know if it possible to reproduce the below T-SQL query in Databricks. This is a sample query, but I want to determine if a query needs to be executed or not. DECLARE       @VariableA AS INT ,     @Vari...
- 10179 Views
- 3 replies
- 1 kudos
- 1 kudos
Since you are looking for a single value back, you can use the CASE function to achieve what you need.%sqlSET var.myvarA = (SELECT 6);SET var.myvarB = (SELECT 7);SELECT CASE WHEN ${var.myvarA} = ${var.myvarB} THEN 'Equal' ELSE 'Not equal' END AS resu...
- 1 kudos
- 2487 Views
- 1 replies
- 0 kudos
Billing of Databricks Job clusters
Hi All,Please help me understand how the billing is calculated for using the Job cluster.Document says they are charged hourly basis, so if my job ran for 1hr 30mins then will be charged for the 30mins based on the hourly rate or it will be charged f...
- 2487 Views
- 1 replies
- 0 kudos
- 0 kudos
Job clusters consume DBUs per hour depending on the VM size. The Databricks billing happens at "per second granularity", see here. That means if you run your job for 1.5 hours, you will be charged DBUs/hour*1.5*SKU_price; accordingly, if you run your...
- 0 kudos
- 4768 Views
- 5 replies
- 0 kudos
SQL code against tables with '>' in headers suddenly failing?
Just want to post this issue we're experiencing here in case other people are facing something similar. Below is the wording of the support ticket request I've raised:SQL code that has been working is suddenly failing due to syntax errors today. Ther...
- 4768 Views
- 5 replies
- 0 kudos
- 0 kudos
The point that we've got to with this is that MS Support / Databricks have acknowledged that they did something and are working on a fix. "The issue occurred due to the regression in the recent DBR maintenance release...Our engineering team is workin...
- 0 kudos
- 5881 Views
- 6 replies
- 2 kudos
Autoingest not working with Unity Catalog in DLT pipeline
Hey Everyone,I've built a very simple pipeline with a single DLT using auto ingest, and it works, provided I don't specify the output location. When I build the same pipeline but set UC as the output location, it fails when setting up S3 notification...
- 5881 Views
- 6 replies
- 2 kudos
- 2 kudos
Hey @Babu_Krishnan I was! I had to reach out to my Databricks support engineer directly and the resolution was to add "cloudfiles.awsAccessKey" and "cloudfiles.awsSecretKey" to the params as in the screenshot below (apologies, i don't know why the sc...
- 2 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
