by
kDev
• New Contributor
- 5010 Views
- 4 replies
- 1 kudos
Our jobs have been running fine so far w/o any issues on a specific workspace. These jobs read data from files on Azure ADLS storage containers and dont use the hive metastore data at all.Now we attached the unity metastore to this workspace, created...
- 5010 Views
- 4 replies
- 1 kudos
Latest Reply
@Wojciech_BUK Thanks a lot for the feedback! I have a couple of questions: When you say "allow workspace clusters to access storage" - I understand when you talk about interactive cluster. In my case I was trying to trigger a Databricks Notebook/Job ...
3 More Replies
- 105 Views
- 2 replies
- 1 kudos
I have a cluster pool with max capacity. I run multiple jobs against that cluster pool.Can on-demand clusters, created within this cluster pool, be shared across multiple different jobs, at the same time?The reason I'm asking is I can see a downgrade...
- 105 Views
- 2 replies
- 1 kudos
Latest Reply
Hi @radothede,
Cluster Pools and On-Demand Clusters: In Azure Databricks, a cluster pool is a collection of idle, pre-configured clusters that can be shared among multiple users or jobs. Instead of giving each user their own dedicated cluster, you...
1 More Replies
- 57 Views
- 0 replies
- 0 kudos
Hello,I am working on a Spark job where I'm reading several tables from PostgreSQL into DataFrames as follows: df = (spark.read
.format("postgresql")
.option("query", query)
.option("host", database_host)
.option("port...
- 57 Views
- 0 replies
- 0 kudos
- 33 Views
- 0 replies
- 0 kudos
The chunk of code in questionsys.path.append(
spark.conf.get("util_path", "/Workspace/Repos/Production/loch-ness/utils/")
)
from broker_utils import extract_day_with_suffix, proper_case_address_udf, proper_case_last_name_first_udf, proper_case_ud...
- 33 Views
- 0 replies
- 0 kudos
- 38 Views
- 0 replies
- 0 kudos
As of this morning we started receiving the following error message on a Databricks job with a single Pyspark Notebook task. The job has not had any code changes in 2 months. The cluster configuration has also not changed. The last successful run of ...
- 38 Views
- 0 replies
- 0 kudos
- 46 Views
- 0 replies
- 0 kudos
Hi,We would like to use Azure Managed Identity to create mount point to read/write data from/to ADLS Gen2?We are also using following code snippet to use MSI authentication to read data from ADLS Gen2 but it is giving error,storage_account_name = "<<...
- 46 Views
- 0 replies
- 0 kudos
- 156 Views
- 2 replies
- 1 kudos
Currently, the bronze table ingests JSON files using @Dlt.table decorator on a spark.readStream functionA daily batch job does some transformation on bronze data and stores results in the silver table.New ProcessBronze still the same.A stream has bee...
- 156 Views
- 2 replies
- 1 kudos
Latest Reply
Thank you thats what I understood too. It is just nice to get validation from someone else that works with this.
1 More Replies
- 349 Views
- 2 replies
- 1 kudos
Hi everyone! I'm new to Databricks and moving my first steps with Delta Live Tables, so please forgive my inexperience. I'm building my first DLT pipeline and there's something that I can't really grasp: how to clear all the objects generated or upda...
- 349 Views
- 2 replies
- 1 kudos
Latest Reply
If you want to reprocess all the data, you can simply for a "Full Refresh" option in the DLT pipeline.
You can read more about it here: https://docs.databricks.com/en/delta-live-tables/updates.html#how-delta-live-tables-updates-tables-and-views
1 More Replies
- 43 Views
- 0 replies
- 0 kudos
I have done the below steps1. Created a databricks managed service principal2. Created a Oauth Secret3. Gave all necessary permissions to the service principalI'm trying to use this Service principal in Azure Devops to automate CI/CD. but it fails as...
- 43 Views
- 0 replies
- 0 kudos
- 51 Views
- 0 replies
- 0 kudos
I have Data Engineering Pipeline workload that run on Databricks.Job cluster has following configuration :- Worker i3.4xlarge with 122 GB memory and 16 coresDriver i3.4xlarge with 122 GB memory and 16 cores ,Min Worker -4 and Max Worker 8 We noticed...
- 51 Views
- 0 replies
- 0 kudos
- 3749 Views
- 3 replies
- 1 kudos
Hi there,I am new to Spark SQL and would like to know if it possible to reproduce the below T-SQL query in Databricks. This is a sample query, but I want to determine if a query needs to be executed or not. DECLARE
      @VariableA AS INT
,     @Vari...
- 3749 Views
- 3 replies
- 1 kudos
Latest Reply
Since you are looking for a single value back, you can use the CASE function to achieve what you need.%sqlSET var.myvarA = (SELECT 6);SET var.myvarB = (SELECT 7);SELECT CASE WHEN ${var.myvarA} = ${var.myvarB} THEN 'Equal' ELSE 'Not equal' END AS resu...
2 More Replies
- 3715 Views
- 4 replies
- 3 kudos
Parameters can be passed to Tasks and the values can be retrieved with:dbutils.widgets.get("parameter_name")More recently, we have been given the ability to add parameters to Jobs.However, the parameters cannot be retrieved like Task parameters.Quest...
- 3715 Views
- 4 replies
- 3 kudos
Latest Reply
an update to my answer: Databricks has advised us that the `dbutils.notebook.entry_point` method is not supported (could be deprecated), and the recommended way to read in a job parameter is through widgets, i.e. `dbutils.widgets.get("param_key")` (...
3 More Replies
- 1451 Views
- 5 replies
- 1 kudos
Hello,I am attempting to configure Autoloader in File Notification mode with Delta Live Tables. I configured an instance profile, but it is not working because I immediately get AWS access denied errors. This is the same issue that is referenced here...
- 1451 Views
- 5 replies
- 1 kudos
by
SreeG
• New Contributor II
- 192 Views
- 2 replies
- 0 kudos
HiI am facing issues when deploying work flows to different environment. The same works for Notebooks and Scripts, when deploying the work flows, it failed with "Authorization Failed. Your token may be expired or lack the valid scope". Anything shoul...
- 192 Views
- 2 replies
- 0 kudos
Latest Reply
Thanks, Yesh. The issue was because of a configuration parameter. After changing that, we could deploy. Thank you
1 More Replies
by
subha2
• New Contributor II
- 268 Views
- 0 replies
- 0 kudos
There are some tables under schema/database under Unity Catalog.The Notebook need to read the table parallel using loop and thread and execute the query configuredBut the sql statement is not getting executed via spark.sql() or spark.read.table().It ...
- 268 Views
- 0 replies
- 0 kudos
