- 53 Views
- 0 replies
- 0 kudos
Hello,I am working on a Spark job where I'm reading several tables from PostgreSQL into DataFrames as follows: df = (spark.read
.format("postgresql")
.option("query", query)
.option("host", database_host)
.option("port...
- 53 Views
- 0 replies
- 0 kudos
- 32 Views
- 0 replies
- 0 kudos
The chunk of code in questionsys.path.append(
spark.conf.get("util_path", "/Workspace/Repos/Production/loch-ness/utils/")
)
from broker_utils import extract_day_with_suffix, proper_case_address_udf, proper_case_last_name_first_udf, proper_case_ud...
- 32 Views
- 0 replies
- 0 kudos
- 38 Views
- 0 replies
- 0 kudos
As of this morning we started receiving the following error message on a Databricks job with a single Pyspark Notebook task. The job has not had any code changes in 2 months. The cluster configuration has also not changed. The last successful run of ...
- 38 Views
- 0 replies
- 0 kudos
- 44 Views
- 0 replies
- 0 kudos
Hi,We would like to use Azure Managed Identity to create mount point to read/write data from/to ADLS Gen2?We are also using following code snippet to use MSI authentication to read data from ADLS Gen2 but it is giving error,storage_account_name = "<<...
- 44 Views
- 0 replies
- 0 kudos
by
kDev
• New Contributor
- 5003 Views
- 3 replies
- 1 kudos
Our jobs have been running fine so far w/o any issues on a specific workspace. These jobs read data from files on Azure ADLS storage containers and dont use the hive metastore data at all.Now we attached the unity metastore to this workspace, created...
- 5003 Views
- 3 replies
- 1 kudos
Latest Reply
Ok, so there is the thing:1. If you were not using Unity Catalog first, that means you have used totaly diffrent approach e.g. mouns.If yes, you were accessing storage via dbfs path.Once you switched to Unity Catalog you need to take cate of few thin...
2 More Replies
- 3888 Views
- 10 replies
- 2 kudos
I need databricks for a university research project. Is there any possibility of EDU discounts on DBU? So far I was unable to reach out to Databricks sales. Can you connect me with someone from DB?
- 3888 Views
- 10 replies
- 2 kudos
Latest Reply
Their EDU discount makes their support for university research projects even more valuable.
9 More Replies
- 154 Views
- 2 replies
- 1 kudos
Currently, the bronze table ingests JSON files using @Dlt.table decorator on a spark.readStream functionA daily batch job does some transformation on bronze data and stores results in the silver table.New ProcessBronze still the same.A stream has bee...
- 154 Views
- 2 replies
- 1 kudos
Latest Reply
Thank you thats what I understood too. It is just nice to get validation from someone else that works with this.
1 More Replies
- 342 Views
- 2 replies
- 1 kudos
Hi everyone! I'm new to Databricks and moving my first steps with Delta Live Tables, so please forgive my inexperience. I'm building my first DLT pipeline and there's something that I can't really grasp: how to clear all the objects generated or upda...
- 342 Views
- 2 replies
- 1 kudos
Latest Reply
If you want to reprocess all the data, you can simply for a "Full Refresh" option in the DLT pipeline.
You can read more about it here: https://docs.databricks.com/en/delta-live-tables/updates.html#how-delta-live-tables-updates-tables-and-views
1 More Replies
- 42 Views
- 0 replies
- 0 kudos
I have done the below steps1. Created a databricks managed service principal2. Created a Oauth Secret3. Gave all necessary permissions to the service principalI'm trying to use this Service principal in Azure Devops to automate CI/CD. but it fails as...
- 42 Views
- 0 replies
- 0 kudos
- 49 Views
- 0 replies
- 0 kudos
I have Data Engineering Pipeline workload that run on Databricks.Job cluster has following configuration :- Worker i3.4xlarge with 122 GB memory and 16 coresDriver i3.4xlarge with 122 GB memory and 16 cores ,Min Worker -4 and Max Worker 8 We noticed...
- 49 Views
- 0 replies
- 0 kudos
- 3747 Views
- 3 replies
- 1 kudos
Hi there,I am new to Spark SQL and would like to know if it possible to reproduce the below T-SQL query in Databricks. This is a sample query, but I want to determine if a query needs to be executed or not. DECLARE
      @VariableA AS INT
,     @Vari...
- 3747 Views
- 3 replies
- 1 kudos
Latest Reply
Since you are looking for a single value back, you can use the CASE function to achieve what you need.%sqlSET var.myvarA = (SELECT 6);SET var.myvarB = (SELECT 7);SELECT CASE WHEN ${var.myvarA} = ${var.myvarB} THEN 'Equal' ELSE 'Not equal' END AS resu...
2 More Replies
- 3714 Views
- 4 replies
- 3 kudos
Parameters can be passed to Tasks and the values can be retrieved with:dbutils.widgets.get("parameter_name")More recently, we have been given the ability to add parameters to Jobs.However, the parameters cannot be retrieved like Task parameters.Quest...
- 3714 Views
- 4 replies
- 3 kudos
Latest Reply
an update to my answer: Databricks has advised us that the `dbutils.notebook.entry_point` method is not supported (could be deprecated), and the recommended way to read in a job parameter is through widgets, i.e. `dbutils.widgets.get("param_key")` (...
3 More Replies
- 1451 Views
- 5 replies
- 1 kudos
Hello,I am attempting to configure Autoloader in File Notification mode with Delta Live Tables. I configured an instance profile, but it is not working because I immediately get AWS access denied errors. This is the same issue that is referenced here...
- 1451 Views
- 5 replies
- 1 kudos
by
SreeG
• New Contributor II
- 148 Views
- 2 replies
- 0 kudos
HiI am facing issues when deploying work flows to different environment. The same works for Notebooks and Scripts, when deploying the work flows, it failed with "Authorization Failed. Your token may be expired or lack the valid scope". Anything shoul...
- 148 Views
- 2 replies
- 0 kudos
Latest Reply
Thanks, Yesh. The issue was because of a configuration parameter. After changing that, we could deploy. Thank you
1 More Replies
by
subha2
• New Contributor II
- 196 Views
- 0 replies
- 0 kudos
There are some tables under schema/database under Unity Catalog.The Notebook need to read the table parallel using loop and thread and execute the query configuredBut the sql statement is not getting executed via spark.sql() or spark.read.table().It ...
- 196 Views
- 0 replies
- 0 kudos