- 3334 Views
- 4 replies
- 1 kudos
What is difference between _RAW tables and _APPEND_RAW tables of Bronze-Layer of Azure Databricks
Hi Team,I would like to know difference between _RAW tables and _APPEND_RAW tables of Bronze-Layer.As both are STREAMING Tables then why we need 2 separate tables.Note: we are following Medalion Architecture. Also above tables are created via Delta L...
- 3334 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod , I saw your replies to other posts, so thought to ask you....would you like to help me on this...!!!
- 1 kudos
- 2147 Views
- 1 replies
- 1 kudos
job failing with below error massage
Caused by: HTTP Error -1; url='https://login.microsoftonline.com/271df5c2-953a-497b-93ad-7adf7a4b3cd7/oauth2/token' AzureADAuthenticator.getTokenCall threw java.net.UnknownHostException : login.microsoftonline.comshaded.databricks.azurebfs.org.apache...
- 2147 Views
- 1 replies
- 1 kudos
- 1 kudos
This was caused for me by the line (pyspark):children = [f for f in dbutils.fs.ls(node)]with the node being a"dbfs:/mnt/lifestrategy-blob/scada/"and this a mounted directory. It seems like the implementation of dbutils.fs is done with the same qualit...
- 1 kudos
- 6915 Views
- 5 replies
- 6 kudos
Resolved! Why is the workflow trigger status always paused?
I create a workflow job via Asset Bundle. However, after deploying the job to the databricks the trigger status is always paused even no matter how I update the cron expression. I can manually trigger it successfully. I cannot figure out why. Am I mi...
- 6915 Views
- 5 replies
- 6 kudos
- 6 kudos
Next to the cron expression, you also need the following property: pause_statusFor example:schedule: quartz_cron_expression: 0 0 6 * * ? timezone_id: Europe/Amsterdam pause_status: UNPAUSEDThe property can be set to PAUSED and UNPAUSED. Hope th...
- 6 kudos
- 2440 Views
- 2 replies
- 1 kudos
Resolved! DLT pipeline - reading from external tables
Hello!I created a DLT pipeline where my sources are external tables. I have to apply changes (stored_as_scd_type = 1). However, when I run my pipeline, I don't see any incremental uploads. The data remains in the same state as when I first created th...
- 2440 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @ksenija, I hope this message finds you well. Is your source table receiving new records? If so, are the fields (operation/sequenceNum) being filled? If possible, please provide a sample of the code you are using to create your target table wit...
- 1 kudos
- 1389 Views
- 0 replies
- 0 kudos
shared serverless vs dedicated serverless?
Hi All,I gone through https://docs.databricks.com/en/admin/system-tables/serverless-billing.html and wondering..How serverless compute is shared across workloads.is there a option to setup that? difference between shared serverless vs dedicated serve...
- 1389 Views
- 0 replies
- 0 kudos
- 3576 Views
- 1 replies
- 0 kudos
Resolved! Databricks Asset Bundles library dependencies - JAR file
Hi there,I have used databricks asset bundles (DAB) to deploy workflows. For each job, I will create a job cluster and install external libraries by specifying libraries in each task, for example:- task_key: my-task job_cluster_key: my-cluster note...
- 3576 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks very much @Retired_mod for your thorough answer.
- 0 kudos
- 2490 Views
- 1 replies
- 0 kudos
Python udfs, Spark Connect, included modules. Compatibility issues with shared compute
Our current system uses Databricks notebooks and we have some shared notebooks that define some python udfs. This was working for us until we tried to switch from single user clusters to shared clusters. Shared clusters and serverless now use Spark C...
- 2490 Views
- 1 replies
- 0 kudos
- 0 kudos
I'm not sure what you mean by "Ensure the Python binary's location is correctly set to resolve runtime issues" . We aren't using any binaries. Everything is just Databricks notebooks. In our case if we define a python udf function in the root notebo...
- 0 kudos
- 7324 Views
- 4 replies
- 5 kudos
Resolved! SQL Server To Databricks Table Migration
Hello,Is there an equivalent SQL code for the following Pyspark code? I'm trying to copy a table from SQL Server to Databricks and save it as a managed delta table.jdbcHostname = "your_sql_server_hostname" jdbcPort = 1433 jdbcDatabase = "your_databas...
- 7324 Views
- 4 replies
- 5 kudos
- 5 kudos
The only option to have it in Databricks SQL is lakehouse federation with a SQL Server connection.
- 5 kudos
- 1375 Views
- 0 replies
- 0 kudos
Performance drop from databricks 12.2 to 14.3 LTS - solved with checkpoint(), looking for root cause
On databricks 12.2, a piece of code which has an action takes a minute to run.On databricks 14.3, the same code unchanged, same inputs, now takes 10 minutes.Attempting to debug using explain() shows the plan is huge (150k plus rows of output)Replacin...
- 1375 Views
- 0 replies
- 0 kudos
- 3449 Views
- 2 replies
- 1 kudos
DLT- apply_changes() SCD2 - Rename_columns : __START_AT & __END_AT
Renaming the column names of the __START_AT and __END_AT columns created when using the dlt.apply_changes() method for performing SCD2 type updates.
- 3449 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @VenkateswarluAd , Greetings of the day! The columns __START_AT and __END_AT are used to track the validity period of each record for SCD Type 2 updates. Please be aware that renaming these columns could disrupt the functionality of the SCD Typ...
- 1 kudos
- 1757 Views
- 0 replies
- 1 kudos
Query execution after establishing Databricks to Information Design Tool JDBC Connection
Hello all,I have created a JDBC connection from Databricks to Information Design Tool using access token generated using Databricks Service Principal.But it’s throwing below error while running query on top of Databricks data in Information Design Bu...
- 1757 Views
- 0 replies
- 1 kudos
- 1538 Views
- 1 replies
- 1 kudos
Java code to read azure storage file in a jar type databricks job
I have a java application, packed as a jar, and will be used as jar dbx job.This application need1. read azure storage file, yaml format.2. need to get passphrase, privatekey stored in dbx, in order to access a snowflake dbmy questions are:1. how to ...
- 1538 Views
- 1 replies
- 1 kudos
- 3767 Views
- 3 replies
- 1 kudos
Accessing Private API in databricks notebook
Hello, I am trying to access an API in databricks python notebook which is available within a restricted network. ​When I try to access that API, it's not able to find the URL used to access the API and throws an HTTP error (max retries exceeded).​d...
- 3767 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi! Could you recommend a way to setup a proxy server that can reroute all HTTP traffic according to the above advice? Thank you!Kind regards,Pim
- 1 kudos
- 2797 Views
- 0 replies
- 1 kudos
Error: cannot create job: 'SERVICE_PRINCIPAL_NAME' cannot be set as run_as_username
Hi Team, I am trying to deploy workflows to Databricks Workspace via Databricks Asset Bundle (DAB) using Azure Service Principle. Below is my databricks.yml file which i am using for DAB.I am replacing the "SERVICE_PRINCIPAL_NAME" variable in my Jenk...
- 2797 Views
- 0 replies
- 1 kudos
- 7163 Views
- 1 replies
- 0 kudos
leaflet not works in notebook(R language)
I saw this notebook: htmlwidgets-azure - Databricks (microsoft.com)However, it is not reproducible. I got a lot errors:there is no package called ‘R.utils’. This is easy to fix, just install the package "R.utils""can not be unloaded". This is not ...
- 7163 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi yalei, Did you have any luck fixing this issue? I am also trying to replicate the htmlwidgets notebook and am running into the same error.Unfortunately, the suggestions provided by Kaniz_Fatma below did not work.
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
