- 3044 Views
- 2 replies
- 2 kudos
how to get dbutils in Runtime 13
We're using the following method (generated by using dbx) to access dbutils, e.g. to retrieve parameters from secret scopes: @staticmethod def _get_dbutils(spark: SparkSession) -> "dbutils": try: from pyspark.dbutils import...
- 3044 Views
- 2 replies
- 2 kudos
- 2 kudos
We have something similar in our code. This worked using runtime 13 until last week. Also the Machine Learning DBR doesn't work either.
- 2 kudos
- 8612 Views
- 4 replies
- 3 kudos
Resolved! Autoloader works on compute cluster, but does not work within a task in workflows
I feel like I am going crazy with this. I have tested a data pipeline on my standard compute cluster. I am loading new files as batch from a Google Cloud Storage bucket. Autoloader works exactly as expected from my notebook on my compute cluster. The...
- 8612 Views
- 4 replies
- 3 kudos
- 3 kudos
I found the issue. I describe the solution in the following SO post. https://stackoverflow.com/questions/76287095/databricks-autoloader-works-on-compute-cluster-but-does-not-work-within-a-task/76313794#76313794
- 3 kudos
- 1311 Views
- 1 replies
- 0 kudos
Function in databricks
Im having a hard time to convert below function from SSMS to databricks function. Any help would be appreciated! CREATE FUNCTION [dbo].[MaxOf5Values] (@D1 [int],@D2 [int],@D3 [int],@D4 [int],@D5 [int]) RETURNS int AS BEGIN DECLARE @Result int ...
- 1311 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Givi Salu​ ,​Please refer to this link that will help you convert this function.
- 0 kudos
- 5960 Views
- 1 replies
- 0 kudos
Execute COPY INTO query with source as temp view
We are having the files that needs to be loaded into the delta tableNow we want to perform some transformation on the files and load that into the tableWhat we didCreate a Spark DF from that fileApply transformation on the DFCreate a temp view from t...
- 5960 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sravan Kumar Mohanraj​ ,Yes, you can use copy query in this case your temp_view will be source.For more info, please visit these links.
- 0 kudos
- 3590 Views
- 2 replies
- 3 kudos
Resolved! API Authentication
I am trying to run some API calls to the account console. I tried with every syntax possible encoded and decoded to get the call successfully but it returns a "user not authenticated" error. But when I tried with the Account admin it worked. I need t...
- 3590 Views
- 2 replies
- 3 kudos
- 828 Views
- 0 replies
- 1 kudos
�� Welcome to Chennai User Group Community! �� We are thrilled to have you join our vibrant and enthusiastic community of users! Whether...
Welcome to Chennai User Group Community! We are thrilled to have you join our vibrant and enthusiastic community of users! Whether you are a seasoned expert or a newcomer, this group is the perfect place for you to connect, learn, and grow alongside...
- 828 Views
- 0 replies
- 1 kudos
- 4248 Views
- 3 replies
- 4 kudos
Resolved! how to create a scala jar using db notebook and save it in a file path inside databricks
I have scala function as below, i am unable to understand how to write a scala jar with the same, please find below code i have used Enforcing Column-Level Encryption - Databrick %scala import com.macasaet.fernet.{Key, StringValidator, Token}import o...
- 4248 Views
- 3 replies
- 4 kudos
- 4 kudos
I had to finally create the jar using teh intellij and sbt iconfiguration on the same env. and then installed the jar in the cluster it worked
- 4 kudos
- 7052 Views
- 5 replies
- 5 kudos
- 7052 Views
- 5 replies
- 5 kudos
- 5 kudos
If you are streaming to delta, not much, the micro batch will fail and in next time the stream will pick up from last successful write (due to ACID). I don't know about other formats, what happens if the stream is aborted in mid micro batch.
- 5 kudos
- 5715 Views
- 6 replies
- 6 kudos
Resolved! Delta Lake’s CDF Feature
https://www.databricks.com/notebooks/delta-lake-cdf.htmlI am trying to understand the above article. Could someone explain to be the below questions?a) From SELECT * FROM table_changes('gold_consensus_eps', 2)why is consensus_eps values of 2.1 and 2....
- 5715 Views
- 6 replies
- 6 kudos
- 6 kudos
Hi @THIAM HUAT TAN​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 6 kudos
- 2828 Views
- 3 replies
- 1 kudos
Resolved! Is it possible to change the sender on an Alert?
Hi there, is it possible to change the sender of an Alert to appear as a company email (abcd@companyname.com) instead of Databricks and the noreply@databricks.com?We're in the process of rebuilding some alerts in Databricks as we're transitioning fro...
- 2828 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Cooper Chung​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 1 kudos
- 4284 Views
- 3 replies
- 2 kudos
Resolved! Z-orderiing df using python
Is there a way to perform Z-ordering using python? With sql you you should be able to use:%sql OPTIMIZE df ZORDER BY (column)however I get the error "Table or view 'df' not found in database 'default''" and since I'm not really using sql, I would lik...
- 4284 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Lukas Goldschmied​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...
- 2 kudos
- 16447 Views
- 7 replies
- 1 kudos
Resolved! What are the advantages of using Delta Live tables (DLT) over Data Build Tool (dbt) in Databricks?
Please explain with some use cases which show the difference between DLT and dbt.
- 16447 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @Prachi Sankhala​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
- 1 kudos
- 5957 Views
- 3 replies
- 1 kudos
Resolved! Alerts Notification Fail
Hi,I set up an alert and apparently it worked because status changes according to the Trigger condition. However I don't received any notification on my email, or Teams channel. Any idea what is happening?Thanks
- 5957 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @uriel chombo​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 1 kudos
- 8326 Views
- 4 replies
- 3 kudos
Connection to on-prem database occasionally goes to the wrong server
We run a dbx job that uses a jdbc connection to three (SQL Server) on-prem databases. Very occasionally a connection will go to the wrong server.In the job log we see the error "com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user '...
- 8326 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Michel Smits​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 3 kudos
- 11322 Views
- 3 replies
- 0 kudos
Resolved! What is the best practice for managing different environments (Staging vs Production) on Databricks?
Should we create separate workspaces for Dev/Test/Prod ? Or should we have 1 workspace and create separate folders for Dev/Test/Prod?
- 11322 Views
- 3 replies
- 0 kudos
- 0 kudos
as per my previous experience, its always good to have different workspaces for different envs, its easy to maintain and helps better with CICD pipeline as well, because lot of organizations provide deployment access to Developers in Dev env but not ...
- 0 kudos
| User | Count |
|---|---|
| 1614 | |
| 773 | |
| 349 | |
| 286 | |
| 253 |
