- 2613 Views
- 2 replies
- 3 kudos
Resolved! API Authentication
I am trying to run some API calls to the account console. I tried with every syntax possible encoded and decoded to get the call successfully but it returns a "user not authenticated" error. But when I tried with the Account admin it worked. I need t...
- 2613 Views
- 2 replies
- 3 kudos
- 609 Views
- 0 replies
- 1 kudos
�� Welcome to Chennai User Group Community! �� We are thrilled to have you join our vibrant and enthusiastic community of users! Whether...
Welcome to Chennai User Group Community! We are thrilled to have you join our vibrant and enthusiastic community of users! Whether you are a seasoned expert or a newcomer, this group is the perfect place for you to connect, learn, and grow alongside...
- 609 Views
- 0 replies
- 1 kudos
- 3428 Views
- 3 replies
- 4 kudos
Resolved! how to create a scala jar using db notebook and save it in a file path inside databricks
I have scala function as below, i am unable to understand how to write a scala jar with the same, please find below code i have used Enforcing Column-Level Encryption - Databrick %scala import com.macasaet.fernet.{Key, StringValidator, Token}import o...
- 3428 Views
- 3 replies
- 4 kudos
- 4 kudos
I had to finally create the jar using teh intellij and sbt iconfiguration on the same env. and then installed the jar in the cluster it worked
- 4 kudos
- 5480 Views
- 5 replies
- 5 kudos
- 5480 Views
- 5 replies
- 5 kudos
- 5 kudos
If you are streaming to delta, not much, the micro batch will fail and in next time the stream will pick up from last successful write (due to ACID). I don't know about other formats, what happens if the stream is aborted in mid micro batch.
- 5 kudos
- 2921 Views
- 2 replies
- 0 kudos
UnsupportedOperationException: Cannot perform Merge as multiple source rows matched and attempted to modify the same
My Code:-- CREATE OR REPLACE TEMPORARY VIEW preprocessed_source ASSELECT Key_ID, Distributor_ID, Customer_ID, Customer_Name, ChannelFROM integr_masterdata.Customer_Master;-- Step 2: Perform the merge operation using the preprocessed source table...
- 2921 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Prashant Joshi​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
- 0 kudos
- 3862 Views
- 6 replies
- 6 kudos
Resolved! Delta Lake’s CDF Feature
https://www.databricks.com/notebooks/delta-lake-cdf.htmlI am trying to understand the above article. Could someone explain to be the below questions?a) From SELECT * FROM table_changes('gold_consensus_eps', 2)why is consensus_eps values of 2.1 and 2....
- 3862 Views
- 6 replies
- 6 kudos
- 6 kudos
Hi @THIAM HUAT TAN​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 6 kudos
- 2149 Views
- 3 replies
- 1 kudos
Resolved! Is it possible to change the sender on an Alert?
Hi there, is it possible to change the sender of an Alert to appear as a company email (abcd@companyname.com) instead of Databricks and the noreply@databricks.com?We're in the process of rebuilding some alerts in Databricks as we're transitioning fro...
- 2149 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Cooper Chung​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 1 kudos
- 2262 Views
- 3 replies
- 2 kudos
Resolved! Z-orderiing df using python
Is there a way to perform Z-ordering using python? With sql you you should be able to use:%sql OPTIMIZE df ZORDER BY (column)however I get the error "Table or view 'df' not found in database 'default''" and since I'm not really using sql, I would lik...
- 2262 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Lukas Goldschmied​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...
- 2 kudos
- 11908 Views
- 7 replies
- 1 kudos
Resolved! What are the advantages of using Delta Live tables (DLT) over Data Build Tool (dbt) in Databricks?
Please explain with some use cases which show the difference between DLT and dbt.
- 11908 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @Prachi Sankhala​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
- 1 kudos
- 4723 Views
- 3 replies
- 1 kudos
Resolved! Alerts Notification Fail
Hi,I set up an alert and apparently it worked because status changes according to the Trigger condition. However I don't received any notification on my email, or Teams channel. Any idea what is happening?Thanks
- 4723 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @uriel chombo​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 1 kudos
- 7488 Views
- 4 replies
- 3 kudos
Connection to on-prem database occasionally goes to the wrong server
We run a dbx job that uses a jdbc connection to three (SQL Server) on-prem databases. Very occasionally a connection will go to the wrong server.In the job log we see the error "com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user '...
- 7488 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Michel Smits​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 3 kudos
- 9676 Views
- 3 replies
- 0 kudos
Resolved! What is the best practice for managing different environments (Staging vs Production) on Databricks?
Should we create separate workspaces for Dev/Test/Prod ? Or should we have 1 workspace and create separate folders for Dev/Test/Prod?
- 9676 Views
- 3 replies
- 0 kudos
- 0 kudos
as per my previous experience, its always good to have different workspaces for different envs, its easy to maintain and helps better with CICD pipeline as well, because lot of organizations provide deployment access to Developers in Dev env but not ...
- 0 kudos
- 1472 Views
- 1 replies
- 0 kudos
run jar file into databricks
I have created a job which run a jar files, but i have this error NoClassDefFoundError: com/google/cloud/hadoop/gcsio/GoogleCloudStorageFileSystemOptions$TimestampUpdatePredicateCaused by: ClassNotFoundException: com.google.cloud.hadoop.gcsio.GoogleC...
- 1472 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey Aubert, seems you are missing dependent class in jar. Either package the dependent classes in jar or add them into class path.
- 0 kudos
- 7032 Views
- 2 replies
- 0 kudos
File copy from local to dbfs
How to copy a file from local disk to databricks dbfs path. I tried as below but it is throwing me error:code i tried: dbutils.fs.cp("file://c:/user/file.txt",dbfs/data/) and dbutils.fs.cp("file:///c:/user/file.txt",dbfs/data/) error: File not found ...
- 7032 Views
- 2 replies
- 0 kudos
- 0 kudos
I assume you cannot copy files from Local machine to dbfs using dbutils. you can upload files to dbfs using below gui option . Data --> Browse DFS --> Upload
- 0 kudos
- 6530 Views
- 2 replies
- 3 kudos
load files filtered by last_modified in PySpark
Hi, community!How do you think what is the best way to load from Azure ADLS (actually, filesystem doesn't matter) into df onli files modified after some point in time?Is there any function like input_file_name() but for last_modified to use it in a w...
- 6530 Views
- 2 replies
- 3 kudos
- 3 kudos
_metadata will provide file modification timestamp. I tried on dbfs but not sure for ADLS.https://docs.databricks.com/ingestion/file-metadata-column.html
- 3 kudos