- 5951 Views
- 1 replies
- 0 kudos
Execute COPY INTO query with source as temp view
We are having the files that needs to be loaded into the delta tableNow we want to perform some transformation on the files and load that into the tableWhat we didCreate a Spark DF from that fileApply transformation on the DFCreate a temp view from t...
- 5951 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sravan Kumar Mohanraj​ ,Yes, you can use copy query in this case your temp_view will be source.For more info, please visit these links.
- 0 kudos
- 3573 Views
- 2 replies
- 3 kudos
Resolved! API Authentication
I am trying to run some API calls to the account console. I tried with every syntax possible encoded and decoded to get the call successfully but it returns a "user not authenticated" error. But when I tried with the Account admin it worked. I need t...
- 3573 Views
- 2 replies
- 3 kudos
- 823 Views
- 0 replies
- 1 kudos
�� Welcome to Chennai User Group Community! �� We are thrilled to have you join our vibrant and enthusiastic community of users! Whether...
Welcome to Chennai User Group Community! We are thrilled to have you join our vibrant and enthusiastic community of users! Whether you are a seasoned expert or a newcomer, this group is the perfect place for you to connect, learn, and grow alongside...
- 823 Views
- 0 replies
- 1 kudos
- 4240 Views
- 3 replies
- 4 kudos
Resolved! how to create a scala jar using db notebook and save it in a file path inside databricks
I have scala function as below, i am unable to understand how to write a scala jar with the same, please find below code i have used Enforcing Column-Level Encryption - Databrick %scala import com.macasaet.fernet.{Key, StringValidator, Token}import o...
- 4240 Views
- 3 replies
- 4 kudos
- 4 kudos
I had to finally create the jar using teh intellij and sbt iconfiguration on the same env. and then installed the jar in the cluster it worked
- 4 kudos
- 7038 Views
- 5 replies
- 5 kudos
- 7038 Views
- 5 replies
- 5 kudos
- 5 kudos
If you are streaming to delta, not much, the micro batch will fail and in next time the stream will pick up from last successful write (due to ACID). I don't know about other formats, what happens if the stream is aborted in mid micro batch.
- 5 kudos
- 5702 Views
- 6 replies
- 6 kudos
Resolved! Delta Lake’s CDF Feature
https://www.databricks.com/notebooks/delta-lake-cdf.htmlI am trying to understand the above article. Could someone explain to be the below questions?a) From SELECT * FROM table_changes('gold_consensus_eps', 2)why is consensus_eps values of 2.1 and 2....
- 5702 Views
- 6 replies
- 6 kudos
- 6 kudos
Hi @THIAM HUAT TAN​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 6 kudos
- 2819 Views
- 3 replies
- 1 kudos
Resolved! Is it possible to change the sender on an Alert?
Hi there, is it possible to change the sender of an Alert to appear as a company email (abcd@companyname.com) instead of Databricks and the noreply@databricks.com?We're in the process of rebuilding some alerts in Databricks as we're transitioning fro...
- 2819 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Cooper Chung​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 1 kudos
- 4248 Views
- 3 replies
- 2 kudos
Resolved! Z-orderiing df using python
Is there a way to perform Z-ordering using python? With sql you you should be able to use:%sql OPTIMIZE df ZORDER BY (column)however I get the error "Table or view 'df' not found in database 'default''" and since I'm not really using sql, I would lik...
- 4248 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Lukas Goldschmied​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...
- 2 kudos
- 16370 Views
- 7 replies
- 1 kudos
Resolved! What are the advantages of using Delta Live tables (DLT) over Data Build Tool (dbt) in Databricks?
Please explain with some use cases which show the difference between DLT and dbt.
- 16370 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @Prachi Sankhala​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
- 1 kudos
- 5926 Views
- 3 replies
- 1 kudos
Resolved! Alerts Notification Fail
Hi,I set up an alert and apparently it worked because status changes according to the Trigger condition. However I don't received any notification on my email, or Teams channel. Any idea what is happening?Thanks
- 5926 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @uriel chombo​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 1 kudos
- 8316 Views
- 4 replies
- 3 kudos
Connection to on-prem database occasionally goes to the wrong server
We run a dbx job that uses a jdbc connection to three (SQL Server) on-prem databases. Very occasionally a connection will go to the wrong server.In the job log we see the error "com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user '...
- 8316 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Michel Smits​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 3 kudos
- 11275 Views
- 3 replies
- 0 kudos
Resolved! What is the best practice for managing different environments (Staging vs Production) on Databricks?
Should we create separate workspaces for Dev/Test/Prod ? Or should we have 1 workspace and create separate folders for Dev/Test/Prod?
- 11275 Views
- 3 replies
- 0 kudos
- 0 kudos
as per my previous experience, its always good to have different workspaces for different envs, its easy to maintain and helps better with CICD pipeline as well, because lot of organizations provide deployment access to Developers in Dev env but not ...
- 0 kudos
- 1781 Views
- 1 replies
- 0 kudos
run jar file into databricks
I have created a job which run a jar files, but i have this error NoClassDefFoundError: com/google/cloud/hadoop/gcsio/GoogleCloudStorageFileSystemOptions$TimestampUpdatePredicateCaused by: ClassNotFoundException: com.google.cloud.hadoop.gcsio.GoogleC...
- 1781 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey Aubert, seems you are missing dependent class in jar. Either package the dependent classes in jar or add them into class path.
- 0 kudos
- 8578 Views
- 2 replies
- 0 kudos
File copy from local to dbfs
How to copy a file from local disk to databricks dbfs path. I tried as below but it is throwing me error:code i tried: dbutils.fs.cp("file://c:/user/file.txt",dbfs/data/) and dbutils.fs.cp("file:///c:/user/file.txt",dbfs/data/) error: File not found ...
- 8578 Views
- 2 replies
- 0 kudos
- 0 kudos
I assume you cannot copy files from Local machine to dbfs using dbutils. you can upload files to dbfs using below gui option . Data --> Browse DFS --> Upload
- 0 kudos
- 8124 Views
- 2 replies
- 3 kudos
load files filtered by last_modified in PySpark
Hi, community!How do you think what is the best way to load from Azure ADLS (actually, filesystem doesn't matter) into df onli files modified after some point in time?Is there any function like input_file_name() but for last_modified to use it in a w...
- 8124 Views
- 2 replies
- 3 kudos
- 3 kudos
_metadata will provide file modification timestamp. I tried on dbfs but not sure for ADLS.https://docs.databricks.com/ingestion/file-metadata-column.html
- 3 kudos
| User | Count |
|---|---|
| 1614 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
