- 3356 Views
- 3 replies
- 4 kudos
Databricks Notebook dataframe loading duplicate data in SQL table
Hi, I am trying to load data from datalake into SQL table using "SourceDataFrame.write" operation in a Notebook using apache spark.This seems to be loading duplicates at random times. The logs don't give much information and I am not sure what else t...
- 3356 Views
- 3 replies
- 4 kudos
- 4 kudos
can you elaborate a bit more on this notebook?And also what databricks runtime version?
- 4 kudos
- 17586 Views
- 9 replies
- 4 kudos
docs.databricks.com
Navigate and discover content more efficiently with Search in DatabricksHi all- Justin Kim here, I'm the Databricks product manager responsible for content organization and navigation in our product, which includes Search. Great to see you on the Com...
- 17586 Views
- 9 replies
- 4 kudos
- 4 kudos
@Justin Kim​ Thank you for quick reply, usually Last Modified is Recent changes right (that can be last 24hrs or cap limit that we add), whereas anytime they should show all Notebooks or Tables from start. that is where i got confused
- 4 kudos
- 2129 Views
- 1 replies
- 3 kudos
Queries with running REST API command in databricks to create a Job
What happens when jobs/create REST API command is run multiple times(say 3 times) with the same JSON configuration? Will 3 jobs are created with the same name or only 1 job will be created?
- 2129 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Santhosh Raj​ , logically only one job should be created.
- 3 kudos
- 9082 Views
- 2 replies
- 1 kudos
Resolved! PARSE_SYNTAX_ERROR: Syntax error at or near 'VACUUM'
I tried to VACUUM a delta table, but there is a Syntax error.Here is the code:%sql set spark.databricks.delta.retentionDurationCheck.enabled = False VACUUM test_deltatable
- 9082 Views
- 2 replies
- 1 kudos
- 1 kudos
@Cheuk Hin Christophe Poon​ Missing semi-colon at end of line 2?%sql set spark.databricks.delta.retentionDurationCheck.enabled = False; VACUUM test_deltatable
- 1 kudos
- 3705 Views
- 2 replies
- 2 kudos
How to write the delta files for managed table? how can I define the sink
I have tried below code to write data in a delta table and save the delta files in a sink. I tried using azure storage as sink but I get error as not enough access, I can confirm that I have enough access to azure storage, however I can run the below...
- 3705 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Ankit Kumar​ Does @Hubert Dudek​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 2 kudos
- 5878 Views
- 4 replies
- 0 kudos
How can I run a scala command line in databricks?
I wish to run a scala command, which I believe would normally be run from a scala command line rather than from within a notebook. It happens to be:scala [-cp scalatest-<version>.jar:...] org.scalatest.tools.Runner [arguments](scalatest_2.12__3.0.8.j...
- 5878 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @David Vardy​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 0 kudos
- 5781 Views
- 4 replies
- 7 kudos
Azure Databricks SCIM API GA Availability plans?
When is the Azure SCIM API 2.0 is going General Availability. Currently I see it is on public preview?And also any security concerns for the APIs In Preview if it is being used in our current production?Reference: https://learn.microsoft.com/en-us/az...
- 5781 Views
- 4 replies
- 7 kudos
- 7 kudos
Hi @Sultan Mohideen Hafeez Aboobacker Mohammed Shahul​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more h...
- 7 kudos
- 2328 Views
- 2 replies
- 0 kudos
Lakehouse Concept
I want to understand lake house concept in very brief If I have to pitch for a customer in 1 minute
- 2328 Views
- 2 replies
- 0 kudos
- 0 kudos
Lakehouse is a concept defined with the following Parameter-Data is stored in an open standard format.Data is stored in a way which support Data Science,ML and BI loads.Delta is just a way or engine on cloud storage that provides control on data and...
- 0 kudos
- 3436 Views
- 2 replies
- 0 kudos
Sign in to Databricks Partner Academy
A few months ago, I signed up for the Databricks Academy, customer version. Since then, my company has become a Databricks Partner (Go Databricks + Stardog!). My colleagues have successfully registered for Databricks Partner Academy, but I can't get ...
- 3436 Views
- 2 replies
- 0 kudos
- 4371 Views
- 2 replies
- 2 kudos
How to configure Databricks token inside Docker File?
I have a docker file where I want toDownload the Databricks CLIConfigure the CLI by adding a host and tokenAnd then running a python file that hits the Databricks tokenI am able to install the CLI in the docker image, and I have a working python file...
- 4371 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi I was facing same issue and searching for the solution but didnt get it, and now after working on it i have the solution if you want to access databricks models/download_artifacts using hostname and access token like how you do on databricks cli ...
- 2 kudos
- 2728 Views
- 1 replies
- 2 kudos
How to get the Databricks Default Pricing through API or any endpoints (*if available)???
Attaching Screenshots FYI from the official site, I've checked the Inspect, but no API calls have been made specifically for this cost default scrapping, Is there any Endpoints available to scrape this?
- 2728 Views
- 1 replies
- 2 kudos
- 2 kudos
@All Users Group​ Any views on this??
- 2 kudos
- 5049 Views
- 3 replies
- 1 kudos
Exam Registration & Voucher
I would like to register for the Data Engineer Associate exam. I have also passed the Databricks Lakehuse Fundamentals accreditation (id: 61076337), but on the registration site: https://www.webassessor.com/ does not allow me to purchase the exam, bu...
- 5049 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Antonio Rodilosso​ ,As far as I know voucher is not compulsory. You can register for an exam by paying the amount. Its absolutely possible to register for an exam without voucherTo earn a voucher you can check the below url.http://msdatalab.net/h...
- 1 kudos
- 2652 Views
- 0 replies
- 0 kudos
Horovod Databricks Job - custom module not found error
We have used the following example to successfully create a distributed deep learning training notebook https://www.databricks.com/blog/2022/09/07/accelerating-your-deep-learning-pytorch-lightning-databricks.html that works as expected.We now want to...
- 2652 Views
- 0 replies
- 0 kudos
- 5290 Views
- 3 replies
- 6 kudos
Append to a record when updating Delta Table
I am updating the delta table in Databricks as follows segments_data.alias('segments_old').merge( segments_data_new.alias("updates"), "segments_old.source_url = updates.source_url", ).whenMatchedUpdate( set={"segments"...
- 5290 Views
- 3 replies
- 6 kudos
- 3301 Views
- 1 replies
- 3 kudos
Resolved! Shortcut for changing cell language (adding magic command)
Is there an option or shortcut (or is it in the pipeline) to quickly change / insert a cell to a specific language in Databricks?Triggering B + P would for instance add a new cell below with %pythonTriggering M would for instance change that cell to ...
- 3301 Views
- 1 replies
- 3 kudos
- 3 kudos
I've passed this message along to the UX UI Feedback team.
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
