- 4203 Views
- 3 replies
- 6 kudos
Append to a record when updating Delta Table
I am updating the delta table in Databricks as follows segments_data.alias('segments_old').merge( segments_data_new.alias("updates"), "segments_old.source_url = updates.source_url", ).whenMatchedUpdate( set={"segments"...
- 4203 Views
- 3 replies
- 6 kudos
- 2587 Views
- 1 replies
- 3 kudos
Resolved! Shortcut for changing cell language (adding magic command)
Is there an option or shortcut (or is it in the pipeline) to quickly change / insert a cell to a specific language in Databricks?Triggering B + P would for instance add a new cell below with %pythonTriggering M would for instance change that cell to ...
- 2587 Views
- 1 replies
- 3 kudos
- 3 kudos
I've passed this message along to the UX UI Feedback team.
- 3 kudos
- 3725 Views
- 2 replies
- 5 kudos
Resolved! Best way to automatically update a delta table schema
We have multiple environments where the same tables are added so it's really hard to manually update the schema of the table across all the environments. We know that it's not ideal to update table schema a lot but our product is still evolving and s...
- 3725 Views
- 2 replies
- 5 kudos
- 2758 Views
- 1 replies
- 1 kudos
Merge didn't fail while inserting wrong data type values
Hello. During some example cases we were running, in order to identify how Databricks treats possible wrong actions we could make, we noticed that merge doesn't fail while inserting different data type values from the ones in the corresponding table....
- 2758 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello. Any update on this please? Thank you in advance
- 1 kudos
- 2172 Views
- 0 replies
- 3 kudos
Using Azure Databricks to manage my R Shiny front-end, need to know the user via R session
Using the Azure Databricks R Shiny example, I am easily able to instantiate a R Shiny web front session. The URL that is provided by Databricks provides the proxy session that allows my targeted audience to directly access my `R Shiny` app using tha...
- 2172 Views
- 0 replies
- 3 kudos
- 3068 Views
- 3 replies
- 1 kudos
Pass_SS
Have completed my certificate on 27th October for Databricks Data Analyst Associate but haven't received my Certificate/ Badge for the same.
- 3068 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello Priyanshu, your certificate was issued today. Please check your email as I resent the information.Thank you
- 1 kudos
- 3218 Views
- 2 replies
- 3 kudos
Scala Connectivity to Databricks Bronze Layer Raw Data from a Non-Databricks Spark environment
Hi All, We are developing a new Scala/Java program which needs to read & process the raw data stored in source ADLS (which is a Databricks Environment) in parallel as the volume of the source data is very high (in GBs & TBs). What kind of connection ...
- 3218 Views
- 2 replies
- 3 kudos
- 3 kudos
hello experts. any advise on this question ?? tagging some folks from whom I have received answers before. Please help on this requirement or tag someone who can help on this@Kaniz Fatma​ , @Vartika Nain​ , @Bilal Aslam​
- 3 kudos
- 2811 Views
- 4 replies
- 0 kudos
Git repo cloning on Databricks
i am running into issues importing the scalable-machine-learning-with-apache-spark library into databricks. specifically, cloning from git library or %pip install from git library directly to Databricks. Any help is appreciated
- 2811 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Aritra Guha​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 0 kudos
- 5053 Views
- 5 replies
- 0 kudos
spark-submit Error "Unrecognized option: --executor-memory 3G" although --executor-memory is available in Options.
Executed a spark-submit job through databricks cli with the following job configurations.{ "job_id": 123, "creator_user_name": "******", "run_as_user_name": "******", "run_as_owner": true, "settings": { "name": "44aa-8447-c123aad310", ...
- 5053 Views
- 5 replies
- 0 kudos
- 0 kudos
Not really sure if running spark on local mode. But have used alternate property spark.executor.memoryand passed it as --conf and now it works
- 0 kudos
- 1606 Views
- 0 replies
- 1 kudos
evidences_pass
I haven't received my Databricks Certified Data Engineer AssociateI have pass my certification exam, Databricks Certified Data Engineer Associate on 27 October 2022.. I am yet to receive a certificate or badge. Any help is much appreciated. I have a ...
- 1606 Views
- 0 replies
- 1 kudos
- 3972 Views
- 2 replies
- 2 kudos
Using the %debug magic in DataBricks notebook
When developing locally, I can write %debug in a new cell after encountering an error, and jump into the function that the error originated from. In Databricks, this freezes the notebook indefinitely.For example:In [1]:def query_data(): df_full = qu...
- 3972 Views
- 2 replies
- 2 kudos
- 2 kudos
I just upgraded my personal node and %debug worked! I appreciate the reminder to use pdb() itself when appropriate too. I'm still interested in whether we should have any concerns about upgrading our main cluster - please do let me know, and then I'l...
- 2 kudos
- 1194 Views
- 0 replies
- 1 kudos
community.databricks.com
Hi I have similiar issue:https://community.databricks.com/s/question/0D58Y00008oTUQPSA4/voucher-code-error
- 1194 Views
- 0 replies
- 1 kudos
- 1259 Views
- 0 replies
- 2 kudos
How can I know if an instance has fallen back to On-demand?
Hi, I have several clusters, some with a 45% max spot price, some more important with a higher value. Want to know what is the best way to configure this but cannot find anything (a value of how many nodes of the last run were On-demand will do the t...
- 1259 Views
- 0 replies
- 2 kudos
- 2287 Views
- 2 replies
- 2 kudos
Full screensmode is not working for spark course, Is there anyone who tried the same and facing the same issue .
Full screensmode is not working for spark course, Is there anyone who tried the same and facing the same issue .
- 2287 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Abhishek Joshi​ Does @Hubert Dudek​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 2 kudos
- 2845 Views
- 3 replies
- 4 kudos
Help optimizing large empty gaps where no executors are running jobs in Spark UI. Structured streaming writing.
Hi, I'm running a structured streaming job on a pipeline with a medallion architecture. In my silver layer, we are reading from the bronze layer using structured streaming, and writing the stream to the silver layer w/ a foreachbatch function doing s...
- 2845 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Danny Liu​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 4 kudos
| User | Count |
|---|---|
| 1626 | |
| 790 | |
| 506 | |
| 349 | |
| 287 |
