- 2420 Views
- 4 replies
- 0 kudos
Git repo cloning on Databricks
i am running into issues importing the scalable-machine-learning-with-apache-spark library into databricks. specifically, cloning from git library or %pip install from git library directly to Databricks. Any help is appreciated
- 2420 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Aritra Guha​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 0 kudos
- 4455 Views
- 5 replies
- 0 kudos
spark-submit Error "Unrecognized option: --executor-memory 3G" although --executor-memory is available in Options.
Executed a spark-submit job through databricks cli with the following job configurations.{ "job_id": 123, "creator_user_name": "******", "run_as_user_name": "******", "run_as_owner": true, "settings": { "name": "44aa-8447-c123aad310", ...
- 4455 Views
- 5 replies
- 0 kudos
- 0 kudos
Not really sure if running spark on local mode. But have used alternate property spark.executor.memoryand passed it as --conf and now it works
- 0 kudos
- 1444 Views
- 0 replies
- 1 kudos
evidences_pass
I haven't received my Databricks Certified Data Engineer AssociateI have pass my certification exam, Databricks Certified Data Engineer Associate on 27 October 2022.. I am yet to receive a certificate or badge. Any help is much appreciated. I have a ...
- 1444 Views
- 0 replies
- 1 kudos
- 3479 Views
- 2 replies
- 2 kudos
Using the %debug magic in DataBricks notebook
When developing locally, I can write %debug in a new cell after encountering an error, and jump into the function that the error originated from. In Databricks, this freezes the notebook indefinitely.For example:In [1]:def query_data(): df_full = qu...
- 3479 Views
- 2 replies
- 2 kudos
- 2 kudos
I just upgraded my personal node and %debug worked! I appreciate the reminder to use pdb() itself when appropriate too. I'm still interested in whether we should have any concerns about upgrading our main cluster - please do let me know, and then I'l...
- 2 kudos
- 1015 Views
- 0 replies
- 1 kudos
community.databricks.com
Hi I have similiar issue:https://community.databricks.com/s/question/0D58Y00008oTUQPSA4/voucher-code-error
- 1015 Views
- 0 replies
- 1 kudos
- 9406 Views
- 5 replies
- 3 kudos
How to fix intermittent 503 errors in 10.4 LTS
I sometimes get the below error recently in version 10.4 LTS. Any solution to fix the intermittent failure? I added retry logic in our code, but Databricks query succeeded (even though it threw an exception) and it leads to the unexpected table statu...
- 9406 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Yuya Ebihara​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 3 kudos
- 1078 Views
- 0 replies
- 2 kudos
How can I know if an instance has fallen back to On-demand?
Hi, I have several clusters, some with a 45% max spot price, some more important with a higher value. Want to know what is the best way to configure this but cannot find anything (a value of how many nodes of the last run were On-demand will do the t...
- 1078 Views
- 0 replies
- 2 kudos
- 1861 Views
- 2 replies
- 2 kudos
Full screensmode is not working for spark course, Is there anyone who tried the same and facing the same issue .
Full screensmode is not working for spark course, Is there anyone who tried the same and facing the same issue .
- 1861 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Abhishek Joshi​ Does @Hubert Dudek​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 2 kudos
- 2462 Views
- 3 replies
- 4 kudos
Help optimizing large empty gaps where no executors are running jobs in Spark UI. Structured streaming writing.
Hi, I'm running a structured streaming job on a pipeline with a medallion architecture. In my silver layer, we are reading from the bronze layer using structured streaming, and writing the stream to the silver layer w/ a foreachbatch function doing s...
- 2462 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Danny Liu​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 4 kudos
- 917 Views
- 0 replies
- 2 kudos
What are the Delta Live Table migration advantages?
Currently we are using medallion architecture using delta tables in the form of notebooks and jobs.1) what is the advantage of migrating the existing implementation to Delta Live Tables? 2) what kind of efforts involved in the migration?3) will the m...
- 917 Views
- 0 replies
- 2 kudos
- 10985 Views
- 3 replies
- 3 kudos
Resolved! How do I fix tabs vs spaces in notebooks?
I am getting IndentationError: unindent does not match any outer indentation levelbecause the code I pasted has tabs that are 4 spaces, but the tabs in Databricks are 2 spaces. How do I fix this? Do I have to copy and paste it back out?
- 10985 Views
- 3 replies
- 3 kudos
- 3422 Views
- 4 replies
- 11 kudos
using ipywidgets in azure databricks dashboards
Is it possible to use ipywidgets in azure databricks dashboards?
- 3422 Views
- 4 replies
- 11 kudos
- 11 kudos
Hi @Hossein Amirinia​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 11 kudos
- 3076 Views
- 4 replies
- 2 kudos
Resolved! Orphan (?) files on Databricks S3 bucket
Hi, I'm seeing a lot of empty (and not) directories on routes like:xxxxxx.jobs/FileStore/job-actionstats/xxxxxx.jobs/FileStore/job-result/xxxxxx.jobs/command-results/Can I create a lifecycle to delete old objects (files/directories)? how many days? w...
- 3076 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi! I didn't know that, Purging right now, is there a way to schedule that so logs are retained for less time? Maybe I want to maintain the last 7 days for everything?Thanks!
- 2 kudos
- 1943 Views
- 2 replies
- 3 kudos
Resolved! Fundamentals of Databricks Lakehouse Badge not recieved.
I passed the databricks fundamental exam and like many others I too did not recieved my badge.I am very much intrested in putting this badge on my linkedin profile, please help.My email id is: rahul.psit.ec@gmail.comWhich databricks is resolving as: ...
- 1943 Views
- 2 replies
- 3 kudos
- 3957 Views
- 1 replies
- 0 kudos
How to Configure PySpark Jobs Using PEX
IssueI am attempting to create a PySpark job via the Databricks UI (with spark-submit) using the parameters below (dependencies are on the PEX file), but I am getting the an exception that the pex file does not exist. It's my understanding that the -...
- 3957 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,I'm facing the same issue trying to execute a pyspark job with spark-submit.I have explored the same solution as you : --files optionspark.pyspark.driver.pythonspark.executorEnv.PEX_ROOTDo you make some progress in the resolution of the problem ?
- 0 kudos
| User | Count |
|---|---|
| 1614 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
