- 4411 Views
- 3 replies
- 1 kudos
dbutils.notebook.run() fails with job aborted but running the notebook individually works
I have a notebook that runs many notebooks in order, along the lines of:```%pythonnotebook_list = ['Notebook1', 'Notebook2'] for notebook in notebook_list: print(f"Now on Notebook: {notebook}") try: dbutils.notebook.run(f'{notebook}', 3600) e...
- 4411 Views
- 3 replies
- 1 kudos
- 1 kudos
I found the problem. Even if a notebook creates and specifies a widget fully, the notebook run process, e.g, dbutils.notebook.run('notebook') will not know how to use it. If I replace my widget with a non-widget provided value, the process works fine...
- 1 kudos
- 947 Views
- 0 replies
- 0 kudos
www.pieseford.ro
Hi, my name is Jerry Maguire and I`m automatic engineer at Piese Ford. Piese originale Ford Fiesta 2008-2012
- 947 Views
- 0 replies
- 0 kudos
- 7187 Views
- 4 replies
- 2 kudos
Resolved! How to view the SQL Query History of traditional Databricks cluster (not Databricks SQL)?
I tried use the Spark Cluster UI. But the queries are truncated.
- 7187 Views
- 4 replies
- 2 kudos
- 2 kudos
In Apache Spark prior to 2.1, once a SQL query was run, there was no way to re-run it; all history was lost. Spark SQL introduced the "replay" functionality in Spark 2.1.0, enabling users to re-run any query they have already run. You can run a query...
- 2 kudos
- 4623 Views
- 2 replies
- 3 kudos
Resolved! Terminated with exception: Could not initialize class org.rocksdb.Options
Problem Statement : When running Delta Live tables ,it is giving the error.Error Message : Could not initialize class org.rocksdb.Optionsorg.apache.spark.sql.streaming.StreamingQueryException: Query cpicpg_us_tgt_amz_bronze [id = a42eec82-0ee8-41b4-9...
- 4623 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi Team ,Thanks for your response, I faced this issue while executing the Delta Live tables / pipeline.Initially i choose product edition as Core and attached 4 notebooks to the pipeline and each notebook have Bronze and silver tables creation. duri...
- 3 kudos
- 6596 Views
- 1 replies
- 0 kudos
Execute tasks parallel to process multiple files parallel
Hi all, If we have multiple tasks under the job, How to invoke a specific task under a job.Do we have any API to invoke Job and its specific tasks instead of Job.Use case: When we receive multiple messages from the event hub, each underlying task in ...
- 6596 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your response, My question is ,if we have multiple tasks in a job ,How can we invoke specific task, I can see API to invoke the job but not a particular task in it. Kindly find attachment for your reference.
- 0 kudos
- 6794 Views
- 3 replies
- 1 kudos
Error as no such file when reading CSV file using pandas
I'm trying to read a CSV file saved in data using pandas read_csv function. But it gives No such file error.%fs ls /FileStore/tables/ df= pd.read_csv('/dbfs/FileStore/tables/CREDIT_1.CSV') df= pd.read_csv('/dbfs:/FileStore/tables/CREDIT_1.CSV')...
- 6794 Views
- 3 replies
- 1 kudos
- 1 kudos
Thanks to @Werner Stinckens for the answer.I understood that I have to use spark to read data from clusters.
- 1 kudos
- 8456 Views
- 3 replies
- 4 kudos
Resolved! Column name, starting with a number
Hi,I see it is possible to start a column name with a number, like `123_test`And store in a hive table with a location in delta.On that documentation https://www.stitchdata.com/docs/destinations/databricks-delta/reference#transformations--column-nami...
- 8456 Views
- 3 replies
- 4 kudos
- 4 kudos
ha ha, yes, I try to find back the right page in DB documentation. If you have it, please, share.
- 4 kudos
- 4313 Views
- 2 replies
- 4 kudos
Resolved! Cache Select on Temp Table?
How might I cache a temp table?The documentation suggests it is possible: https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-cache.htmlConsider the following on DBR 10.5 and Spark 3.2.1:```%pythondf.createOrReplaceTempView("chan...
- 4313 Views
- 2 replies
- 4 kudos
- 8323 Views
- 5 replies
- 2 kudos
Get current date as string in databricks using scala
I want to get current date in scala as a string for example today current date is 3rd jan, want to store it as a new variable dynamically as below, how to get it.val currdate : String = “20220103”when I am using val currdate = Calendar.getInstance.ge...
- 8323 Views
- 5 replies
- 2 kudos
- 2 kudos
Hey @Vibhor Sethi Hope you are well!Thank you for posting your question and letting us know that you were able to resolve the issue. Would you be happy to mark it as the best solution? It would be really helpful for the other members too.Cheers!
- 2 kudos
- 4597 Views
- 2 replies
- 5 kudos
An error occurred while calling o303.mount: Operation failed: "This request is not authorized to perform this operation
Hi Team,We are unable to mount storage container in below scenario We created Gen 2 using VNet and added firewall restrictions (i.e allow trusted sources)And deployed Data bricks workspace with out VNet injection. Is it possible to add databricks pub...
- 4597 Views
- 2 replies
- 5 kudos
- 5 kudos
Hey @Sailaja B Hope everything is great!Does Hubert's response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?Thanks!
- 5 kudos
- 3966 Views
- 3 replies
- 1 kudos
Resolved! I can't access to my account.
I can't access to my account.This acccount was created today(not community, after 14 days trial it will chargable)when I'm try to access my account it gives meInvalid email address or passwordNote: Emails/usernames are case-sensitiveI tried to reset ...
- 3966 Views
- 3 replies
- 1 kudos
- 1 kudos
I got a reset link from the community. Actually the problem was with my username ,it did not identify a character within my username which was my email id.
- 1 kudos
- 4797 Views
- 1 replies
- 2 kudos
How to install JAR libraries from ADLS? I'm having an error
I mounted the ADLS to my Azure Databricks resource and I keep on getting this error when I try to install a JAR from a container:Library installation attempted on the driver node of cluster 0331-121709-buk0nvsq and failed. Please refer to the followi...
- 4797 Views
- 1 replies
- 2 kudos
- 21332 Views
- 6 replies
- 6 kudos
Resolved! Spark Driver Out of Memory Issue
Hi, I am executing a simple job in Databricks for which I am getting below error. I increased the Driver size still I faced same issue. Spark config :from pyspark.sql import SparkSessionspark_session = SparkSession.builder.appName("Demand Forecasting...
- 21332 Views
- 6 replies
- 6 kudos
- 6 kudos
I am getting the above issue while writing a Spark DF as a parquet file to AWS S3. Not doing any broadcast join actually.
- 6 kudos
- 3930 Views
- 1 replies
- 2 kudos
Resolved! Best way to encrypt PII data
Hi guys, I have around 600GB per load, in you opnion, what is the best way to encrypt PII data in terms of performance ? (lib, cluster type, etc.)Thank youWilliam
- 3930 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @William Scardua please check if the blog helps you.https://databricks.com/blog/2020/11/20/enforcing-column-level-encryption-and-avoiding-data-duplication-with-pii.html
- 2 kudos
- 4156 Views
- 1 replies
- 1 kudos
Facing mount/unmount issue while running same job parallelly with scala.
Using above configuration in cluster, when I run databricks job parallelly with multiple request at a same time, then I am getting mount/unmount issue. For an example : When I make three request to databricks job , it run 3 jobs parallelly but somet...
- 4156 Views
- 1 replies
- 1 kudos
- 1 kudos
hi @rahul upadhyay are you using the same mount path /mnt/rahul in all the 3 jobs? Could you please add the full error message?
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 573 | |
| 349 | |
| 287 |
