- 42150 Views
- 6 replies
- 10 kudos
How do I prevent _success and _committed files in my write output?
Is there a way to prevent the _success and _committed files in my output. It's a tedious task to navigate to all the partitions and delete the files. Note : Final output is stored in Azure ADLS
- 42150 Views
- 6 replies
- 10 kudos
- 10 kudos
Please find the below steps to remove _SUCCESS, _committed and _started files.spark.conf.set("spark.databricks.io.directoryCommit.createSuccessFile","false") to remove success file.run vacuum command multiple times until _committed and _started files...
- 10 kudos
- 3570 Views
- 3 replies
- 1 kudos
dbutils.notebook.run() fails with job aborted but running the notebook individually works
I have a notebook that runs many notebooks in order, along the lines of:```%pythonnotebook_list = ['Notebook1', 'Notebook2'] for notebook in notebook_list: print(f"Now on Notebook: {notebook}") try: dbutils.notebook.run(f'{notebook}', 3600) e...
- 3570 Views
- 3 replies
- 1 kudos
- 1 kudos
I found the problem. Even if a notebook creates and specifies a widget fully, the notebook run process, e.g, dbutils.notebook.run('notebook') will not know how to use it. If I replace my widget with a non-widget provided value, the process works fine...
- 1 kudos
- 801 Views
- 0 replies
- 0 kudos
www.pieseford.ro
Hi, my name is Jerry Maguire and I`m automatic engineer at Piese Ford. Piese originale Ford Fiesta 2008-2012
- 801 Views
- 0 replies
- 0 kudos
- 6357 Views
- 4 replies
- 2 kudos
Resolved! How to view the SQL Query History of traditional Databricks cluster (not Databricks SQL)?
I tried use the Spark Cluster UI. But the queries are truncated.
- 6357 Views
- 4 replies
- 2 kudos
- 2 kudos
In Apache Spark prior to 2.1, once a SQL query was run, there was no way to re-run it; all history was lost. Spark SQL introduced the "replay" functionality in Spark 2.1.0, enabling users to re-run any query they have already run. You can run a query...
- 2 kudos
- 4245 Views
- 2 replies
- 3 kudos
Resolved! Terminated with exception: Could not initialize class org.rocksdb.Options
Problem Statement : When running Delta Live tables ,it is giving the error.Error Message : Could not initialize class org.rocksdb.Optionsorg.apache.spark.sql.streaming.StreamingQueryException: Query cpicpg_us_tgt_amz_bronze [id = a42eec82-0ee8-41b4-9...
- 4245 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi Team ,Thanks for your response, I faced this issue while executing the Delta Live tables / pipeline.Initially i choose product edition as Core and attached 4 notebooks to the pipeline and each notebook have Bronze and silver tables creation. duri...
- 3 kudos
- 6307 Views
- 1 replies
- 0 kudos
Execute tasks parallel to process multiple files parallel
Hi all, If we have multiple tasks under the job, How to invoke a specific task under a job.Do we have any API to invoke Job and its specific tasks instead of Job.Use case: When we receive multiple messages from the event hub, each underlying task in ...
- 6307 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your response, My question is ,if we have multiple tasks in a job ,How can we invoke specific task, I can see API to invoke the job but not a particular task in it. Kindly find attachment for your reference.
- 0 kudos
- 5916 Views
- 3 replies
- 1 kudos
Error as no such file when reading CSV file using pandas
I'm trying to read a CSV file saved in data using pandas read_csv function. But it gives No such file error.%fs ls /FileStore/tables/ df= pd.read_csv('/dbfs/FileStore/tables/CREDIT_1.CSV') df= pd.read_csv('/dbfs:/FileStore/tables/CREDIT_1.CSV')...
- 5916 Views
- 3 replies
- 1 kudos
- 1 kudos
Thanks to @Werner Stinckens for the answer.I understood that I have to use spark to read data from clusters.
- 1 kudos
- 7400 Views
- 3 replies
- 4 kudos
Resolved! Column name, starting with a number
Hi,I see it is possible to start a column name with a number, like `123_test`And store in a hive table with a location in delta.On that documentation https://www.stitchdata.com/docs/destinations/databricks-delta/reference#transformations--column-nami...
- 7400 Views
- 3 replies
- 4 kudos
- 4 kudos
ha ha, yes, I try to find back the right page in DB documentation. If you have it, please, share.
- 4 kudos
- 3558 Views
- 2 replies
- 4 kudos
Resolved! Cache Select on Temp Table?
How might I cache a temp table?The documentation suggests it is possible: https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-cache.htmlConsider the following on DBR 10.5 and Spark 3.2.1:```%pythondf.createOrReplaceTempView("chan...
- 3558 Views
- 2 replies
- 4 kudos
- 7598 Views
- 5 replies
- 2 kudos
Get current date as string in databricks using scala
I want to get current date in scala as a string for example today current date is 3rd jan, want to store it as a new variable dynamically as below, how to get it.val currdate : String = “20220103”when I am using val currdate = Calendar.getInstance.ge...
- 7598 Views
- 5 replies
- 2 kudos
- 2 kudos
Hey @Vibhor Sethi Hope you are well!Thank you for posting your question and letting us know that you were able to resolve the issue. Would you be happy to mark it as the best solution? It would be really helpful for the other members too.Cheers!
- 2 kudos
- 4258 Views
- 2 replies
- 5 kudos
An error occurred while calling o303.mount: Operation failed: "This request is not authorized to perform this operation
Hi Team,We are unable to mount storage container in below scenario We created Gen 2 using VNet and added firewall restrictions (i.e allow trusted sources)And deployed Data bricks workspace with out VNet injection. Is it possible to add databricks pub...
- 4258 Views
- 2 replies
- 5 kudos
- 5 kudos
Hey @Sailaja B Hope everything is great!Does Hubert's response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?Thanks!
- 5 kudos
- 3500 Views
- 3 replies
- 1 kudos
Resolved! I can't access to my account.
I can't access to my account.This acccount was created today(not community, after 14 days trial it will chargable)when I'm try to access my account it gives meInvalid email address or passwordNote: Emails/usernames are case-sensitiveI tried to reset ...
- 3500 Views
- 3 replies
- 1 kudos
- 1 kudos
I got a reset link from the community. Actually the problem was with my username ,it did not identify a character within my username which was my email id.
- 1 kudos
- 4355 Views
- 1 replies
- 2 kudos
How to install JAR libraries from ADLS? I'm having an error
I mounted the ADLS to my Azure Databricks resource and I keep on getting this error when I try to install a JAR from a container:Library installation attempted on the driver node of cluster 0331-121709-buk0nvsq and failed. Please refer to the followi...
- 4355 Views
- 1 replies
- 2 kudos
- 19616 Views
- 6 replies
- 6 kudos
Resolved! Spark Driver Out of Memory Issue
Hi, I am executing a simple job in Databricks for which I am getting below error. I increased the Driver size still I faced same issue. Spark config :from pyspark.sql import SparkSessionspark_session = SparkSession.builder.appName("Demand Forecasting...
- 19616 Views
- 6 replies
- 6 kudos
- 6 kudos
I am getting the above issue while writing a Spark DF as a parquet file to AWS S3. Not doing any broadcast join actually.
- 6 kudos
- 3396 Views
- 1 replies
- 2 kudos
Resolved! Best way to encrypt PII data
Hi guys, I have around 600GB per load, in you opnion, what is the best way to encrypt PII data in terms of performance ? (lib, cluster type, etc.)Thank youWilliam
- 3396 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @William Scardua please check if the blog helps you.https://databricks.com/blog/2020/11/20/enforcing-column-level-encryption-and-avoiding-data-duplication-with-pii.html
- 2 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 493 | |
| 349 | |
| 287 |
