- 13886 Views
- 7 replies
- 5 kudos
Resolved! Job fails after runtime upgrade
I have a job running with no issues in Databricks runtime 7.3 LTS. When I upgraded to 8.3 it fails with error An exception was thrown from a UDF: 'pyspark.serializers.SerializationError'... SparkContext should only be created and accessed on the driv...
- 13886 Views
- 7 replies
- 5 kudos
- 5 kudos
Adding to @Sean Owen comments, The only reason this is working is that the optimizer is evaluating this locally rather than creating a context on executors and evaluating it.
- 5 kudos
- 10144 Views
- 1 replies
- 1 kudos
Resolved! query array in SQL
Hello I have a databricks question I was not able to answer myselfI have this queryselect count(*) from tablewhere object[0].value is not null and object[0].value.value1 = "s"and created_year = 2022 and created_month = 7 and created_day = 4you can se...
- 10144 Views
- 1 replies
- 1 kudos
- 1 kudos
SELECT count(*)FROM ( SELECT explode(mmycolumn) FROM table WHERE created_year = 2022 and created_month = 7 and created_day = 5)WHERE col.field is not null and col.field.field! = "signal"
- 1 kudos
- 2867 Views
- 0 replies
- 2 kudos
Delta Table Merge Operation logs Output is not correct number of updated records?
Hi all,I am performing merge operation on my delta table in spark. I have existing delta table , it already has some records. Now I created another dataframe of csv file, and added one new record and updated one records in that. Please check below sn...
- 2867 Views
- 0 replies
- 2 kudos
- 11111 Views
- 7 replies
- 2 kudos
Resolved! Unable to read file from S3
I tried to read a file from S3, but facing the below error:org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 53.0 failed 4 times, most recent failure: Lost task 0.3 in stage 53.0 (TID 82, xx.xx.xx.xx, executor 0): com...
- 11111 Views
- 7 replies
- 2 kudos
- 2 kudos
Which DBR version are you using? Could you please test it with a different DBR version probably DBR 9.x?
- 2 kudos
- 4238 Views
- 4 replies
- 6 kudos
Resolved! why this not able to go through?
https://textdoc.co/index.php/UFEQdwxWn60LtOVfError:https://textdoc.co/index.php/3JisnHKGkvLIaAOF
- 4238 Views
- 4 replies
- 6 kudos
- 6 kudos
It would be best if you used Databricks ML runtime (in cluster settings), not the standard one.
- 6 kudos
- 3122 Views
- 2 replies
- 0 kudos
Resolved! Save data from Spark DataFrames to TFRecords
https://docs.microsoft.com/en-us/azure/databricks/_static/notebooks/deep-learning/tfrecords-save-load.htmlI could not run the Cell # 2java.lang.ClassNotFoundException: --------------------------------------------------------------------------- Py4JJ...
- 3122 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @THIAM HUAT TAN,Which DBR version are you using? are you using the ML runtime?
- 0 kudos
- 3387 Views
- 1 replies
- 1 kudos
Unity Catalog will allow you to bring your own HMS
Anyone know more about how the Unity Catalog will allow you to bring your own HMS (eg Glue)?Will this be treated as a separate 'catalog', which you can access but you can't use the other features of Unity Catalog on eg ABAC etcAny reading on this top...
- 3387 Views
- 1 replies
- 1 kudos
- 1 kudos
@Kunal Gaurav yes, it is treated as a synthetic catalog. You can query it by using the convention "hive_metastore" as the catalog name. i.e. SELECT * FROM hive_metastore.schema_name.table_nameThis will work for internal HMS, external HMS and Glue.Yo...
- 1 kudos
- 5223 Views
- 2 replies
- 2 kudos
Resolved! How would you integrate Unity Catalog with existing Data governance solutions like Collibra?
- 5223 Views
- 2 replies
- 2 kudos
- 2 kudos
@Prashanth P We offer a fully featured REST API with Unity Catalog that provides the ability to CRUD objects such as catalogs/schemas/tables/acls/lineage etc.Companies like Colliba/Alation/MS Purview etc use these in middleware integrations to integ...
- 2 kudos
- 2814 Views
- 2 replies
- 1 kudos
Dbt vs databricks sql
Can I use db sql to replace dbt?
- 2814 Views
- 2 replies
- 1 kudos
- 3840 Views
- 2 replies
- 1 kudos
Parallelize spark jobs on the same cluster?
Whats the best way to parallelize multiple spark jobs on the same cluster during a backfill?
- 3840 Views
- 2 replies
- 1 kudos
- 1 kudos
In the past I used direct multi-threaded orchestration inside of driver applications, but that was prior to Databricks supporting multi-task jobs.If you create a job through the workflows tab, you can set up multiple notebooks, python, or jar tasks t...
- 1 kudos
- 4505 Views
- 3 replies
- 3 kudos
Resolved! Databricks Workflows
How do I get started with Databricks Workflows?
- 4505 Views
- 3 replies
- 3 kudos
- 6335 Views
- 4 replies
- 4 kudos
Resolved! SQL Serverless Endpoint failing to start with Instance Profile
Hi allSuper stoked about the PP of SQL Serverless, but it does seem that the instance profile Im using doesnt have the required trust relationship for it to work with the Sererless Endpoint. Altough on "classic" mode it works fine. Does Serverless re...
- 6335 Views
- 4 replies
- 4 kudos
- 4 kudos
Thank you for sharing your valuable solution, it's work properly.mymilestonecard
- 4 kudos
- 10758 Views
- 2 replies
- 7 kudos
Resolved! Library installation fails with mirror sync issue
While trying to install ffmpeg package using an init script on Databricks cluster, it fails with the below error.Init script:#! /bin/bash set -e sudo apt-get update sudo apt-get -y install ffmpegError message:E: Failed to fetch http://security.ubuntu...
- 10758 Views
- 2 replies
- 7 kudos
- 7 kudos
Cause: The VMs are pointing to the cached old mirror which is not up-to-date. Hence there is a problem with downloading the package and it's failing. Workaround: Use the below init script to install the package "ffmpeg". To revert to the original lis...
- 7 kudos
- 11103 Views
- 7 replies
- 4 kudos
Resolved! Retrieve job id and run id from scala
I need to retrieve job id and run id of the job from a jar file in Scala.When I try to compile below code in IntelliJ, below error is shown.import com.databricks.dbutils_v1.DBUtilsHolder.dbutils object MainSNL { @throws(classOf[Exception]) de...
- 11103 Views
- 7 replies
- 4 kudos
- 4 kudos
Maybe its worth going through the Task Parameter variables section of the below dochttps://docs.databricks.com/data-engineering/jobs/jobs.html#task-parameter-variables
- 4 kudos
- 6712 Views
- 1 replies
- 2 kudos
Resolved! Databricks jobs create API throws unexpected error
Databricks jobs create API throws unexpected errorError response :{"error_code": "INVALID_PARAMETER_VALUE","message": "Cluster validation error: Missing required field: settings.cluster_spec.new_cluster.size"}Any idea on this?
- 6712 Views
- 1 replies
- 2 kudos
- 2 kudos
Could you please specify num_workers in the json body and try API again.Also, another recommendation can be configuring what you want in UI, and then pressing “JSON” button that should show corresponding JSON which you can use for API
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
