Databricks Community

Wayne · 02-21-2025

Try to follow the instructions to create custom IAM role for EC2 instance in Databricks clusters, but I can't find the instance profile ARN on the role. If I create a regual IAM role on EC2, I can find both role ARN and instance profile ARN.https://d...

Wayne · 11-30-2023

This is from Spark Event log on Event SparkListenerSQLExecutionStart.How to flatten the sparkPlanInfo struct into an array of the same struct, then later explode it. Note that the element children is an array containing the parent struct, and the lev...

Wayne · 11-08-2023

Reference:PY SDK I can't find any API directly to download the logs. Did I miss anything?Otherwise, I can download via DBFS cluster logs location.

Wayne · 10-02-2023

case #00378268 I passed Spark Developer associate exam about 6 month ago with great experience. However, this time the proctor did not even bother to show up to start the exam - checking ID, the room and the surroundings. Somehow, I was able to st...

Wayne · 09-30-2023

I got some issues with Databricks online certification. I filed twice at (https://help.databricks.com/s/contact-us?ReqType=training), but did not get any confirmation emails.@Cert-Team

Wayne · 10-06-2023

That will be great.

Wayne · 10-03-2023

Thanks a lot. Is there any way to continue the test outside the USA? Tried to take the exam before my vacation.

Wayne · 10-02-2023

@Kaniz @Cert-Team Finally figured out why my request got dropped silently - I included a link in the form. Please indicate no link in the form submission section. Thanks a lot.

Wayne · 05-05-2022

No error, just seeing the EXPAND DISK in cluster event logs. This is just a regular spark application. I am not sure if the cloud storage matters - a spark application uses it as input and output.

Databricks Community

User Stats

User Activity

AWS custom role for Databricks clusters - no instance profile ARN

How to flatten a nested recursive JSON struct to a list of struct

Download Spark Driver logs and event logs from Databricks using API

My exam has suspended by an unprofessional proctor- need help to reschedule

Databricks Contact us form not working

Re: My exam has suspended by an unprofessional proctor- need help to reschedule

Re: My exam has suspended by an unprofessional proctor- need help to reschedule

Re: Databricks Contact us form not working

Re: Question Submitted How to tune a job to avoid paying extra cost for EXPAND DISK? Is it due to the shuffle or data skew? Is there a way to configure the workers with larger disk? If not having EXPAND DISK, it will fail since no space left on the disk.