Here are some of the questions and answers from the September 28 Office Hours. Join our October 12 session to get your questions answered!

Q: Is there any benefit to transforming data via dataframes in pyspark vs using databricks SQL?

A: Performance-wsie they will match But usability wise SQL will be the better option due to its wide usability and readability

Q: I have a pyspark streaming script written in databrick and the script keeps dying. I have a hunch that its the cluster its running on. What is the proper way to run streaming in databricks? What is the proper to monitor spark streaming?

A: Great question. 1) I would check with your Admin to see if you have the ability to attach to the cluster. 2) If you can attach to the cluster, check the

configuration of the cluster. It may be the case that the cluster has very little memory. So streaming usually spark streaming usually works better if you have more performance, more memory-optimized machines 3) If none of these things are the case, check the auto termination time. Could be that the

query is not getting attached, leading to your notebook not getting attached to the cluster, hence it is getting terminated. 4) I would also recommend using Autoloader or Delta Live Tables instead which are better ways to handle and monitor streaking.

Q: I can access data on dbfs, but can't see it in the UI. Why?

A: This may be due to a lack of object privileges from your workspace admin. Please check out this doc for more information on object privileges.

Q: Is there a way to create a template notebook so that when starting to create a notebook, we can use it, which could contain a structure of the notebook like secion: read, transform, write, and other required documentation. Even better would be if Databricks can suggest to create a notebook from existing template (custom made notebook template).

A: We currently do not have a feature to create template notebooks, but you can use the notebook in HTML format to mark your cell for what usages you have the cell written and differentiate. This doc should help. You can also check out our Notebook gallery. This is a great idea to suggest to our Idea Portal.

Q: We've been having a regular issue with a production workflow. On some runs, we encounter the error Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 54 in stage 127.1 failed 4 times, most recent failure: Lost task 54.3 in stage 127.1 (TID 7380) (10.149.195.104 executor 8): ExecutorLostFailure (executor 8 exited caused by one of the running tasks) Reason: worker lost. When retrying the workflow it always succeeds. Do you know of a permanent fix for this issue? Additionally, the job is running on i3.4xlarge. Would z1d work better in this case?

A: You need to check the cluster usages. Maybe try ganglia matrices and monitor the log from there. Are you seeing any GC failure? You may need to pull more logs for the same if moving to memory consuming cluster does not help. This may be due to it trying to consume more memory from your cluster, so you should consider trying to increase it to a memory-consuming instance type. As for how the job is running, z1d should work, though even this depends upon the job you are running. Monitoring the ganglia will give you more insight how the cluster and (executors) level is behaving.

Q: Recommendation options for Auto Loader: Use case is the customer places many data files in the raw zone in one directory, the files are overwritten once a day... many data sets are in this directory.... I assume that I should create many Auto Loader jobs specific to each data set (file). How can Auto Loader look into a directory and pull selected files... One point of confusion is that Auto Loader gets notified when a new file lands BUT there are different data sets landing... so how do I get the right Auto Loader process to fire??

A: If I am understanding correctly, your use case is asking about a file name pattern matching, This is not currently supported, each data set is expected to be in a separate folder. You can find the options for to help with what you are trying to achieve, here.

Q: A question about Spark. Are Vectorized Spark Native functions (through pySpark), always faster than UDF using Scala?

A: Unfortunately there is no one answer that can fit in this case. Because as you might know partitioning and data key distribution... In general yes vectorized read is expected to outperform but this can be affected by other circumstances.

Q: When using azure private link, why is it suggested to use a different vnet for front end private endpoint? If we use a different vnet for front end private endpoint, should that vnet be shared across all workspaces in the region? Or are you suggesting 2 vnets per workspaces that use private link?

A: It is applied to per MG. This is due to a back end need for Secure cluster connectivity. You can find more additional details here with architecture