Databricks

BriGuy · ‎11-15-2023

I have created a process that runs a notebook multiple times in parallel with different parameters. This was working quite quickly. However I've added several logging steps that are appending log details to a dataframe then using dataframewriter to append the existing delta table. This seems to significantly slow down the process. I think each appends to the table are blocking other appends.

Is there a best practice for logging into 1 table from multiple processes? If I were using SQL server I'd consider setting up a message queue. It is acceptable for log entries to be inserted out of sequence as we can sort by timestamp but it is essential that all log entries are inserted even if the parent process is terminated.

As the table will only ever be appended to I don't need the time travel functionality if that makes a difference.

Thanks

Kaniz · ‎11-23-2023

Hi @BriGuy, Regarding your Databricks SQL Editor issue, you’re not alone! Several users have faced similar problems.

Here are some steps you can take:

Contact Databricks Support:

I recommend contacting Databricks support. File a support ticket to get personalized troubleshooting steps or solutions for this specific issue.

Workaround:

Some users have found that pasting in code chunks can throw off the tab indentation.
Try manually adjusting the cursor position and hitting the tab key multiple times until it starts wo....

Workspace Settings:

As a workspace admin, you can configure settings related to Databricks SQL access. Check if any specific settings are affecting the tab behaviour.

Visual Studio Code (VS Code) Integration:

If you’re using VS Code with Databricks, ensure that the Databricks extension is up to date.
Sometimes extensions can cause unexpected behavior.

Remember, troubleshooting such issues can be frustrating, but persistence pays off. Hopefully, Databricks support or community feedback will provide a solution soon! 🚀