Databricks Community

This repo has examples that you can use in your Databricks workspace for FastAPI and Streamlit.  I recommend only using these for development or lightweight use cases. 

 use a separate a Databricks Git folder mapped to a remote Git repo for each user who works in their own development branch .Run Git operations on Databricks Repos | Databricks on AWS

Hi @Krubug, Optimizing SQL queries can significantly improve performance and reduce costs. Let’s explore some techniques to enhance the query you’ve provided: Minimize Wildcard Characters: The use of wildcard characters (such as % and _) in SQL ...

Hi @Kaizen, Thank you for sharing your experience with the community! It’s essential to raise awareness about issues like this. Let’s dive into some troubleshooting steps and potential workarounds for the mesa packages installation problem. Conf...

Then removing the "_commited_" file stops spark form reading in the other files

Hi @385653  It works from single user clusters using dbfs path.  On Shared clusters, please set the spark conf at the notebook level where you would convert the json content into base64 string. This is a workaround as shared clusters do not support d...

Hi @DE-cat, The message you’re encountering about the deprecated format is related to the Databricks job configuration. Let’s break it down: Deprecated Format: The format used by your job has been considered outdated since 2016.Databricks recomm...

Hi @Madalian, Let’s address the issue you’re facing while converting your Parquet files from the Bronze layer to Delta live tables in the Silver layer. Column Addition: It appears that you’re trying to add a current timestamp column named SILVER_...

Hi @nachog99, Working with VCF (Variant Call Format) files in Spark can be done using various approaches. Let’s explore some options: Glow Library: You’ve already been using the Glow library (Maven: io.projectglow:glow-spark3_2.12:1.2.1) to read ...

Hi @JonLaRose, When you create an external table in Unity Catalog by referencing an existing Delta Lake table, the behavior you’ve observed is indeed expected. Let’s delve into the reasons behind this: Unity Catalog and Delta Lake: Unity Catalog...

Hi @LiLO, You can achieve this using PySpark and the integrated connection between Databricks and Azure Blob storage. Let’s break down the steps: Read the Flat File from Azure Blob Storage: Use the integrated connection to read the flat file dire...

Hi @manoj_2355ca, To manage your Personal Access Tokens (PATs) effectively, here are some key points: Expiration of PATs: PATs have an expiration date, but the good news is that if you regenerate or remove a PAT, your deployment groups will conti...