Databricks Community

The problem was fixed by this importfrom pyspark.sql import functions as F then using F.lit() instead of F.col.withColumn('ingestion_date', F.lit(folder_date)) Sorry code formatting is not working at the moment.

Hi @minhhung0507,The VACUUM command on a Delta table does not delete the _delta_log folder, as this folder contains all the metadata related to the Delta table. The _delta_log folder acts as a pointer where all changes are tracked. In the event that ...

To install dependencies from a private GitHub repository in a Databricks Asset Bundle, you need to set up the GitHub user and token in the resource.yml file and authenticate the GitHub package. Here are the steps: Generate a GitHub Personal Access T...

  To reference the HTTP Path using substitutions in Databricks Asset Bundles and job clusters, you can use the variables section in your databricks.yml configuration file In your databricks.yml file, you can define a variable for the HTTP Path. For e...

Hi @Filippo To prevent privilege escalations, only a metastore admin can transfer ownership of a view, function, or model to any user, service principal, or group in the account. Current owners and users with the MANAGE privilege are restricted to tr...

It is possible to create tasks that run on serverless clusters using dbx. Also, please note that Databricks recommends that you use Databricks Asset Bundles instead of dbx by Databricks Labs. See What are Databricks Asset Bundles and Migrate from dbx...