Databricks Community

Hello,I made some transform on pyspark.sql.Column object: file_path_splitted=f.split(df[filepath_col_name],'/') # return Column object file_name = file_path_splitted[f.size(file_path_splitted) - 1] # return Column object Next I used variable "file_na...

Spark write method (df.write.parquet) to parquet files is transactional. I mean after write is sucessfull file _SUCCESS is created in path where parquet files was loaded.Is it possible to configure AutoLoader to load parquet files only in case when w...

Hello,I have some trouble with AutoLoader. Currently we use many diffrent source location on ADLS to read parquet files and write it to delta table using AutoLoader. Files in locations have the same schema.Every things works fine untill we have to ad...