Hi! General speaking, it's good practice to avoid collect() action unless you absolutely need to, because collect() is action operation that will retrieve all the elements of the RDD/DataFrame/Dataset from all nodes to the driver node. If the dataset is large enough, you might even run into out of memory issues.

In your pipeline, you're reading the data, performing ETL, and then write the data out directly to an object storage (in this case, DBFS), you don't need to add this collect() operation here.