Hi @Luiz Carneiro​ ,

Could you split your Spark's actions into more CMD (paragraphs) and run one at a time to check where it could be taking the extra time. Also, Pandas only runs on your driver. Have you try to use Python or Scala APIs instead? in case you are familiar with Pandas APIs, then you can use Koalas. Docs here https://koalas.readthedocs.io/en/latest/