topic Re: How to make spark-submit work on windows? in Data Engineering

How to make spark-submit work on windows?

weldermartins — Fri, 03 Feb 2023 21:04:07 GMT

I have Jupyter Notebook installed on my machine working normally. I tested running a Spark application by running the spark-submit command and it returns the message that the file was not found. What do you need to do to make it work?

Below is a file with a simple example.

from pyspark.sql import SparkSession
from pyspark.sql.functions import *
 
if __name__ == "__main__":
  spark = SparkSession.builder.appName("Exemplo").getOrCreate()
 
 arqschema = "id INT, nome STRING,status STRING, cidade STRING,vendas INT,data STRING"
 
 despachantes = spark.read.csv("C:\test-spark\despachantes.csv",header=False, schema=arqschema)
 
  calculo = despachantes.select("date").groupBy(year("date")).count()
 
  calculo.write.format("console").save()
 
 
 
  spark.stop()

Re: How to make spark-submit work on windows?

weldermartins — Tue, 07 Feb 2023 01:38:17 GMT

I managed to resolve. It was java and python incompatibility with the Spark version I was using.

I will create a video, explaining how to use Spark without Jupyter notebook.

Re: How to make spark-submit work on windows?

Debayan — Sun, 05 Feb 2023 18:20:21 GMT

Hi, yet this is not tested in my lab, but could you please check and confirm if this works: https://stackoverflow.com/questions/37861469/how-to-submit-spark-application-on-cmd