Thank you Bhardwaj for checking the issue.

Below is the configuration I am using, Please let me know if required more details.

If I use read CSV, it is working and I am facing an issue if I create a data frame manually.

I installed Spark in my local system.

Configuration :

[('spark.app.name', 'oracle_queries'),

 ('spark.driver.extraJavaOptions',

 '-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED'),

 ('spark.master', 'local[4]'),

 ('spark.driver.port', '50219'),

 ('spark.app.startTime', '1671929660604'),

 ('spark.executor.id', 'driver'),

 ('spark.app.id', 'local-1671929663025'),

 ('spark.sql.warehouse.dir', 'file:/C:/softwares/git/pyspark/hive'),

 ('spark.sql.catalogImplementation', 'hive'),

 ('spark.rdd.compress', 'True'),

 ('spark.executor.extraJavaOptions',

 '-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED'),

 ('spark.app.submitTime', '1671929660216'),

 ('spark.serializer.objectStreamReset', '100'),

 ('spark.submit.pyFiles', ''),

 ('spark.submit.deployMode', 'client'),

 ('spark.driver.host', 'DESKTOP-test.attlocal.net'),

 ('spark.ui.showConsoleProgress', 'true')]

CSV Code :

source_path="C:\\softwares\\git\\pyspark\\source\\emp.csv"

emp=spark.read.csv(source_path,header=True,inferSchema=True)

emp.show()

Output:

+-----+------+---------+----+-----------+----+----+------+

|empno| ename| job| mgr| hiredate| sal|comm|deptno|

+-----+------+---------+----+-----------+----+----+------+

| 7369| SMITH| CLERK|7902|17-DEC-1980| 800|null| 20|

| 7499| ALLEN| null|7698|20-FEB-1981|1600| 300| 30|

| 7521| WARD| SALESMAN|7698|22-FEB-1981|1250| 500| 30|

| 7566| JONES| MANAGER|7839| 2-APR-1981|2975|null| 20|

| 7654|MARTIN| SALESMAN|7698|28-SEP-1981|1250|1400| 30|

| 7698| BLAKE| MANAGER|7839| 1-MAY-1981|2850| 0| 30|

| 7782| CLARK| MANAGER|7839| 9-JUN-1981|2450| 0| 10|

| 7788| SCOTT| ANALYST|7566|09-DEC-1982|3000| 0| 20|

| 7839| KING|PRESIDENT| 0|17-NOV-1981|5000| 0| 10|

| 7844|TURNER| SALESMAN|7698| 8-SEP-1981|1500| 0| 30|

| 7876| ADAMS| CLERK|7788|12-JAN-1983|1100| 0| 20|

| 7900| JAMES| CLERK|7698| 3-DEC-1981| 950| 0| 30|

| 7902| FORD| ANALYST|7566| 3-DEC-1981|3000| 0| 20|

| 7934|MILLER| CLERK|7782|23-JAN-1982|1300| 0| 10|

+-----+------+---------+----+-----------+----+----+------+