java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class Error on writing a dataframe to a SQL DB in azure
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-28-2022 04:43 AM
I have to write the extracted data from XML to DB , i am using Dataframe for transformation and trying to load that to DB.
I have installed these libraries,
com.databricks:spark-xml_2.12:0.15.0
com.microsoft.azure:spark-mssql-connector_2.11_2.4:1.0.2
and my cell has the below code ,
df_Driver.write.format("com.microsoft.sqlserver.jdbc.spark").option("url", jdbcUrl).mode("overwrite").option("dbtable", "table_name").option("user", jdbcUsername).option("password", jdbcPassword).save()
but it throws an error as
java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class
Py4JJavaError: An error occurred while calling o1055.save.
: java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class
at com.microsoft.sqlserver.jdbc.spark.DefaultSource.<init>(DefaultSource.scala:32)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:809)
at org.apache.spark.sql.DataFrameWriter.lookupV2Provider(DataFrameWriter.scala:983)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:293)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:258)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
at py4j.Gateway.invoke(Gateway.java:295)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging$class
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader.loadClass(ClassLoaders.scala:151)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 21 more
Please help me with this issue. TIA
- Labels:
-
Xml
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-29-2022 04:24 AM
To write to Azure SQL you don't need to install the driver just use:
jdbcUrl = "jdbc:sqlserver://{0}:{1};database={2};user={3};password={4}".format(jdbcHostname, jdbcPort, jdbcDatabase, username, password)
(df
.write
.jdbc(jdbcUrl, "table_name"))
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-29-2022 08:42 AM
Thanks, its working now. But I have got another issue that the data in dataframe save as a new table instead of inserting the data into existing table in DB.
df
.write
.format("jdbc")\
.option("url", jdbcUrl)\
.mode("overwrite")\
.option("dbtable","Driver_DVCSD")\
.option("user", jdbcUsername)\
.option("password", jdbcPassword).save()
The dataframe hold the XML data and the table has audit columns created_date and created_by which is been dropped when the data is written from the dataframe to table.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2022 03:35 AM
just add this audit columns as new standard ones in dataframe using .withColumn
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-30-2022 01:41 PM
Hi @shafana Roohi Jahubar Could you please check the table name in your write command? If it is the same, overwrite mode should overwrite that table.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2022 03:07 AM
Hi @shafana Roohi Jahubar
Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!

