Spark read /write jdbc netsuite TIMESTAMP error NullPointerException

KristiLogos
Contributor

‎07-10-2025 02:00 AM - edited ‎07-10-2025 02:06 AM

I've connected to Netsuite via Suite Analytics JDBC connection in a Databricks cluster. I'm seeing a NullPointerException because of the lastmodifieddate column in my netsuite classification table (ie. if I do Select * from classification I see the NullPointerException and I've narrowed it down to seeing this issue because of the


except Exception as e:
print("\nERROR: An exception occurred.")
# The full error will be raised, giving you the complete stack trace.
raise e

 

Error

SparkException: Job aborted due to stage failure: Task 0 in stage 31.0 failed 4 times, most recent failure: Lost task 0.3 in stage 31.0 (TID 44) (10.21.40.199 executor 0): java.lang.NullPointerException at com.netsuite.jdbc.base.hk.a(oajc:654) at com.netsuite.jdbc.base.dj.c(oajc:494) at com.netsuite.jdbc.base.ax.a(oajc:1926) at com.netsuite.jdbc.base.fv.getTimestamp(oajc:5211) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$makeGetter$15(JdbcUtils.scala:521) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$makeGetter$15$adapted(JdbcUtils.scala:520) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anon$1.getNext(JdbcUtils.scala:385) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anon$1.getNext(JdbcUtils.scala:366) at  scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460) at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:32) at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1139) at com.databricks.photon.NativeRowBatchIterator.hasNext(NativeRowBatchIterator.java:44) at 0xc7c5392 <photon>.HasNext(external/workspace_spark_3_5/photon/jni-wrappers/jni-row-batch-iterator.cc:50)

-> 90 raise e
File  line 74 66 print(f"\nAttempting to write data to CSV at: {output_path_csv}") 67 # df_cleaned = df.na.fill('') 68 # print('cleaned the df') 70 df.write \ 71 .format("csv") \ 72 .option("header", "true") \ 73 .mode("overwrite") \ ---> 74 .save(output_path_csv) 76 print(f"SUCCESS: Wrote data to CSV.") 78 # # --- Write to Delta Table --- 79 # print(f"\nAttempting to write data to Delta table at: {output_path_delta}") 80 # df.write \ (...) 83 # .save(output_path_delta) 84 # print(f"SUCCESS: Wrote data to Delta table.")
0 Kudos