mark_ott
Databricks Employee
Databricks Employee

If you are on a Spark version that supports .option("query", ...), you can do:

python
df = spark.read \ .format("jdbc") \ .option("url", jdbc_url) \ .option("query", "SELECT TOP 10 * FROM Customer") \ .option("user", "xxxx") \ .option("password", "xxx") \ .option("driver", "com.netsuite.jdbc.openaccess.OpenAccessDriver") \ .load() display(df)

But note that some JDBC drivers, including NetSuite, may have restrictions, so review your Spark and JDBC connector documentation and test with preliminary table queries first.

Troubleshooting Common Issues

  • Jar File Location: Confirm the .jar is on all Spark worker nodes and that the cluster recognizes the com.netsuite.jdbc.openaccess.OpenAccessDriver class.

  • NetSuite Permissions: Double-check that the user/role (AccountID, RoleID) has correct access rights to view the "Customer" table.

  • Port and Endpoint: Ensure firewall/network settings allow traffic to the required NetSuite JDBC endpoint and port.

  • Driver Class: Confirm the driver class is spelled correctly and matches your .jar file.

  • Error Messages: Carefully check any exception traceback for specific hints; common errors include authentication failures, missing drivers, or SQL syntax not supported by NetSuite JDBC.

Recommendations

  • Start with .option("dbtable", "Customer") for a basic read.

  • Gradually introduce more complex queries if .option("query", ...) is supported.

  • Review NetSuite JDBC and Spark documentation for compatibility notes.

View solution in original post