@cfregly​  @Vida Ha​ 

I'm having trouble with the same "task not serializable" error when calling foreachPartition.

My code looks like:

myDF
.foreachPartition { (rddpartition: Iterator[Row]) =>
  val url = "jdbc:sqlserver://<myurl>"
  val un = dbutils.secrets.get("my-secret-scope", "my-username")
  val pw = dbutils.secrets.get("my-secret-scope", "my-password")
  val connection = DriverManager.getConnection(url, un, pw)
  var statement = connection.createStatement()
  rddpartition.foreach { (row: Row) =>
    statement.addBatch("INSERT INTO dbo.Table(Field1, Field2) VALUES (${row.get(0)}, ${row.get(1)})")
  }
  statement.executeBatch()
  connection.close()
}

The above code results in the error "org.apache.spark.SparkException: Task not serializable"

When I modify the code to use the username and password as strings instead as shown below, it works just fine. Are you aware of a way to get around the serializability of dbutils.secrets.get?

DriverManager.getConnection(url, "<username string>", "<password string>")