Databricks

laurencewells · 05-31-2022

Hi All, We are trying to use the Spark 3 structured streaming feature/option ".option('cleanSource','archive')" to archive processed files. This is working as expected using the standard spark implementation, however does not appear to work using aut...

laurencewells · 01-11-2022

Hi All, Hoping you can help. I am looking to set up a custom logging process that captures application ETL logs and Streaming logs I have set up multiple custom logging appenders using the guide here: https://kb.databricks.com/clusters/overwrite-log4...

laurencewells · 06-06-2022

Apologies i meant the cleanSource option is part of native spark 3.0, therefore if it doesn't work in autoloader i would expect to see that its not supported in the docs. Or Error if its included in the code. Currently it accepts it and does nothing....

laurencewells · 06-01-2022

Yes, but I'm guessing as part of the native spark implementation for file streaming i think it should specify either way?

laurencewells · 05-31-2022

Hey unfortunately not. That is a blog about log4j vulnerability. Fortunately databricks are upgrading to log4j v2 in runtime 11 so its a mute point now. v2 has much better filters etc

Databricks

User Stats

User Activity

Autoloader and "cleanSource"

Log4J Custom Filter Not Working

Re: Autoloader and "cleanSource"

Re: Autoloader and "cleanSource"

Re: Log4J Custom Filter Not Working