I have a multi-part question around Databricks integration with Splunk?

r_van_niekerk
Databricks Employee
Databricks Employee

Use Case Background

We have an ongoing SecOps project going live here in 4 weeks. We have set up a Splunk to monitor syslogs logs and want to integrate this with Delta. Our forwarder collect the data from remote machines then forwards data to the index in real-time; our indexer processes the incoming stream in real-time and we typically query that data directly in vai the Splunk UI/Search Head.

We would like to provide our end users the ability to store historical logs in Delta; then query those directly logs via the Databricks UI/Notebooks/Databricks SQL.

Question

  1. Whether there are any example notebooks or documentation/tips on Splunk integration with Databricks?
  2. Whether you can query our logs directly via Databricks?

Thank you!