topic I have a multi-part question around Databricks integration with Splunk? in Data Engineering

I have a multi-part question around Databricks integration with Splunk?

r_van_niekerk — Mon, 07 Jun 2021 18:22:53 GMT

Use Case Background

We have an ongoing SecOps project going live here in 4 weeks. We have set up a Splunk to monitor syslogs logs and want to integrate this with Delta. Our forwarder collect the data from remote machines then forwards data to the index in real-time; our indexer processes the incoming stream in real-time and we typically query that data directly in vai the Splunk UI/Search Head.

We would like to provide our end users the ability to store historical logs in Delta; then query those directly logs via the Databricks UI/Notebooks/Databricks SQL.

Question

Whether there are any example notebooks or documentation/tips on Splunk integration with Databricks?
Whether you can query our logs directly via Databricks?

Thank you!

Re: I have a multi-part question around Databricks integration with Splunk?

aladda — Mon, 21 Jun 2021 20:28:28 GMT

Yes. Please see the following post for details - https://uat-databrickspartner.cs165.force.com/forums/s/question/0D56s00000CxDvqCAF/does-databricks-integrate-with-splunk-what-are-some-ways-to-send-metricslogs-to-splunk

Re: I have a multi-part question around Databricks integration with Splunk?

aladda — Mon, 21 Jun 2021 20:28:46 GMT

The Databricks Add-on for Splunk built as part of Databricks Labs can be leveraged for Splunk integration

It’s a bi-directional framework that allows for in-place querying of data in databricks from within Splunk by running queries, notebooks or jobs so you don’t have to move the data and still have access to it from within. Docs are here - https://github.com/databrickslabs/splunk-integration#Documentation