How to get data from Splunk on daily basis?

[Arch_dbxlearner]

I am finding the ways to get the data to Databricks from Splunk (similar to other data sources like S3, Kafka, etc.,). I have received a suggestion to use the Databricks add-on to get/put the data from/to Splunk. To pull the data from Databricks to Splunk is easy via setting up this add-on at Splunk side.

But to push the data from Splunk to Databricks, I don't find any documentation in setting up the add-on. If anyone can help me with procedure of setting up this add-on at Databricks side, it will helpful for me to proceed on this. I have got another set of procedure to pull the data from Splunk to Databricks via a github document - here

The plan is to send the data from Splunk to Databricks on daily basis and build a dashboards on top those data. As it is daily basis data, it could be high volume of data. I would like to know the limitation of sending the data in the respective tools.

I tried to check in Databricks document, but I could not find any information with respect to the communication with Splunk.

Could anyone please help me on finding the best way to send the Splunk data to Databricks?

[shan_chandra]

@Arch_dbxlearner  - could you please follow the post for more details.  https://community.databricks.com/t5/data-engineering/does-databricks-integrate-with-splunk-what-are-...

[Arch_dbxlearner]

Hi @shan_chandra ,

Already I have gone through the post which you have shared above. It is mentioned that the add-on is bi-directional so the communication between Splunk and Databricks can be done.

My requirement is the data to be sent from Splunk to Databricks. I need only one directional activity, where the Splunk data to be used in Databricks and do further activity on Databricks.

So my doubt is where the add-on should be installed. I am going to push the data from Splunk to Databricks. I am aware that it requires HEC but ideally where my Databricks add-on should be placed.

The name says that it is "Databricks add-on for Splunk". I would like to know the process to setup this add-on to push the data only from Splunk to Databricks.

Could you please help me on this? 

[shan_chandra]

@Arch_dbxlearner - we can limit access to the user only to read the data from Splunk into Databricks. Please refer below. 

https://github.com/databrickslabs/splunk-integration/blob/master/docs/markdown/Splunk%20DB%20Connect...

[hukel]

In my experience with the Splunk add-on,  it is typically used to pull Databricks data into Splunk, not to push.   If the data sets are small then it could probably push as well, but I think you'd have to write some sort of Splunk map loop to issue INSERT statements against Databricks.

It would probably be more manageable to use this approach,  https://github.com/databrickslabs/splunk-integration/blob/master/docs/markdown/Databricks%20-%20Pull....

This may also provide guidance:  https://registry.terraform.io/modules/databricks/examples/databricks/latest/examples/adb-splunk

[hukel]

Another idea (if you need to do small lookups, not bulk transfer) .... what about using Splunk's splunk-sdk to create a notebook function that hits Splunk via REST API?