topic How to get data from Splunk on daily basis? in Get Started Discussions

How to get data from Splunk on daily basis?

Arch_dbxlearner — Tue, 16 Jan 2024 10:56:46 GMT

I am finding the ways to get the data to Databricks from Splunk (similar to other data sources like S3, Kafka, etc.,). I have received a suggestion to use the Databricks add-on to get/put the data from/to Splunk. To pull the data from Databricks to Splunk is easy via setting up this add-on at Splunk side.

But to push the data from Splunk to Databricks, I don't find any documentation in setting up the add-on. If anyone can help me with procedure of setting up this add-on at Databricks side, it will helpful for me to proceed on this. I have got another set of procedure to pull the data from Splunk to Databricks via a github document - here

The plan is to send the data from Splunk to Databricks on daily basis and build a dashboards on top those data. As it is daily basis data, it could be high volume of data. I would like to know the limitation of sending the data in the respective tools.

I tried to check in Databricks document, but I could not find any information with respect to the communication with Splunk.

Could anyone please help me on finding the best way to send the Splunk data to Databricks?

Re: How to get data from Splunk on daily basis?

shan_chandra — Tue, 16 Jan 2024 19:40:58 GMT

@Arch_dbxlearner - could you please follow the post for more details. https://community.databricks.com/t5/data-engineering/does-databricks-integrate-with-splunk-what-are-some-ways-to-send/td-p/22048

Re: How to get data from Splunk on daily basis?

Arch_dbxlearner — Wed, 17 Jan 2024 06:03:48 GMT

Hi @shan_chandra ,

Already I have gone through the post which you have shared above. It is mentioned that the add-on is bi-directional so the communication between Splunk and Databricks can be done.

My requirement is the data to be sent from Splunk to Databricks. I need only one directional activity, where the Splunk data to be used in Databricks and do further activity on Databricks.

So my doubt is where the add-on should be installed. I am going to push the data from Splunk to Databricks. I am aware that it requires HEC but ideally where my Databricks add-on should be placed.

The name says that it is "Databricks add-on for Splunk". I would like to know the process to setup this add-on to push the data only from Splunk to Databricks.

Could you please help me on this?

Re: How to get data from Splunk on daily basis?

shan_chandra — Fri, 19 Jan 2024 17:33:20 GMT

@Arch_dbxlearner - we can limit access to the user only to read the data from Splunk into Databricks. Please refer below.

https://github.com/databrickslabs/splunk-integration/blob/master/docs/markdown/Splunk%20DB%20Connect%20guide%20for%20Databricks.md#limit-the-access-of-identities-and-connection-to-particular-users

Re: How to get data from Splunk on daily basis?

hukel — Tue, 11 Jun 2024 10:00:57 GMT

In my experience with the Splunk add-on, it is typically used to pull Databricks data into Splunk, not to push. If the data sets are small then it could probably push as well, but I think you'd have to write some sort of Splunk map loop to issue INSERT statements against Databricks.

It would probably be more manageable to use this approach, https://github.com/databrickslabs/splunk-integration/blob/master/docs/markdown/Databricks%20-%20Pull%20from%20Splunk.md.

This may also provide guidance: https://registry.terraform.io/modules/databricks/examples/databricks/latest/examples/adb-splunk

Re: How to get data from Splunk on daily basis?

hukel — Tue, 29 Oct 2024 16:53:53 GMT

Another idea (if you need to do small lookups, not bulk transfer) .... what about using Splunk's splunk-sdk to create a notebook function that hits Splunk via REST API?