topic Re: What are the different ways to pull the log data from Splunk to Databricks? in Get Started Discussions

What are the different ways to pull the log data from Splunk to Databricks?

Arch_dbxlearner — Wed, 03 Jan 2024 06:27:43 GMT

Hi,

I have recently started Splunk Integration with Databricks. Basically I am trying to ingest the data from Splunk to Databricks. I have gone through the documentation regarding Splunk Integration. There are some basic information about the integration but I am looking for something else which is not available in the document.

I would like to know the ways which are possible with data ingestion from Splunk.

- Can we send the log data directly from Splunk to Databricks?
- Do any intermediate tools/api required for the communication? If it's mandatory, then what are the possible tools/api?
- Splunk have event data and metric data. Is it possible to pick both these type of data by Databricks?

Could anyone please help me out with these queries?

Re: What are the different ways to pull the log data from Splunk to Databricks?

Arch_dbxlearner — Wed, 03 Jan 2024 10:53:12 GMT

Thankyou @Retired_mod for the clear explanation.

I have another set of questions. Please provide your suggestion on these as well.

I have gone through a tool called "Open Telemetry", which collects the logs, metrics, etc.,. Can this tool be used as an intermediate between Databricks and Splunk?
Instead of having the Splunk to collect the event data from our application, is there any way to send the application/system logs directly to Databricks?
Is it possible to send the complete raw data and/or customised filter data from application to Databricks?

Thankyou!

Re: What are the different ways to pull the log data from Splunk to Databricks?

Arch_dbxlearner — Fri, 05 Jan 2024 10:04:43 GMT

Thankyou @Retired_mod .

Currently I am planning to check the possible ways to send the sample data to Databricks from Splunk without any third party tool's intervention.

Let me play around with those and get back to you if I need any guidance at any place.

Thanks once again!

Re: What are the different ways to pull the log data from Splunk to Databricks?

Arch_dbxlearner — Tue, 09 Jan 2024 11:41:25 GMT

@Retired_mod

You have mentioned that Databricks Add-on for Splunk, is bidirectional. Do we need to install this app on Databricks itself, to fetch the data from Splunk?

I tried to check this add-on on Databricks Marketplace but I could not find this. Can you please let me know the process to install the add-on?

I am looking to push the data from Splunk to Databricks and do some process and activity on daily basis. Could you please suggest me on this?

Re: What are the different ways to pull the log data from Splunk to Databricks?

Arch_dbxlearner — Wed, 10 Jan 2024 05:36:37 GMT

Hi @Retired_mod

Can you please guide me on this?

Re: What are the different ways to pull the log data from Splunk to Databricks?

Arch_dbxlearner — Mon, 22 Jan 2024 08:34:46 GMT

Hi @Retired_mod

I have gone through the github page of Databricks - Splunk integration. In the architecture diagram it is mentioned with 3 sections.

Setting up Databricks add-on for Splunk
Configuring Splunk DB Connect app
Creating Notebook for push and pull data from Splunk

My requirement is only to fetch the data from Splunk and put in Databricks to do analysis and create dashboard. so I assumed, for my usecase 3rd option is the method to be done and I have followed the github page - here.

I have installed the databricks cli, created a secret scope to save Splunk credentials. Now I am working on the Notebook part to create Python code to fetch data.

Am I going in the right way?
Is it sufficient to follow and setup only the 3rd method, if I need only one way communication which is from Splunk to Databricks?
Also I am referring this document here for Python script of my Notebook. Can I use this?

Could you please guide me on this to proceed further?

Re: What are the different ways to pull the log data from Splunk to Databricks?

Arch_dbxlearner — Tue, 23 Jan 2024 08:03:55 GMT

@Retired_mod Can you please guide me on this?

Re: What are the different ways to pull the log data from Splunk to Databricks?

Arch_dbxlearner — Mon, 29 Jan 2024 11:32:51 GMT

Hi @Retired_mod

Still I am awaiting for your response on this. Can you please go through my above reply and guide me accordingly?

Thank you!

Re: What are the different ways to pull the log data from Splunk to Databricks?

SainnathReddy_M — Fri, 19 Jul 2024 17:22:26 GMT

Hi @Arch_dbxlearner Did you done integration with splunk if yes can you please help