01-02-2024 10:27 PM
Hi,
I have recently started Splunk Integration with Databricks. Basically I am trying to ingest the data from Splunk to Databricks. I have gone through the documentation regarding Splunk Integration. There are some basic information about the integration but I am looking for something else which is not available in the document.
I would like to know the ways which are possible with data ingestion from Splunk.
- Can we send the log data directly from Splunk to Databricks?
- Do any intermediate tools/api required for the communication? If it's mandatory, then what are the possible tools/api?
- Splunk have event data and metric data. Is it possible to pick both these type of data by Databricks?
Could anyone please help me out with these queries?
01-03-2024 01:01 AM
Hi @Arch_dbxlearner,
As for your other questions:
I hope this helps you understand how to integrate Splunk with Databricks better. If you have any more questions, feel free to ask.
01-03-2024 02:53 AM
Thankyou @Kaniz for the clear explanation.
I have another set of questions. Please provide your suggestion on these as well.
Thankyou!
01-03-2024 08:48 AM
Hi @Arch_dbxlearner, The Databricks receiver allows the Splunk Distribution of OpenTelemetry Collector to collect metrics.... This integration can be used to view and monitor the health of your Databricks clusters.
As for sending application/system logs directly to Databricks, there are a few ways to do this:
To send complete raw data and/or customized filter data from an application to Databricks, you can c.... This includes ingesting raw data, transforming the data, and running analyses on the processed data. You can also use a connector like Confluent, which simplifies the architecture and implementation fo...
Let me know if you need more information! 😊
01-05-2024 02:04 AM
Thankyou @Kaniz .
Currently I am planning to check the possible ways to send the sample data to Databricks from Splunk without any third party tool's intervention.
Let me play around with those and get back to you if I need any guidance at any place.
Thanks once again!
01-09-2024 03:41 AM
You have mentioned that Databricks Add-on for Splunk, is bidirectional. Do we need to install this app on Databricks itself, to fetch the data from Splunk?
I tried to check this add-on on Databricks Marketplace but I could not find this. Can you please let me know the process to install the add-on?
I am looking to push the data from Splunk to Databricks and do some process and activity on daily basis. Could you please suggest me on this?
01-09-2024 09:36 PM
Hi @Kaniz
Can you please guide me on this?
01-18-2024 01:47 AM
Hi @Arch_dbxlearner, You can send data from Splunk to Databricks without using any third-party tools by leveraging the Da.... This add-on is a bidirectional connector that allows you to run queries and execute actions in Datab.... It can also push data to Splunk via its HTTP Event Collector (HEC).
Here are the steps you can follow:
Install the Databricks Add-on for Splunk: You can find the add-on on the Databricks Labs GitHub page. Follow the installation instructions provided.
Configure the Add-on: After installing the add-on, you’ll need to configure it to connect to your Databricks workspace. This typically involves providing your Databricks workspace URL and access token.
Send Data: Once the add-on is installed and configured, you can use it to send data from Splunk to Databricks. This can be done by running queries, notebooks, or jobs in Databricks from within Splunk.
If you encounter any issues or need further assistance, feel free to ask. I’m here to help! 😊
01-22-2024 12:33 AM - edited 01-22-2024 12:34 AM
Hi @Kaniz
I have gone through the github page of Databricks - Splunk integration. In the architecture diagram it is mentioned with 3 sections.
My requirement is only to fetch the data from Splunk and put in Databricks to do analysis and create dashboard. so I assumed, for my usecase 3rd option is the method to be done and I have followed the github page - here.
I have installed the databricks cli, created a secret scope to save Splunk credentials. Now I am working on the Notebook part to create Python code to fetch data.
Could you please guide me on this to proceed further?
01-23-2024 12:03 AM
@Kaniz Can you please guide me on this?
01-29-2024 03:32 AM
Hi @Kaniz
Still I am awaiting for your response on this. Can you please go through my above reply and guide me accordingly?
Thank you!