[Splunk] Initial release of the splunk by sharadcrest · Pull Request #13085 · elastic/integrations

sharadcrest · 2025-03-12T16:21:16Z

Proposed commit message

The initial release includes an alert data stream and associated dashboard
and visualizations.

Splunk fields are mapped to their corresponding ECS fields where possible.

Test samples were derived from live data samples, which were subsequently
sanitized.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/splunk directory.
Run the following command to run tests.

elastic-package test

--- Test results for package: splunk - START ---
╭─────────┬─────────────┬───────────┬─────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                       │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼─────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ splunk  │             │ asset     │ dashboard splunk-eef0d14a-7e42-414d-a0dc-a41423800158 is loaded │ PASS   │       1.94µs │
│ splunk  │ alert       │ asset     │ index_template logs-splunk.alert is loaded                      │ PASS   │        196ns │
│ splunk  │ alert       │ asset     │ ingest_pipeline logs-splunk.alert-0.1.0 is loaded               │ PASS   │         85ns │
╰─────────┴─────────────┴───────────┴─────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: splunk - END   ---
Done
Run pipeline tests for the package
--- Test results for package: splunk - START ---
╭─────────┬─────────────┬───────────┬───────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                 │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────────────────────────────────────┼────────┼──────────────┤
│ splunk  │ alert       │ pipeline  │ (ingest pipeline warnings test-alert.log) │ PASS   │ 312.589584ms │
│ splunk  │ alert       │ pipeline  │ test-alert.log                            │ PASS   │ 142.020276ms │
╰─────────┴─────────────┴───────────┴───────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: splunk - END   ---
Done
Run policy tests for the package
--- Test results for package: splunk - START ---
No test results
--- Test results for package: splunk - END   ---
Done
Run static tests for the package
--- Test results for package: splunk - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ splunk  │ alert       │ static    │ Verify sample_event.json │ PASS   │ 117.602376ms │
╰─────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: splunk - END   ---
Done
Run system tests for the package
2025/03/12 19:41:29  INFO License text found in "/home/devuser/bitbucket/integrations/LICENSE.txt" will be included in package
2025/03/12 19:41:29  INFO Import ECS mappings into the built package (technical preview)
2025/03/12 19:42:21  INFO Write container logs to file: /home/devuser/bitbucket/integrations/build/container-logs/splunk-1741788741444729491.log
2025/03/12 19:42:24  INFO Write container logs to file: /home/devuser/bitbucket/integrations/build/container-logs/elastic-agent-1741788744249682778.log
--- Test results for package: splunk - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ splunk  │ alert       │ system    │ default   │ PASS   │ 39.611582238s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: splunk - END   ---
Donee

Screenshots

kcreddy · 2025-03-17T07:32:04Z

/test

elasticmachine · 2025-03-17T07:32:48Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6

Error: there is no owner for "packages/splunk" in ".github/CODEOWNERS"

efd6 · 2025-03-18T06:18:27Z

/test

elastic-vault-github-plugin-prod · 2025-03-18T06:38:56Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/splunk/_dev/build/docs/README.md

packages/splunk/data_stream/alert/manifest.yml

packages/splunk/data_stream/alert/_dev/test/pipeline/test-alert.log-expected.json

packages/splunk/img/splunk-alert-dashboard.png

packages/splunk/_dev/build/docs/README.md

packages/splunk/data_stream/alert/_dev/test/pipeline/test-alert.log-expected.json

packages/splunk/data_stream/alert/agent/stream/cel.yml.hbs

packages/splunk/data_stream/alert/elasticsearch/ingest_pipeline/default.yml

jamiehynds · 2025-03-19T14:13:59Z

packages/splunk/data_stream/alert/_dev/test/pipeline/test-alert.log-expected.json

+                    "_time": "2025-02-10T06:20:16.000Z",
+                    "annotations": {
+                        "mitre_attack": {
+                            "mitre_tactic": "mite",


Could some of these mitre fields be mapped the threat.* fields in ECS? https://www.elastic.co/guide/en/ecs/current/ecs-threat.html

jamiehynds · 2025-03-19T14:15:26Z

packages/splunk/data_stream/alert/_dev/test/pipeline/test-alert.log-expected.json

+                    "app": "ssl-web",
+                    "count": 5.0,
+                    "dest_count": 1,
+                    "friendly_name": "Access - Excessive Failed Logins - Rule",


Could rule.name be a suitable field for this one? Alternatively, we populate the message fields with this friendly name and use that as the Alert Name in our promotion rule, meaning users will see 'Access - Excessive Failed Logins' as the alert name in our UI.

peluja1012 · 2025-03-20T02:36:51Z

packages/splunk/data_stream/alert/elasticsearch/ingest_pipeline/default.yml

+      type: string
+      ignore_missing: true
+  - convert:
+      field: splunk.alert.risk_score


Should we also set event.risk_score with this value since event.risk_score is an ECS field?

set event.risk_score mapping.👍

peluja1012 · 2025-03-20T03:30:10Z

packages/splunk/data_stream/alert/fields/base-fields.yml

+- name: event.dataset
+  type: constant_keyword
+  description: Event dataset.
+  value: splunk.alert


Should this be splunk.alerts instead to be consistent with the endpoint integration where the value is set to endpoint.alerts?

We used to follow the data stream name as singular as per the best practices. @kcreddy let me know your thoughts on this.

There is no clear guidelines on this in any Elastic docs I can find.
I checked Integration guide, ECS definition, an introduction blog, and fleet docs, none of which mentions which is preferred.

In our current integrations alone:
find . -maxdepth 3 -type d -name "*alert*", we have 9 plural and 16 singular names for alert(s).

./prisma_cloud/data_stream/alert ./microsoft_sentinel/data_stream/alert ./cloud_defend/data_stream/alerts ./crowdstrike/data_stream/alert ./zerofox/data_stream/alerts ./trend_micro_vision_one/data_stream/alert ./darktrace/data_stream/ai_analyst_alert ./darktrace/data_stream/model_breach_alert ./darktrace/data_stream/system_status_alert ./ti_rapid7_threat_command/data_stream/alert ./sysdig/data_stream/alerts ./sophos_central/data_stream/alert ./google_secops/data_stream/alert ./m365_defender/data_stream/alert ./mongodb_atlas/data_stream/alert ./jamf_protect/data_stream/alerts ./google_workspace/data_stream/alert ./zscaler_zia/data_stream/alerts ./falco/data_stream/alerts ./blacklens/data_stream/alerts ./sentinel_one/data_stream/alert ./panw_cortex_xdr/data_stream/alerts ./netskope/data_stream/alerts ./carbon_black_cloud/data_stream/alert_v7 ./carbon_black_cloud/data_stream/alert

I don't have a preference. We can go ahead with @peluja1012 suggestion.
@jamiehynds, any thoughts?

Hi @sharadcrest, apologies for confusion. Please go ahead and use the singular form as you have it currently. Thanks!

kcreddy · 2025-03-21T16:57:42Z

/test

packages/splunk/data_stream/alert/agent/stream/cel.yml.hbs

packages/splunk/img/splunk-logo.svg

packages/splunk/data_stream/alert/manifest.yml

packages/splunk/data_stream/alert/agent/stream/cel.yml.hbs

packages/splunk/data_stream/alert/_dev/test/pipeline/test-alert.log-expected.json

packages/splunk/data_stream/alert/elasticsearch/ingest_pipeline/default.yml

packages/splunk/data_stream/alert/_dev/test/pipeline/test-alert.log-expected.json

kcreddy

@sharadcrest could you address the review comments?

efd6

There are remaining unaddressed comments.

kcreddy · 2025-04-02T11:39:57Z

/test

efd6 · 2025-04-03T20:32:42Z

/test

efd6

Thanks

efd6 · 2025-04-03T21:25:22Z

@sharadcrest Before this is merged, can you note where the test cases were obtained?

sharadcrest · 2025-04-04T03:41:58Z

@sharadcrest Before this is merged, can you note where the test cases were obtained?

@efd6 Test cases were derived from live data samples, which were subsequently sanitized.

efd6 · 2025-04-04T04:27:36Z

Thanks @sharadcrest. For whoever merges, I'll propose this commit message body (the title line looks fine to me)

The initial release includes an alert data stream and associated dashboard
and visualizations.

Splunk fields are mapped to their corresponding ECS fields where possible.

Test samples were derived from live data samples, which were subsequently
sanitized.

kcreddy

@sharadcrest, I'm blocking the merge as some requirements are still being discussed.
cc: @jamiehynds @piyush-elastic

…eter.

kcreddy · 2025-04-14T02:25:44Z

/test

kcreddy · 2025-04-14T02:34:39Z

packages/splunk/data_stream/alert/agent/stream/cel.yml.hbs

+          "application/x-www-form-urlencoded",
+          {
+            "output_mode":["json"],
+            "search":["search index=notable AND " + state.?search.orValue('""')],


Does this search query work in Splunk?

search index=notable AND ""

Good catch. I bet it doesn't.

we removed the search parameter from the manifest to avoid confusion to the user as per the further discussions.

@kcreddy, @efd6 – As per my last discussion with @jamiehynds, we’ve removed SPL support and retained only the notable index as the default value. We've also updated the logo and submitted the corresponding changes in the PR. Kindly review the latest commit when you get a chance.

Thanks @piyush-elastic @sharadcrest
@jamiehynds let me know if this sounds good, I can merge the PR.

kcreddy · 2025-04-14T09:35:18Z

/test

elasticmachine · 2025-04-14T10:14:47Z

💚 Build Succeeded

Buildkite Build
Commit: 2fe7c37

History

💚 Build #24728 succeeded 83dd26a
💚 Build #24315 succeeded 1f679e9
💚 Build #24220 succeeded 2483ada
💚 Build #24125 succeeded 7838572
💚 Build #23709 succeeded 924ddfa
💚 Build #23494 succeeded 7d1c28c

elastic-sonarqube · 2025-04-14T10:14:50Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
85.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-04-14T13:28:18Z

Package splunk - 0.1.0 containing this change is available at https://epr.elastic.co/package/splunk/0.1.0/

sharadcrest added 2 commits March 12, 2025 20:08

Initial release

0192a97

Update changelog entry

a623c46

andrewkroh added New Integration Issue or pull request for creating a new integration package. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. labels Mar 13, 2025

kcreddy added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Mar 17, 2025

kcreddy added the Crest Contributions from Crest developement team. label Mar 17, 2025

efd6 reviewed Mar 17, 2025

View reviewed changes

Add owner for splunk package

7d1c28c

sharadcrest requested a review from efd6 March 18, 2025 05:53

kcreddy reviewed Mar 18, 2025

View reviewed changes

efd6 reviewed Mar 18, 2025

View reviewed changes

jamiehynds reviewed Mar 19, 2025

View reviewed changes

peluja1012 reviewed Mar 20, 2025

View reviewed changes

Resolve comments given by @kcreddy and @efd6

924ddfa

sharadcrest requested review from efd6 and kcreddy March 20, 2025 16:28

efd6 reviewed Mar 23, 2025

View reviewed changes

kcreddy reviewed Mar 25, 2025

View reviewed changes

sharadcrest requested review from efd6, jamiehynds, kcreddy and peluja1012 March 25, 2025 10:25

kcreddy reviewed Mar 26, 2025

View reviewed changes

efd6 requested changes Apr 1, 2025

View reviewed changes

resolve remaining comments of @efd6

2483ada

sharadcrest requested review from andrewkroh and efd6 April 2, 2025 11:05

remove in state mutatioion

1f679e9

efd6 approved these changes Apr 3, 2025

View reviewed changes

kcreddy approved these changes Apr 4, 2025

View reviewed changes

mrodm approved these changes Apr 7, 2025

View reviewed changes

kcreddy requested changes Apr 7, 2025

View reviewed changes

kgeller mentioned this pull request Apr 10, 2025

[AI4DSOC] Add configurations integrations page elastic/kibana#217905

Merged

As discussed with @jamiehynds, removed the search configuration param…

83dd26a

…eter.

sharadcrest requested a review from a team as a code owner April 11, 2025 05:03

sharadcrest requested a review from kcreddy April 11, 2025 05:04

kcreddy reviewed Apr 14, 2025

View reviewed changes

removed search index parameter

951f8bf

sharadcrest requested a review from kcreddy April 14, 2025 06:31

update logo

2fe7c37

kcreddy approved these changes Apr 14, 2025

View reviewed changes

kcreddy merged commit e4c1f04 into elastic:main Apr 14, 2025
7 checks passed

andrewkroh added the Integration:splunk Splunk label Apr 25, 2025

Conversation

sharadcrest commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Screenshots

Uh oh!

kcreddy commented Mar 17, 2025

Uh oh!

elasticmachine commented Mar 17, 2025

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 commented Mar 18, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Mar 18, 2025

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy commented Mar 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

sharadcrest commented Mar 12, 2025 •

edited

Loading

kcreddy Mar 21, 2025 •

edited

Loading

kcreddy left a comment •

edited

Loading

kcreddy Apr 14, 2025 •

edited

Loading