azure: fix Grok processor error for firewall network rule logs by JulienOrain · Pull Request #13920 · elastic/integrations

JulienOrain · 2025-05-15T15:06:11Z

Proposed commit message

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

elasticmachine · 2025-05-15T16:00:41Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6 · 2025-05-15T21:43:20Z

/test

efd6

LGTM after nit is addressed.

efd6 · 2025-05-15T21:41:53Z

packages/azure/data_stream/firewall_logs/_dev/test/pipeline/test-networkrules-raw.log

 {"category":"AzureFirewallNetworkRule","operationName":"AzureFirewallNetworkRuleLog","properties":{"msg":"HTTP request from 192.168.0.2:54314 to ocsp.sca1b.amazontrust.com:80. Url: ocsp.sca1b.amazontrust.com. Action: Deny. ThreatIntel: Bot Networks"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2022-06-08T20:40:56.4525380Z"}
 {"category":"AzureFirewallNetworkRule","operationName":"AzureFirewallNetworkRuleLog","properties":{"msg":"ICMP request from 192.168.0.2: to 175.16.199.1:. Action: alert. Signature: 2100366. IDS: ICMP_INFO PING *NIX. Priority: 3. Classification: Misc activity"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2022-06-08T20:40:56.4525380Z"}
 {"category":"AzureFirewallNetworkRule","operationName":"AzureFirewallNetworkRuleLog","properties":{"msg":"TCP request from 192.168.0.2:50306 to 89.160.20.156:3389. Action: Allow.. Rule Collection: Permit_RFC1918. Rule: Permit_RFC1918"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2025-03-13T07:11:59.992099+00:00"}
+{"category":"AzureFirewallNetworkRule","operationName":"AzureFirewallNatRuleLog","properties":{"msg":"TCP request from 192.168.0.2:50306 to 89.160.20.156:3389 was DNAT'ed to 10.0.0.2:3389. Rule Collection: DNAT. Rule: rule"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2022-06-08T20:40:56.4525380Z"}


Please add the missing final new line.

efd6 · 2025-05-15T22:05:21Z

Please run elastic-package test pipeline -d firewall-logs -g.

test case failed: Expected results are different from actual ones: --- want
+++ got
@@ -896,7 +896,7 @@
                     "network"
                 ],
                 "kind": "event",
-                "original": "{\"category\":\"AzureFirewallNetworkRule\",\"operationName\":\"AzureFirewallNatRuleLog\",\"properties\":{\"msg\":\"TCP request from 192.168.0.2:50306 to 89.160.20.156:3389 was DNAT'ed to 10.0.0.2:3389. Policy: policy-01. Rule Collection Group: DefaultDnatRuleCollectionGroup. Rule Collection: DNAT. Rule: rule\"},\"resourceId\":\"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01\",\"time\":\"2022-06-08T20:40:56.4525380Z\"}",
+                "original": "{\"category\":\"AzureFirewallNetworkRule\",\"operationName\":\"AzureFirewallNatRuleLog\",\"properties\":{\"msg\":\"TCP request from 192.168.0.2:50306 to 89.160.20.156:3389 was DNAT'ed to 10.0.0.2:3389. Rule Collection: DNAT. Rule: rule\"},\"resourceId\":\"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01\",\"time\":\"2022-06-08T20:40:56.4525380Z\"}",
                 "type": [
                     "connection"
                 ]

JulienOrain · 2025-05-16T08:06:08Z

Run pipeline tests for the package
--- Test results for package: azure - START ---
╭─────────┬───────────────┬───────────┬─────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM   │ TEST TYPE │ TEST NAME                                                           │ RESULT │ TIME ELAPSED │
├─────────┼───────────────┼───────────┼─────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-applicationrules-raw.log)            │ PASS   │ 338.244791ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-applicationrules-structured-raw.log) │ PASS   │ 301.347125ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-dnsproxy-structured-raw.log)         │ PASS   │ 294.812875ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-dnsproxyrules-raw.log)               │ PASS   │ 301.480917ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-natrule-structured-raw.log)          │ PASS   │ 302.623875ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-networkrule-structured-raw.log)      │ PASS   │    288.899ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-networkrules-raw.log)                │ PASS   │ 321.231334ms │
│ azure   │ firewall_logs │ pipeline  │ (ingest pipeline warnings test-sdh3075-raw.log)                     │ PASS   │  313.15875ms │
│ azure   │ firewall_logs │ pipeline  │ test-applicationrules-raw.log                                       │ PASS   │ 135.278834ms │
│ azure   │ firewall_logs │ pipeline  │ test-applicationrules-structured-raw.log                            │ PASS   │  69.822334ms │
│ azure   │ firewall_logs │ pipeline  │ test-dnsproxy-structured-raw.log                                    │ PASS   │   69.52575ms │
│ azure   │ firewall_logs │ pipeline  │ test-dnsproxyrules-raw.log                                          │ PASS   │     87.385ms │
│ azure   │ firewall_logs │ pipeline  │ test-natrule-structured-raw.log                                     │ PASS   │  68.191667ms │
│ azure   │ firewall_logs │ pipeline  │ test-networkrule-structured-raw.log                                 │ PASS   │  68.047167ms │
│ azure   │ firewall_logs │ pipeline  │ test-networkrules-raw.log                                           │ PASS   │ 161.161041ms │
│ azure   │ firewall_logs │ pipeline  │ test-sdh3075-raw.log                                                │ PASS   │  64.328209ms │
╰─────────┴───────────────┴───────────┴─────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: azure - END   ---
Done

efd6 · 2025-05-16T08:31:51Z

/test

elastic-vault-github-plugin-prod · 2025-05-16T08:52:33Z

🚀 Benchmarks report

Package `azure` 👍(5) 💚(5) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`eventhub`	1e+06	500000	-500000 (-50%)	💔

To see the full report comment with /test benchmark fullreport

elastic-sonarqube · 2025-05-16T08:52:52Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
50.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

packages/azure/changelog.yml

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

JulienOrain · 2025-06-05T07:42:43Z

hello,
am i missing something for this MR ?

efd6 · 2025-06-22T22:15:58Z

/test

elasticmachine · 2025-06-22T22:40:58Z

💚 Build Succeeded

Buildkite Build
Commit: 8139392

History

💚 Build #26019 succeeded d0997b0
💔 Build #26004 failed 50a01a3

elastic-sonarqube · 2025-06-22T22:41:07Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
50.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

efd6

Thanks

elastic-vault-github-plugin-prod · 2025-06-22T23:03:01Z

Package azure - 1.27.1 containing this change is available at https://epr.elastic.co/package/azure/1.27.1/

…ic#13920)

azure: fix Grok processor error for firewall network rule logs

af162d7

JulienOrain requested review from a team as code owners May 15, 2025 15:06

add pr id

50a01a3

andrewkroh added Integration:azure Azure Logs Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels May 15, 2025

efd6 reviewed May 15, 2025

View reviewed changes

JulienOrain added 3 commits May 16, 2025 07:13

add missing final line

6297214

Merge branch 'elastic:main' into main

ddadafa

fix tests

d0997b0

efd6 reviewed May 18, 2025

View reviewed changes

packages/azure/changelog.yml Outdated Show resolved Hide resolved

JulienOrain and others added 5 commits June 2, 2025 16:20

Update packages/azure/changelog.yml

ebe424d

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Merge branch 'main' into main

d8c07ce

Merge branch 'main' into main

ce0c08b

Merge branch 'main' into main

f37b7e5

Merge branch 'main' into main

56220de

JulienOrain and others added 2 commits June 16, 2025 08:56

Merge branch 'main' into main

10b1a29

Merge branch 'main' into main

8139392

efd6 approved these changes Jun 22, 2025

View reviewed changes

efd6 merged commit 2be7f18 into elastic:main Jun 22, 2025
7 checks passed

shmsr pushed a commit to shmsr/integrations that referenced this pull request Jun 30, 2025

azure: fix Grok processor error for firewall network rule logs (elast…

4c68243

…ic#13920)

Conversation

JulienOrain commented May 15, 2025

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented May 15, 2025

Uh oh!

efd6 commented May 15, 2025

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 May 15, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 commented May 15, 2025

Uh oh!

JulienOrain commented May 16, 2025

Uh oh!

efd6 commented May 16, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package azure 👍(5) 💚(5) 💔(1)

Uh oh!

elastic-sonarqube bot commented May 16, 2025

Quality Gate passed

Uh oh!

Uh oh!

JulienOrain commented Jun 5, 2025

Uh oh!

efd6 commented Jun 22, 2025

Uh oh!

elasticmachine commented Jun 22, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Jun 22, 2025

Quality Gate passed

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Jun 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

elastic-vault-github-plugin-prod bot commented May 16, 2025 •

edited

Loading

Package `azure` 👍(5) 💚(5) 💔(1)