[azure logs] Normalize capitalized Category field for proper data stream routing#14415
Merged
zmoog merged 6 commits intoelastic:mainfrom Jul 9, 2025
Merged
Conversation
CategoryCategory
constanca-m
reviewed
Jul 3, 2025
Contributor
constanca-m
left a comment
There was a problem hiding this comment.
Thank you so much @zmoog ! I thought we had to add Category support for everything, didn't even notice it was possible to just lowercase it.
Contributor
Author
|
can't recall the details, but I skipped tests because I thought the reroute processor "can't be tested" properly, but it turns out it's definitely possible. We can add a few test documents with the I added a few tests for the invalid field |
Category385a639 to
c85d42b
Compare
🚀 Benchmarks reportTo see the full report comment with |
mykola-elastic
approved these changes
Jul 4, 2025
9beea8d to
58393d5
Compare
Contributor
Author
|
Added one routing test for each supported log category: $ elastic-package test pipeline -d events
Run pipeline tests for the package
--- Test results for package: azure - START ---
╭─────────┬─────────────┬───────────┬────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ azure │ events │ pipeline │ (ingest pipeline warnings test-activitylogs-raw.log) │ PASS │ 558.513166ms │
│ azure │ events │ pipeline │ (ingest pipeline warnings test-application_gateway-raw.log) │ PASS │ 482.429333ms │
│ azure │ events │ pipeline │ (ingest pipeline warnings test-auditlogs-raw.log) │ PASS │ 475.755834ms │
│ azure │ events │ pipeline │ (ingest pipeline warnings test-firewall_logs-raw.log) │ PASS │ 503.368542ms │
│ azure │ events │ pipeline │ (ingest pipeline warnings test-graphactivitylogs-raw.log) │ PASS │ 488.667917ms │
│ azure │ events │ pipeline │ (ingest pipeline warnings test-identity_protection-raw.log) │ PASS │ 512.766917ms │
│ azure │ events │ pipeline │ (ingest pipeline warnings test-invalid-category-field-raw.log) │ PASS │ 530.038041ms │
│ azure │ events │ pipeline │ (ingest pipeline warnings test-provisioning-raw.log) │ PASS │ 493.391417ms │
│ azure │ events │ pipeline │ (ingest pipeline warnings test-signinlogs-raw.log) │ PASS │ 511.469667ms │
│ azure │ events │ pipeline │ (ingest pipeline warnings test-springcloudlogs-raw.log) │ PASS │ 558.499916ms │
│ azure │ events │ pipeline │ test-activitylogs-raw.log │ PASS │ 43.934875ms │
│ azure │ events │ pipeline │ test-application_gateway-raw.log │ PASS │ 39.646875ms │
│ azure │ events │ pipeline │ test-auditlogs-raw.log │ PASS │ 38.612042ms │
│ azure │ events │ pipeline │ test-firewall_logs-raw.log │ PASS │ 41.445125ms │
│ azure │ events │ pipeline │ test-graphactivitylogs-raw.log │ PASS │ 38.820708ms │
│ azure │ events │ pipeline │ test-identity_protection-raw.log │ PASS │ 38.927917ms │
│ azure │ events │ pipeline │ test-invalid-category-field-raw.log │ PASS │ 38.571375ms │
│ azure │ events │ pipeline │ test-provisioning-raw.log │ PASS │ 39.498334ms │
│ azure │ events │ pipeline │ test-signinlogs-raw.log │ PASS │ 39.363542ms │
│ azure │ events │ pipeline │ test-springcloudlogs-raw.log │ PASS │ 40.66325ms │
╰─────────┴─────────────┴───────────┴────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: azure - END ---
Done |
Contributor
Author
|
/test |
💚 Build Succeeded
History
|
|
|
Package azure - 1.28.1 containing this change is available at https://epr.elastic.co/package/azure/1.28.1/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Rename invalid log category field
Categoryascategoryto correctly infer theevent.dataset, and perform the routing to the target data stream, if available.Unfortunately, some Azure services produce logs with capitalized field names, violating the spec.
❌ Invalid format
{ "Category": "ServicePrincipalSignInLogs", "CorrelationId": "ed598e34-c91b-41da-a89e-e5a97da4209d", "DurationMs": 0, ... }Instead, it should be something similar:
✅ Expected format
{ "category": "ServicePrincipalSignInLogs", "correlationId": "ed598e34-c91b-41da-a89e-e5a97da4209d", "durationMs": 0, ... }To properly identify the event dataset and apply the correct routing rules, we need to convert them to lowercase.
Note: the target data stream is responsible for dealing with the capitalized field names. The
eventsdata stream's responsibility is to route to the target data stream, if available, or fall back to platform logs.Checklist
changelog.ymlfile.I have verified that any added dashboard complies with Kibana's Dashboard good practicesHow to test this PR locally
Related issues