[azure] update routing for signinlogs and springcloudlogs by mmahacek · Pull Request #15807 · elastic/integrations

mmahacek · 2025-10-29T18:00:11Z

Proposed commit message

Fix for routing Azure signinlogs from Azure AD.

The "Windows Azure Active Directory" resource generates logs with the category MicrosoftServicePrincipalSignInLogs. When using the Azure Events v2 stream, these are handled as platform logs instead of as signinLogs. This PR adds routing to flag these events as the correct dataset.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
~~I have verified that all data streams collect metrics or logs.~~
I have added an entry to my package's changelog.yml file.
~~I have verified that Kibana version constraints are current according to guidelines.~~
~~I have verified that any added dashboard complies with Kibana's Dashboard good practices~~

Author's Checklist

[ ]

How to test this PR locally

Ingest data with a {"category":"MicrosoftServicePrincipalSignInLogs"} and verify the document ends up in the signinlogs dataset.

Related issues

Screenshots

packages/azure/docs/events.md

elastic-vault-github-plugin-prod · 2025-10-30T23:32:19Z

🚀 Benchmarks report

Package `azure` 👍(2) 💚(5) 💔(5)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`signinlogs`	2583.98	2004.01	-579.97 (-22.44%)	💔
`springcloudlogs`	8064.52	5464.48	-2600.04 (-32.24%)	💔
`eventhub`	500000	250000	-250000 (-50%)	💔
`identity_protection`	6666.67	4784.69	-1881.98 (-28.23%)	💔
`platformlogs`	5263.16	4273.5	-989.66 (-18.8%)	💔

To see the full report comment with /test benchmark fullreport

zmoog · 2025-10-31T08:23:32Z

Hey @mmahacek, thanks for contributing this update!

I wonder if we should consider sing-in logs all *SignInLogs log categories, @kaiyan-sheng WDYT?

mmahacek · 2025-10-31T14:14:27Z

The signinlogs pipeline does drop any non "*signinlogs" before processing, so this does make some sense.

https://github.com/elastic/integrations/blob/main/packages/azure/data_stream/signinlogs/elasticsearch/ingest_pipeline/default.yml#L46-L48

mmahacek · 2025-10-31T16:05:36Z

While I'm updating routing rules, I notice that the router for springcloudlogs has a few different values:

integrations/packages/azure/data_stream/events/elasticsearch/ingest_pipeline/default.yml

Lines 106 to 110 in 80b9687

    
           - set: 
        
               field: event.dataset 
        
               value: azure.springcloudlogs 
        
               if: 'ctx.tmp_json?.category == "ApplicationConsole" || ctx.tmp_json?.category == "SystemLogs" || ctx.tmp_json?.category == "IngressLogs" || ctx.tmp_json?.category == "BuildLogs" || ctx.tmp_json?.category == "ContainerEventLogs"' 
        
               description: 'Azure Spring Apps log categories (refs: https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-logs/microsoft-appplatform-spring-logs)'

But the pipeline for that dataset only keeps two of the values:

integrations/packages/azure/data_stream/springcloudlogs/elasticsearch/ingest_pipeline/default.yml

Lines 53 to 55 in 80b9687

    
           - drop: 
        
               if: "ctx.azure.springcloudlogs.category != 'SystemLogs' && ctx.azure.springcloudlogs.category != 'ApplicationConsole'" 
        
               ignore_failure: true

Should this be updated too?

This brings the event routing in line with the logic within the signinlogs data stream to only keep these documents.

kaiyan-sheng · 2025-10-31T18:12:55Z

I wonder if we should consider sing-in logs all *SignInLogs log categories, @kaiyan-sheng WDYT?

@zmoog Yeah I think that's probably safer for whatever new signin logs Azure is going to add!

… into mm/azure-aad-signin

packages/azure/data_stream/events/elasticsearch/ingest_pipeline/default.yml

mmahacek · 2025-11-06T17:24:11Z

😅 Another update came in for this integration. Pushing a version bump - should be no other functional changes.

mmahacek · 2025-11-06T23:03:51Z

@elastic/obs-ds-hosted-services This is ready for a code owners approval. Thanks in advance!

packages/azure/manifest.yml

rwestECI · 2025-11-25T14:15:41Z

Any update here guys? My org is the one who reported this and needs it resolved.

rwestECI · 2025-12-16T19:12:19Z

Still waiting for this to be merged.

btrieger · 2025-12-16T19:55:52Z

#16538 which I have added adds another source type for a SignInLog. Can modify my PR after this is merged to only add the new fields.

elasticmachine · 2025-12-17T05:49:31Z

💚 Build Succeeded

Buildkite Build
Commit: 9d464fa

History

💔 Build #35583 failed 1079457
💚 Build #34640 succeeded 40c8ebb
💚 Build #33778 succeeded 215f2c2
💚 Build #33610 succeeded 3777a5a
💚 Build #33511 succeeded 9983786

cc @mmahacek

elastic-vault-github-plugin-prod · 2025-12-17T21:20:04Z

Package azure - 1.32.0 containing this change is available at https://epr.elastic.co/package/azure/1.32.0/

[azure] add routing for MicrosoftServicePrincipalSignInLogs category

fe7e40b

mmahacek self-assigned this Oct 29, 2025

mmahacek requested review from a team as code owners October 29, 2025 18:00

mmahacek added Integration:azure Azure Logs bugfix Pull request that fixes a bug issue Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels Oct 29, 2025

mmahacek added 2 commits October 29, 2025 11:00

[azure] update changelog

65037d9

[azure] docs: Add MicrosoftServicePrincipal to signinlogs mapping

791b638

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Oct 30, 2025

efd6 reviewed Oct 30, 2025

View reviewed changes

packages/azure/docs/events.md Show resolved Hide resolved

mmahacek added 2 commits October 30, 2025 15:58

[azure] fix docs edit

ebaa3a0

[azure] add signinlog tests for all event router category values

bbe285e

andrewkroh added the Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] label Oct 31, 2025

mmahacek added 5 commits October 31, 2025 08:42

[azure] add routing for MicrosoftServicePrincipalSignInLogs category

0b504b2

[azure] update changelog

010af37

[azure] docs: Add MicrosoftServicePrincipal to signinlogs mapping

1080539

[azure] fix docs edit

95ebb46

[azure] add signinlog tests for all event router category values

aabdd01

[azure] update event router to match *SignInLogs category

e56bc79

This brings the event routing in line with the logic within the signinlogs data stream to only keep these documents.

[azure] update springcloudlogs categories to match v2 routing

2c7dd40

mmahacek changed the title ~~[azure] add routing for MicrosoftServicePrincipalSignInLogs category~~ [azure] update routing for signinlogs and springcloudlogs Oct 31, 2025

Merge branch 'mm/azure-aad-signin' of github.com:elastic/integrations…

42df83b

… into mm/azure-aad-signin

mmahacek requested a review from a team as a code owner October 31, 2025 18:21

mmahacek marked this pull request as draft October 31, 2025 18:22

[azure] update changelog

9983786

mmahacek marked this pull request as ready for review October 31, 2025 19:12

efd6 reviewed Nov 2, 2025

View reviewed changes

packages/azure/data_stream/events/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

[azure] remove unneeded commented line

3777a5a

mykola-elastic approved these changes Nov 6, 2025

View reviewed changes

mmahacek added 2 commits November 6, 2025 09:18

Merge branch 'main' into mm/azure-aad-signin

be43633

[azure] bump version for other update that came in first

215f2c2

mmahacek enabled auto-merge (squash) November 6, 2025 17:38

kaiyan-sheng reviewed Nov 10, 2025

View reviewed changes

packages/azure/manifest.yml Outdated Show resolved Hide resolved

Merge branch 'main' into mm/azure-aad-signin

40c8ebb

Merge branch 'main' into mm/azure-aad-signin

1079457

ishleenk17 approved these changes Dec 17, 2025

View reviewed changes

azure: update changelog notes

9d464fa

zmoog approved these changes Dec 17, 2025

View reviewed changes

mmahacek merged commit ba2714f into main Dec 17, 2025
8 checks passed

mmahacek deleted the mm/azure-aad-signin branch December 17, 2025 21:02

mmahacek mentioned this pull request Dec 17, 2025

[Azure] Add ADFS Sign-In Logs support to signinlogs data stream. #16538

Open

5 tasks

Conversation

mmahacek commented Oct 29, 2025

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package azure 👍(2) 💚(5) 💔(5)

Uh oh!

zmoog commented Oct 31, 2025

Uh oh!

mmahacek commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmahacek commented Oct 31, 2025

Uh oh!

kaiyan-sheng commented Oct 31, 2025

Uh oh!

Uh oh!

mmahacek commented Nov 6, 2025

Uh oh!

mmahacek commented Nov 6, 2025

Uh oh!

Uh oh!

rwestECI commented Nov 25, 2025

Uh oh!

rwestECI commented Dec 16, 2025

Uh oh!

btrieger commented Dec 16, 2025

Uh oh!

elasticmachine commented Dec 17, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Dec 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

elastic-vault-github-plugin-prod bot commented Oct 30, 2025 •

edited

Loading

Package `azure` 👍(2) 💚(5) 💔(5)

mmahacek commented Oct 31, 2025 •

edited

Loading