aws/data_stream/route53_public_logs: Handle non-%HOSTNAME entries by cafuego · Pull Request #9249 · elastic/integrations

cafuego · 2024-02-28T05:43:20Z

Proposed commit message

Change the HOSTNAME pattern to DATA in the grok snippet to match route53 public logs, so that DNS records that contain characters not included in [0-9A-Za-z][0-9A-Za-z-] can still be processed.

This notably includes entries such as _dmarc, _spf and various types of domain ownership verification records.

Those are a non-exhaustive list that are being dropped from my logs, but note that the DNS spec doesn't prevent other characters being present. The spec does not inform what a client might query for, which is what we're dealing with, not a zone file.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Use the provided patterns in packages/aws/data_stream/route53_public_logs/_dev/test/pipeline/test-route53.log

… log, so non-hostname logs can be parsed.

…sion.

botelastic · 2024-03-29T22:21:26Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

cafuego · 2024-03-29T23:48:23Z

Yup, still relevant and does not contain a dependency on xz 🤪

botelastic · 2024-04-29T00:21:35Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

ishleenk17 · 2024-04-29T04:51:49Z

/test

ishleenk17 · 2024-04-29T04:54:01Z

@cafuego : cAn we please resolve the merge conflicts

cafuego · 2024-04-29T22:09:08Z

/test

cafuego · 2024-04-29T22:10:18Z

@shmsr Yes, but buildkite seems to be unhappy about something; it's not completing the test or pinging back with the result (which has been fine previously)

botelastic · 2024-05-29T22:54:05Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2024-06-28T23:47:11Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

elasticmachine · 2024-07-19T20:42:36Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

packages/aws/changelog.yml

efd6 · 2024-07-21T21:34:32Z

packages/aws/data_stream/route53_public_logs/elasticsearch/ingest_pipeline/default.yml

+        - '%{BASE10NUM} %{TIMESTAMP_ISO8601:_tmp.timestamp} %{DATA:aws.route53.hosted_zone_id} %{DATA:_tmp.question} %{WORD:dns.question.type} %{WORD:dns.response_code} %{WORD:network.transport} %{EDGE_LOCATION:aws.route53.edge_location} %{IP:source.address} (%{SUBNET:aws.route53.edns_client_subnet}|-)'
      pattern_definitions:


The alternative would be to provided an augmented hostname pattern that includes all the codepoints that occur in DNS names. This only requires that we add _ to the possible codepoints.

Suggested change

- '%{BASE10NUM} %{TIMESTAMP_ISO8601:_tmp.timestamp} %{DATA:aws.route53.hosted_zone_id} %{DATA:_tmp.question} %{WORD:dns.question.type} %{WORD:dns.response_code} %{WORD:network.transport} %{EDGE_LOCATION:aws.route53.edge_location} %{IP:source.address} (%{SUBNET:aws.route53.edns_client_subnet}|-)'

pattern_definitions:

- '%{BASE10NUM} %{TIMESTAMP_ISO8601:_tmp.timestamp} %{DATA:aws.route53.hosted_zone_id} %{DNS_QUESTION:_tmp.question} %{WORD:dns.question.type} %{WORD:dns.response_code} %{WORD:network.transport} %{EDGE_LOCATION:aws.route53.edge_location} %{IP:source.address} (%{SUBNET:aws.route53.edns_client_subnet}|-)'

pattern_definitions:

DNS_QUESTION: \b(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62})(?:\.(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62}))*(\.?|\b)

At the very least you also need to add an escaped backslash, or you'll still miss any encoded UTF-8 data.

DNS entries can also contain any binary rubbish you'd care to put in (although Route53 may not allow you to add that) so from a log analysis perspective it's important to grab those logs and allow them to be analysed.

DNS entries can also contain any binary rubbish you'd care to put in

That's horrifying, but point well made.

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

efd6 · 2024-07-22T00:53:24Z

/test

packages/aws/data_stream/route53_public_logs/_dev/test/pipeline/test-route53.log-expected.json

cafuego · 2024-07-22T03:37:19Z

Aww, sad trombone. I'll update the test.

…e/test-route53.log-expected.json Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

efd6 · 2024-07-22T04:28:38Z

/test

elasticmachine · 2024-07-22T06:15:59Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2024-07-22T06:16:53Z

💚 Build Succeeded

Buildkite Build
Commit: f93d5e4

History

💔 Build #13811 failed fe9303c
💔 Build #10881 failed e34c87a

elastic-sonarqube · 2024-07-22T06:16:56Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
28.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

efd6

Thanks

elasticmachine · 2024-07-22T06:34:07Z

Package aws - 2.21.0 containing this change is available at https://epr.elastic.co/search?package=aws

cafuego · 2024-07-22T06:57:18Z

@efd6 thanks mate!

cafuego requested review from a team as code owners February 28, 2024 05:43

cafuego added 3 commits February 28, 2024 16:45

Expand the set of allowed characters in the question.name part of the…

b68ade9

… log, so non-hostname logs can be parsed.

Add a test case for _spf (previously failed to match) to avoid regres…

cac5cc4

…sion.

Bump the changelog and manifest files.

d49c066

cafuego force-pushed the route53-log-hostnames branch from 5a39b42 to d49c066 Compare February 28, 2024 05:46

cafuego added 2 commits February 28, 2024 21:31

Merge branch 'main' into route53-log-hostnames

e48a7ec

Bump the manifest file after resolving the changelog merge conflict.

95ac269

botelastic bot added the Stalled label Mar 29, 2024

Merge branch 'main' into route53-log-hostnames

e34c87a

botelastic bot removed the Stalled label Mar 29, 2024

botelastic bot added the Stalled label Apr 29, 2024

botelastic bot removed the Stalled label Apr 29, 2024

shmsr changed the title ~~Enhancement: - handle non-%HOSTNAME entries in route53 logs~~ aws/data_stream/route53_public_logs: Handle non-%HOSTNAME entries Apr 29, 2024

shmsr added the enhancement New feature or request label Apr 29, 2024

shmsr assigned cafuego Apr 29, 2024

Merge branch 'main' into route53-log-hostnames

98ab6a1

botelastic bot added the Stalled label May 29, 2024

Merge branch 'main' into route53-log-hostnames

1a134c1

botelastic bot removed the Stalled label May 29, 2024

cafuego removed their assignment May 29, 2024

botelastic bot added the Stalled label Jun 28, 2024

andrewkroh added the Integration:aws AWS label Jul 18, 2024

botelastic bot removed the Stalled label Jul 18, 2024

andrewkroh added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Jul 19, 2024

efd6 reviewed Jul 21, 2024

View reviewed changes

cafuego and others added 2 commits July 22, 2024 10:20

Update packages/aws/changelog.yml

f7c9392

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Merge branch 'main' into route53-log-hostnames

fe9303c

efd6 reviewed Jul 22, 2024

View reviewed changes

packages/aws/data_stream/route53_public_logs/_dev/test/pipeline/test-route53.log-expected.json Outdated Show resolved Hide resolved

Update packages/aws/data_stream/route53_public_logs/_dev/test/pipelin…

f93d5e4

…e/test-route53.log-expected.json Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

efd6 approved these changes Jul 22, 2024

View reviewed changes

efd6 merged commit 782c4ea into elastic:main Jul 22, 2024

		- '%{BASE10NUM} %{TIMESTAMP_ISO8601:_tmp.timestamp} %{DATA:aws.route53.hosted_zone_id} %{DATA:_tmp.question} %{WORD:dns.question.type} %{WORD:dns.response_code} %{WORD:network.transport} %{EDGE_LOCATION:aws.route53.edge_location} %{IP:source.address} (%{SUBNET:aws.route53.edns_client_subnet}\|-)'
		pattern_definitions:

Conversation

cafuego commented Feb 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Uh oh!

botelastic bot commented Mar 29, 2024

Uh oh!

cafuego commented Mar 29, 2024

Uh oh!

botelastic bot commented Apr 29, 2024

Uh oh!

ishleenk17 commented Apr 29, 2024

Uh oh!

ishleenk17 commented Apr 29, 2024

Uh oh!

cafuego commented Apr 29, 2024

Uh oh!

cafuego commented Apr 29, 2024

Uh oh!

botelastic bot commented May 29, 2024

Uh oh!

botelastic bot commented Jun 28, 2024

Uh oh!

elasticmachine commented Jul 19, 2024

Uh oh!

Uh oh!

efd6 Jul 21, 2024

Choose a reason for hiding this comment

Uh oh!

cafuego Jul 22, 2024

Choose a reason for hiding this comment

Uh oh!

efd6 Jul 22, 2024

Choose a reason for hiding this comment

Uh oh!

efd6 commented Jul 22, 2024

Uh oh!

Uh oh!

cafuego commented Jul 22, 2024

Uh oh!

efd6 commented Jul 22, 2024

Uh oh!

elasticmachine commented Jul 22, 2024

🚀 Benchmarks report

Uh oh!

elasticmachine commented Jul 22, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Jul 22, 2024

Quality Gate passed

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jul 22, 2024

Uh oh!

cafuego commented Jul 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cafuego commented Feb 28, 2024 •

edited

Loading