[Cisco Secure Email Gateway] Fix wrong grok for mail_logs and add new field. by leandrojmp · Pull Request #8573 · elastic/integrations

leandrojmp · 2023-11-24T19:29:20Z

Enhancement

Proposed commit message

This PR fix an error on a grok pattern for the mail_logs ingest pipelines and creates a new field named cisco_secure_email_gateway.log.email_participants as keyword and with a multi-field as text.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Closes [Cisco Secure Email Gateway] Mail Logs Ingest Pipeline broken Grok #8569

fix wrong grok on mail_logs ingest pipelines and add new field

elasticmachine · 2023-11-24T19:33:12Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-11-30T06:19:21.937+0000
Duration: 17 min 15 sec

Test stats 🧪

Test	Results
Failed	0
Passed	25
Skipped	0
Total	25

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

efd6 · 2023-11-26T21:31:33Z

packages/cisco_secure_email_gateway/changelog.yml

+      link: https://github.com/elastic/integrations/pull/XXXX
+    - description: Add new field cisco_secure_email_gateway.log.email_participants.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/XXXX


Suggested change

link: https://github.com/elastic/integrations/pull/XXXX

- description: Add new field cisco_secure_email_gateway.log.email_participants.

type: enhancement

link: https://github.com/elastic/integrations/pull/XXXX

link: https://github.com/elastic/integrations/pull/8573

- description: Add new field cisco_secure_email_gateway.log.email_participants.

type: enhancement

link: https://github.com/elastic/integrations/pull/8573

yeah, sorry, I added the PR number, but forget to send the push to my branch.

...ages/cisco_secure_email_gateway/data_stream/log/_dev/test/pipeline/test-common-text-mail.log

efd6 · 2023-11-26T21:35:43Z

...cure_email_gateway/data_stream/log/elasticsearch/ingest_pipeline/pipeline_text_mail_logs.yml

        - '^%{DATA:cisco_secure_email_gateway.log.message_status} %{WORD:network.protocol} DCID %{NUMBER:cisco_secure_email_gateway.log.delivery_connection_id} interface %{IP:cisco_secure_email_gateway.log.interface} address %{IP:cisco_secure_email_gateway.log.address}$'
-        - '^%{GREEDYDATA:cisco_secure_email_gateway.log.message_status} DCID %{NUMBER:cisco_secure_email_gateway.log.delivery_connection_id} MID %{NUMBER:email.message_id} to RID \[%{DATA:cisco_secure_email_gateway.log.recipient_id}\]$'
-        - '^%{GREEDYDATA:cisco_secure_email_gateway.log.message_status} DCID %{NUMBER:cisco_secure_email_gateway.log.delivery_connection_id} MID %{NUMBER:email.message_id} to RID \[%{DATA:cisco_secure_email_gateway.log.recipient_id}\]$'
+        - '^%{GREEDYDATA:cisco_secure_email_gateway.log.message_status} DCID %{NUMBER:cisco_secure_email_gateway.log.delivery_connection_id} MID %{NUMBER:email.message_id} to RID \[%{DATA:cisco_secure_email_gateway.log.recipient_id}\]\s\[%{DATA:cisco_secure_email_gateway.log.email_participants}\]$'


Consider populating email.{from,reply_to,to}.address form this?

Yeah, I'm still working on that and planned to do that on another PR.

The content that would go to the field email_participants can have multiple patterns, the order for from, reply-to and to can change, some of those may not be always present, you can have the emails between < and > sometimes, you also can have just the email or the name and the email, you can have multiple emais with a \r\n\t separator and a couple of more different patterns, so it is not a simple grok in this case.

Yeah, fair enough. That sounds like it should be in a separate PR.

efd6 · 2023-11-27T00:57:19Z

/test

leandrojmp · 2023-11-27T02:54:08Z

@efd6

It seems that the test failed because some issue related to the README.md file.

[2023-11-27T01:08:08.125Z] README.md is outdated. Rebuild the package with 'elastic-package build'
[2023-11-27T01:08:08.125Z] --- want
[2023-11-27T01:08:08.125Z] +++ got
[2023-11-27T01:08:08.125Z] @@ -400,2 +400,4 @@
[2023-11-27T01:08:08.125Z]  | cisco_secure_email_gateway.log.email |  | keyword |
[2023-11-27T01:08:08.125Z] +| cisco_secure_email_gateway.log.email_participants |  | keyword |
[2023-11-27T01:08:08.125Z] +| cisco_secure_email_gateway.log.email_participants.text | Multi-field of `cisco_secure_email_gateway.log.email_participants`. | text |
[2023-11-27T01:08:08.125Z]  | cisco_secure_email_gateway.log.email_tracker_header | Header consisting of (but not typically displaying) critical information for efficient email tracking and delivery. | keyword |
[2023-11-27T01:08:08.125Z] Error: checking package failed: checking readme files are up-to-date failed: files do not match

Since I'm adding a new field it seems that I also need to add it in the table in the README.md file.

Is that correct?

efd6 · 2023-11-27T03:45:54Z

If you run elastic-package build it will resolve this for you.

leandrojmp · 2023-11-27T12:15:20Z

If you run elastic-package build it will resolve this for you.

Thanks! Just added a description on fields.yml and ran elastic-package build to update the README.md

leandrojmp · 2023-11-28T12:45:11Z

Hello @efd6,

Can we try to run the tests again?

efd6 · 2023-11-28T19:52:46Z

/test

elasticmachine · 2023-11-28T20:10:04Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	84.615% (`11/13`)	👎 -15.385
Classes	84.615% (`11/13`)	👎 -15.385
Methods	89.831% (`53/59`)	👍 64.831
Lines	89.215% (`761/853`)	👎 -10.785
Conditionals	100.0% (`0/0`)	💚

efd6 · 2023-11-28T20:15:20Z

packages/cisco_secure_email_gateway/data_stream/log/fields/fields.yml

+      multi_fields:
+        - name: text
+          type: text


This is the first multi_fields field in the set. Do we need this if the field will be further dissected into ECS fields in a future PR?

I think this is useful because it enables you to search for a string and see if it was present as a participant on the e-mail, for example we had some use cases where we need to search if a particular email address appears as the sender or receiver, so without this we would need to search twice, one on the from field and another on the to field.

Also, all the email.*.address should have the e-mail address only, so if in the from field we have something like Sender Name <anythingelse@example.com>, we need to have just sender@example.com in the email.from.address, and this way we wouldn't be able to search for Sender Name as an email participant.

But, I can remove it and use an alternate custom field in my case.

Let's remove it for now, but if you file a feature request we can look into adding it in the future.

Done, removed the multi-field, I will just keep the mapping on my side because it is useful for our use case.

efd6 · 2023-11-30T06:19:08Z

/test

efd6

Thanks

elasticmachine · 2023-11-30T20:58:59Z

Package cisco_secure_email_gateway - 1.20.0 containing this change is available at https://epr.elastic.co/search?package=cisco_secure_email_gateway

elasticmachine · 2023-11-30T20:59:00Z

Package cisco_secure_email_gateway - 1.20.0 containing this change is available at https://epr.elastic.co/search?package=cisco_secure_email_gateway

fix wrong grok for mail_logs

1a98e8b

fix wrong grok on mail_logs ingest pipelines and add new field

leandrojmp requested a review from a team as a code owner November 24, 2023 19:29

efd6 reviewed Nov 26, 2023

View reviewed changes

add PR number, change grok to work for both old and new use case

1e1d7ad

edit fields.yml and run elastic-package build to update README.md

d25133b

efd6 reviewed Nov 28, 2023

View reviewed changes

remove multi-field for cisco_secure_email_gateway.log.email_participants

07d1f18

efd6 approved these changes Nov 30, 2023

View reviewed changes

efd6 merged commit 86964ce into elastic:main Nov 30, 2023

leandrojmp deleted the fix_recipient_id_add_email_participants branch December 1, 2023 19:50

andrewkroh added the Integration:cisco_secure_email_gateway Cisco Secure Email Gateway label Jul 22, 2024

Conversation

leandrojmp commented Nov 24, 2023

Proposed commit message

Checklist

Related issues

Uh oh!

elasticmachine commented Nov 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leandrojmp Nov 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 commented Nov 27, 2023

Uh oh!

leandrojmp commented Nov 27, 2023

Uh oh!

efd6 commented Nov 27, 2023

Uh oh!

leandrojmp commented Nov 27, 2023

Uh oh!

leandrojmp commented Nov 28, 2023

Uh oh!

efd6 commented Nov 28, 2023

Uh oh!

elasticmachine commented Nov 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 commented Nov 30, 2023

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Nov 30, 2023

Uh oh!

elasticmachine commented Nov 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

elasticmachine commented Nov 24, 2023 •

edited

Loading

leandrojmp Nov 27, 2023 •

edited

Loading

elasticmachine commented Nov 28, 2023 •

edited

Loading