cisco_secure_email_gateway: update ingest pipeline to handle v15 by efd6 · Pull Request #7809 · elastic/integrations

efd6 · 2023-09-14T00:50:00Z

What does this PR do?

The release notes for v15 note that some fields have changed name:

endTime => end
startTime => start
sourceAddress => src
sourceHostName => shost

To maintain backwards compatibility, provide alternations for both cases as pattern definitions in the relevant grok.

After this change, the only place that the old names are found in the elasticsearch directory is in the pattern definitions.

$ rg '(endTime|startTime|sourceAddress|sourceHostName)' data_stream/log/elasticsearch
data_stream/log/elasticsearch/ingest_pipeline/pipeline_consolidated_event.yml
58:        STARTTIME: '(startTime|start)'
59:        ENDTIME: '(endTime|end)'
60:        SOURCEADDRESS: '(sourceAddress|src)'
61:        SOURCEHOSTNAME: '(sourceHostName|shost)'

~~Ideally, we would have test cases for this, but they are not available at this stage.~~ This is now added.

The change also fixes the handling of logs that do not include a syslog priority, and the conversion of event.{start,end} to correctly formatted dates, and their type mappings.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

elasticmachine · 2023-09-14T00:53:16Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-09-22T05:30:48.987+0000
Duration: 16 min 21 sec

Test stats 🧪

Test	Results
Failed	0
Passed	23
Skipped	0
Total	23

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-09-14T01:07:30Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	83.333% (`10/12`)	👎 -12.765
Classes	83.333% (`10/12`)	👎 -12.765
Methods	89.286% (`50/56`)	👎 -2.333
Lines	88.098% (`681/773`)	👍 0.137
Conditionals	100.0% (`0/0`)	💚

elasticmachine · 2023-09-14T01:18:15Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

bhapas

There seem to be some examples here
https://www.cisco.com/c/en/us/td/docs/security/ces/ces_15-0/user_guide/b_ESA_Admin_Guide_ces_15-0/b_ESA_Admin_Guide_12_1_chapter_0100111.html

Can we not use them in pipeline tests additional to the existing ones?

efd6 · 2023-09-19T04:40:24Z

@bhapas I've taken a look at those examples and none exercises the changes here. Adding them as tests would probably be a sensible thing to do in the long term, but it would be sensible to have a directed plan of attack. @LaZyDK Do you have suggestions about a priority ordering of messages in that example set?

bhapas · 2023-09-19T04:44:18Z

@efd6 Yes. Just that we release a package without any tests on it. That was my only concern.

efd6 · 2023-09-19T05:08:52Z

It's not that it has no tests, it just has no tests for this change. Taking the examples in that document, according to the release notes, doesn't appear to change that. There are a lot of examples in the document, and they are not structured in a way that makes it obvious which to include.

bhapas · 2023-09-19T06:07:51Z

@efd6 If it the Consolidated Event Logs Format that is changed

Then there is an example , isn't it?

Example of Consolidated Event Logs

In this example, the log shows all the available fields selected when you configure a log subscription with the log type as Consolidated Event Logs.

Thu Jun 30 08:04:50 2022: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.0.0-657
|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=42127C7DDEE76852677B-F80CE8074CD3
ESACustomLogs={'label2': ['value20'], 'label1': ['value1', 'value2']} 
ESALogHeaders={'reply-to': ['test@esa.com ', 'newsletter@esa.com'], 'from': ['any@esa.com']}
ESAMID=1053 ESAICID=134 ESAAMPVerdict=UNKNOWN ESAASVerdict=NEGATIVE ESAAVVerdict=NEGATIVE 
ESACFVerdict=MATCH end=Thu Mar 18 08:04:46 2021 ESADLPVerdict=NOT_EVALUATED 
dvc=10.10.193.13 ESAAttachmentDetails={'test.txt': {'AMP': {'Verdict': 'FILE UNKNOWN', 
'fileHash': '7f843d263304fb0516d6210e9de4fa7f01f2f623074aab6e3ee7051f7b785cfa'}, 'BodyScanner':
{'fsize': 10059}}} ESAFriendlyFrom=test@esa.com ESAGMVerdict=NEGATIVE start=Thu Mar 18 08:04:29 2021 
deviceInboundInterface=Incomingmail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=test@esa.com 
cs1Label=MailPolicy cs1=DEFAULT ESAMFVerdict=NOT_EVALUATED act=QUARANTINED ESAFinalActionDetails=To POLICY 
cs4Label=ExternalMsgID cs4='<20210318070601.40490.18684@mail1.example.com>' ESAMsgSize=11873 ESAOFVerdict=POSITIVE 
duser=9076@testing.com ESAHeloIP=10.11.1.2 cfp1Label=SBRSScore cfp1=None ESASDRDomainAge=27 years 2 months 15 days
cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Weak ESASPFVerdict={'mailfrom': {'result': 'None', 
'sender': 'test@esa.com'}, 'helo': {'result': 'None', 'sender': 'postmaster'}, 'pra': {'result': 'None', 'sender':
'test@esa.com'}}shost=unknown ESASenderGroup=UNKNOWNLIST src=10.11.1.2 msg='Testing'

LaZyDK · 2023-09-19T08:05:22Z

@efd6 I would add the above Consolidated Event log to the tests. I think that it coveres what you need.

efd6 · 2023-09-19T08:18:34Z

Yeah, we had a discussion about this and agreed that this is probably what needs to be done. The log line there fails to parse out the details, but I am not convinced of copy/pasted text from documentation, so ideally we could find examples from real instances.

LaZyDK · 2023-09-19T08:24:05Z

I don't have access to any on prem devices, and I cannot grab Consolidated Events from a management cloud instance.
One SMA manages two or more ESA's.

..._gateway/data_stream/log/_dev/test/pipeline/test-common-consolidated-event.log-expected.json

bhapas

LGTM

The release notes[1] for v15 note that some fields have changed name: - `endTime` => `end` - `startTime` => `start` - `sourceAddress` => `src` - `sourceHostName` => `shost` To maintain backwards compatibility, provide alternations for both cases as pattern definitions in the relevant grok. [1]https://www.cisco.com/c/dam/en/us/td/docs/security/esa/esa15-0/release_notes/Secure_Email_15-0_Release_Notes.pdf After this change, the only place that the old names are found in the elasticsearch directory is in the pattern definitions. ``` $ rg '(endTime|startTime|sourceAddress|sourceHostName)' data_stream/log/elasticsearch data_stream/log/elasticsearch/ingest_pipeline/pipeline_consolidated_event.yml 58: STARTTIME: '(startTime|start)' 59: ENDTIME: '(endTime|end)' 60: SOURCEADDRESS: '(sourceAddress|src)' 61: SOURCEHOSTNAME: '(sourceHostName|shost)' ```

elasticmachine · 2023-09-22T06:13:12Z

Package cisco_secure_email_gateway - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=cisco_secure_email_gateway

elasticmachine · 2023-09-22T06:13:14Z

Package cisco_secure_email_gateway - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=cisco_secure_email_gateway

elasticmachine · 2023-09-22T06:13:16Z

Package cisco_secure_email_gateway - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=cisco_secure_email_gateway

efd6 added enhancement New feature or request Team:Security-External Integrations Integration:cisco_secure_email_gateway Cisco Secure Email Gateway labels Sep 14, 2023

efd6 self-assigned this Sep 14, 2023

efd6 force-pushed the 7806-cisco_secure_email_gateway branch from 1705603 to cb56cea Compare September 14, 2023 00:50

efd6 marked this pull request as ready for review September 14, 2023 01:18

efd6 requested a review from a team as a code owner September 14, 2023 01:18

bhapas reviewed Sep 18, 2023

View reviewed changes

efd6 mentioned this pull request Sep 20, 2023

[cisco_secure_email_gateway] Missing event.start mapping #7877

Closed

efd6 force-pushed the 7806-cisco_secure_email_gateway branch from a87f641 to 9d55351 Compare September 20, 2023 12:47

efd6 requested a review from bhapas September 20, 2023 12:47

bhapas reviewed Sep 20, 2023

View reviewed changes

..._gateway/data_stream/log/_dev/test/pipeline/test-common-consolidated-event.log-expected.json Outdated Show resolved Hide resolved

efd6 force-pushed the 7806-cisco_secure_email_gateway branch from 9d55351 to c452a23 Compare September 21, 2023 09:04

efd6 requested a review from bhapas September 21, 2023 20:50

bhapas approved these changes Sep 22, 2023

View reviewed changes

efd6 force-pushed the 7806-cisco_secure_email_gateway branch from c452a23 to 793f6cc Compare September 22, 2023 04:34

add test case and fix event.{start,end} and priority handling

bb76772

efd6 force-pushed the 7806-cisco_secure_email_gateway branch from 793f6cc to bb76772 Compare September 22, 2023 05:30

efd6 merged commit 0bf108a into elastic:main Sep 22, 2023

efd6 deleted the 7806-cisco_secure_email_gateway branch February 5, 2025 22:04

Conversation

efd6 commented Sep 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Sep 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Sep 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

elasticmachine commented Sep 14, 2023

Uh oh!

bhapas left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 commented Sep 19, 2023

Uh oh!

bhapas commented Sep 19, 2023

Uh oh!

efd6 commented Sep 19, 2023

Uh oh!

bhapas commented Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LaZyDK commented Sep 19, 2023

Uh oh!

efd6 commented Sep 19, 2023

Uh oh!

LaZyDK commented Sep 19, 2023

Uh oh!

Uh oh!

bhapas left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Sep 22, 2023

Uh oh!

elasticmachine commented Sep 22, 2023

Uh oh!

elasticmachine commented Sep 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

efd6 commented Sep 14, 2023 •

edited

Loading

elasticmachine commented Sep 14, 2023 •

edited

Loading

elasticmachine commented Sep 14, 2023 •

edited

Loading

bhapas commented Sep 19, 2023 •

edited

Loading