Fix parsing error in CiscoSecureEmail V15 by pkoutsovasilis · Pull Request #9474 · elastic/integrations

pkoutsovasilis · 2024-03-29T16:04:28Z

Proposed commit message

This PR refactors the extraction and parsing of the fields of a consolidated event for the cisco_secure_email_gateway integration. Specifically the issue is that the order of the keys (key1=val1 key2=val) defined here is not guaranteed. Thus this PR revisits this integration and makes it parsing all the possible fields without relying in order of appearance. The fixes are present even for pre-existing testing logs like missing verdicts here, missing in tls info here, missing subject here

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

N/A

How to test this PR locally

Related issues

Relates https://github.com/elastic/sdh-beats/issues/4566

Screenshots

N/A

elasticmachine · 2024-03-29T16:24:00Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

…consolidated event

narph · 2024-04-02T15:06:35Z

@elastic/security-service-integrations can you help with a review here?

packages/cisco_secure_email_gateway/changelog.yml

..._email_gateway/data_stream/log/elasticsearch/ingest_pipeline/pipeline_consolidated_event.yml

…ld for better grouping

…for ip types

… the mail subject

… the cs4 field

..._gateway/data_stream/log/_dev/test/pipeline/test-common-consolidated-event.log-expected.json

…lude them in the reported error

…raction to improve readability

efd6 · 2024-04-02T21:57:51Z

...o_secure_email_gateway/data_stream/log/_dev/test/pipeline/test-common-consolidated-event.log

+<14>Jul 04 06:21:54 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.3.0-032|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=4FEF3A4372664BCCB404-20EE1767D434 ESAMID=786324 ESAICID=35635425 ESADCID=970897 endTime=Tue Jul  4 06:21:54 2023 ESADKIMVerdict=pass ESADLPVerdict=NOT_EVALUATED dvc=81.2.69.144 ESAFriendlyFrom=Beaches <playas@example.com> ESAGMVerdict=NEGATIVE startTime=Tue Jul  4 14:42:34 2023 deviceInboundInterface=IncomingMail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=westinghouse-thoreau\\=example.com@example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=NZ ESAMFVerdict=MATCH act=QUARANTINED ESAFinalActionDetails=To SPAM cs4Label=ExternalMsgID cs4='<490b2a15fa4742331779cdaa4e@example.com>' ESAMsgSize=20668 ESAOFVerdict=NEGATIVE duser=thoreau@example.com ESAHeloDomain=example.com ESAHeloIP=89.160.20.112 ESAReplyTo=lane@example.com cfp1Label=SBRSScore cfp1=None ESASDRDomainAge=30 days (or greater) cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Neutral ESASPFVerdict={'mailfrom': {'result': 'Pass', 'sender': 'westinghouse-thoreau=example.com@example.com'}, 'helo': {'result': 'Pass', 'sender': 'postmaster@example.com'}} sourceHostName=example.com sourceAddress=89.160.20.112 msg="Totally not suspicious email subject" ESATLSInCipher=ECDHE-RSA-AES256-GCM-SHA384 ESATLSInConnStatus=Success ESATLSInProtocol=TLSv1.2 ESAAMPVerdict=SKIPPED ESAASVerdict=SUSPECT ESAAVVerdict=NOT_EVALUATED ESACFVerdict=NO_MATCH
+<14>Jul 04 06:21:54 test.hostname.co consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.3.0-032|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=4FEF3A4372664BCCB404-20EE1767D434 ESAMID=786324 ESAICID=35635425 ESADCID=970897 endTime=Tue Jul  4 06:21:54 2023 ESADKIMVerdict=pass ESADLPVerdict=NOT_EVALUATED dvc=81.2.69.144 ESAFriendlyFrom=Beaches <playas@example.com> ESAGMVerdict=NEGATIVE startTime=Tue Jul  4 14:42:34 2023 deviceInboundInterface=IncomingMail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=westinghouse-thoreau\\=example.com@example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=NZ ESAMFVerdict=MATCH act=QUARANTINED ESAFinalActionDetails=To SPAM cs4Label=ExternalMsgID cs4='<490b2a15fa4742331779cdaa4e@example.com>' ESAMsgSize=20668 ESAOFVerdict=NEGATIVE duser=thoreau@example.com ESAHeloDomain=example.com ESAHeloIP=89.160.20.112 ESAReplyTo=lane@example.com cfp1Label=SBRSScore cfp1=None ESASDRDomainAge=30 days (or greater) cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Neutral ESASPFVerdict={'mailfrom': {'result': 'Pass', 'sender': 'westinghouse-thoreau=example.com@example.com'}, 'helo': {'result': 'Pass', 'sender': 'postmaster@example.com'}} sourceHostName=example.com sourceAddress=89.160.20.112 msg="Totally not suspicious email subject" ESATLSInCipher=ECDHE-RSA-AES256-GCM-SHA384 ESATLSInConnStatus=Success ESATLSInProtocol=TLSv1.2 ESAAMPVerdict=SKIPPED ESAASVerdict=SUSPECT ESAAVVerdict=NOT_EVALUATED ESACFVerdict=NO_MATCH


For other reviewers, the escaping that is removed here was incorrectly introduced in #6879.

..._gateway/data_stream/log/_dev/test/pipeline/test-common-consolidated-event.log-expected.json

…omain, sourceHostName, and shost

elasticmachine · 2024-04-03T09:22:38Z

💚 Build Succeeded

Buildkite Build
Commit: 2a9150c

History

💚 Build #10034 succeeded a13e9b2
💚 Build #10030 succeeded 62bd7a5
💚 Build #9988 succeeded 5405909
💚 Build #9984 succeeded 0e36acdce9511eb9f8dbcfdd8def79b65f790522
💚 Build #9970 succeeded 923b58193f7c7e0037286cc1b9d8c10431a51918

elastic-sonarqube · 2024-04-03T09:22:40Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
100.0% 100.0% Coverage on New Code
0.0% 0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-04-03T13:44:17Z

Package cisco_secure_email_gateway - 1.23.1 containing this change is available at https://epr.elastic.co/search?package=cisco_secure_email_gateway

pkoutsovasilis force-pushed the pkoutsovasilis/fix_cisco_secure_email_gateway branch from 15df2e2 to 923b581 Compare March 29, 2024 16:09

pkoutsovasilis marked this pull request as ready for review March 29, 2024 16:11

pkoutsovasilis requested a review from a team as a code owner March 29, 2024 16:11

fix[cisco_secure_email_gateway]: revisit parsing of all fields for a …

5405909

…consolidated event

pkoutsovasilis force-pushed the pkoutsovasilis/fix_cisco_secure_email_gateway branch from 0e36acd to 5405909 Compare April 1, 2024 14:35

narph requested a review from a team April 2, 2024 14:52

efd6 reviewed Apr 2, 2024

View reviewed changes

pkoutsovasilis added 6 commits April 2, 2024 23:27

fix(cisco_secure_email_gateway): improve changelog description

2495bae

fix(cisco_secure_email_gateway): reorder target_field right below fie…

5b5661c

…ld for better grouping

fix(cisco_secure_email_gateway): yaml indentation

44aebbf

fix(cisco_secure_email_gateway): replace grok processor with convert …

4c6e903

…for ip types

fix(cisco_secure_email_gateway): replace grok processor with gsub for…

e3cbfa2

… the mail subject

fix(cisco_secure_email_gateway): replace grok processor with gsub for…

63b0854

… the cs4 field

efd6 reviewed Apr 2, 2024

View reviewed changes

..._gateway/data_stream/log/_dev/test/pipeline/test-common-consolidated-event.log-expected.json Outdated Show resolved Hide resolved

pkoutsovasilis added 3 commits April 3, 2024 00:00

fix(cisco_secure_email_gateway): add tag on possible failures and inc…

62bd7a5

…lude them in the reported error

fix(cisco_secure_email_gateway): reword pattern for email.subject ext…

234b9f6

…raction to improve readability

fix(cisco_secure_email_gateway): remove escaped quotes from test input

a13e9b2

efd6 approved these changes Apr 2, 2024

View reviewed changes

bhapas reviewed Apr 3, 2024

View reviewed changes

..._gateway/data_stream/log/_dev/test/pipeline/test-common-consolidated-event.log-expected.json Show resolved Hide resolved

..._gateway/data_stream/log/_dev/test/pipeline/test-common-consolidated-event.log-expected.json Show resolved Hide resolved

fix(cisco_secure_email_gateway): use urldecode processor for ESAHeloD…

2a9150c

…omain, sourceHostName, and shost

aleksmaus approved these changes Apr 3, 2024

View reviewed changes

pkoutsovasilis merged commit 09ef12c into main Apr 3, 2024

pkoutsovasilis deleted the pkoutsovasilis/fix_cisco_secure_email_gateway branch April 3, 2024 13:30

andrewkroh added the Integration:cisco_secure_email_gateway Cisco Secure Email Gateway label Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix parsing error in CiscoSecureEmail V15#9474

Fix parsing error in CiscoSecureEmail V15#9474
pkoutsovasilis merged 11 commits intomainfrom
pkoutsovasilis/fix_cisco_secure_email_gateway

pkoutsovasilis commented Mar 29, 2024 •

edited

Loading

Uh oh!

elasticmachine commented Mar 29, 2024

Uh oh!

narph commented Apr 2, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 Apr 2, 2024

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Apr 3, 2024

Uh oh!

elastic-sonarqube bot commented Apr 3, 2024

Uh oh!

elasticmachine commented Apr 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

		<14>Jul 04 06:21:54 consolidated_event: CEF:0\|Cisco\|C100V Email Security Virtual Appliance\|14.3.0-032\|ESA_CONSOLIDATED_LOG_EVENT\|Consolidated Log Event\|5\|deviceExternalId=4FEF3A4372664BCCB404-20EE1767D434 ESAMID=786324 ESAICID=35635425 ESADCID=970897 endTime=Tue Jul 4 06:21:54 2023 ESADKIMVerdict=pass ESADLPVerdict=NOT_EVALUATED dvc=81.2.69.144 ESAFriendlyFrom=Beaches <playas@example.com> ESAGMVerdict=NEGATIVE startTime=Tue Jul 4 14:42:34 2023 deviceInboundInterface=IncomingMail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=westinghouse-thoreau\\=example.com@example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=NZ ESAMFVerdict=MATCH act=QUARANTINED ESAFinalActionDetails=To SPAM cs4Label=ExternalMsgID cs4='<490b2a15fa4742331779cdaa4e@example.com>' ESAMsgSize=20668 ESAOFVerdict=NEGATIVE duser=thoreau@example.com ESAHeloDomain=example.com ESAHeloIP=89.160.20.112 ESAReplyTo=lane@example.com cfp1Label=SBRSScore cfp1=None ESASDRDomainAge=30 days (or greater) cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Neutral ESASPFVerdict={'mailfrom': {'result': 'Pass', 'sender': 'westinghouse-thoreau=example.com@example.com'}, 'helo': {'result': 'Pass', 'sender': 'postmaster@example.com'}} sourceHostName=example.com sourceAddress=89.160.20.112 msg="Totally not suspicious email subject" ESATLSInCipher=ECDHE-RSA-AES256-GCM-SHA384 ESATLSInConnStatus=Success ESATLSInProtocol=TLSv1.2 ESAAMPVerdict=SKIPPED ESAASVerdict=SUSPECT ESAAVVerdict=NOT_EVALUATED ESACFVerdict=NO_MATCH
		<14>Jul 04 06:21:54 test.hostname.co consolidated_event: CEF:0\|Cisco\|C100V Email Security Virtual Appliance\|14.3.0-032\|ESA_CONSOLIDATED_LOG_EVENT\|Consolidated Log Event\|5\|deviceExternalId=4FEF3A4372664BCCB404-20EE1767D434 ESAMID=786324 ESAICID=35635425 ESADCID=970897 endTime=Tue Jul 4 06:21:54 2023 ESADKIMVerdict=pass ESADLPVerdict=NOT_EVALUATED dvc=81.2.69.144 ESAFriendlyFrom=Beaches <playas@example.com> ESAGMVerdict=NEGATIVE startTime=Tue Jul 4 14:42:34 2023 deviceInboundInterface=IncomingMail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=westinghouse-thoreau\\=example.com@example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=NZ ESAMFVerdict=MATCH act=QUARANTINED ESAFinalActionDetails=To SPAM cs4Label=ExternalMsgID cs4='<490b2a15fa4742331779cdaa4e@example.com>' ESAMsgSize=20668 ESAOFVerdict=NEGATIVE duser=thoreau@example.com ESAHeloDomain=example.com ESAHeloIP=89.160.20.112 ESAReplyTo=lane@example.com cfp1Label=SBRSScore cfp1=None ESASDRDomainAge=30 days (or greater) cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Neutral ESASPFVerdict={'mailfrom': {'result': 'Pass', 'sender': 'westinghouse-thoreau=example.com@example.com'}, 'helo': {'result': 'Pass', 'sender': 'postmaster@example.com'}} sourceHostName=example.com sourceAddress=89.160.20.112 msg="Totally not suspicious email subject" ESATLSInCipher=ECDHE-RSA-AES256-GCM-SHA384 ESATLSInConnStatus=Success ESATLSInProtocol=TLSv1.2 ESAAMPVerdict=SKIPPED ESAASVerdict=SUSPECT ESAAVVerdict=NOT_EVALUATED ESACFVerdict=NO_MATCH

Conversation

pkoutsovasilis commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Mar 29, 2024

🚀 Benchmarks report

Uh oh!

narph commented Apr 2, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 Apr 2, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Apr 3, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Apr 3, 2024

Quality Gate passed

Uh oh!

elasticmachine commented Apr 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

pkoutsovasilis commented Mar 29, 2024 •

edited

Loading