[auditd_manager] Update fields and sample_event.json#12541
[auditd_manager] Update fields and sample_event.json#12541mrodm merged 11 commits intoelastic:mainfrom
Conversation
There was a problem hiding this comment.
If this field is declared as flattened, this is not created as mapping currently.
Probably, because there are other fields auditd.data.xxx.
If it set as a dynamic template, it keeps all the fields above plus the ones not able to match (auditd.data.a0-N)
There was a problem hiding this comment.
If flattened is expected maybe this should be defined without the wildcard:
- name: auditd.data
description: Auditd related data
type: flattened
There was a problem hiding this comment.
If that definition is changed as:
- name: auditd.data
description: Auditd related data
type: flattenedThis mapping is not created. I think this is caused because there are other definitions present for auditd.data.<field> fields. For instance auditd.data.action (as keyword).
If it is required to be flattened, I'm afraid it should be needed to remove all the auditd.data.<field> definitions.
There was a problem hiding this comment.
This mapping is not created. I think this is caused because there are other definitions present for auditd.data. fields. For instance auditd.data.action (as keyword).
On what version of the stack? This looks like a case of elastic/kibana#204104
There was a problem hiding this comment.
I did run the tests with 8.16.2 (Kibana version set in the manifest).
Just tested with 8.18.0-SNAPSHOT and it does not generate the flattened type for auditd.data neither.
Complete mapping for `auditd.data` (8.18.0-SNAPSHOT)
"auditd": {
"properties": {
"data": {
"properties": {
"acct": {
"type": "keyword",
"ignore_above": 1024
},
"acl": {
"type": "keyword",
"ignore_above": 1024
},
"action": {
"type": "keyword",
"ignore_above": 1024
},
"added": {
"type": "long"
},
"addr": {
"type": "keyword",
"ignore_above": 1024
},
"apparmor": {
"type": "keyword",
"ignore_above": 1024
},
"arch": {
"type": "keyword",
"ignore_above": 1024
},
"argc": {
"type": "long"
},
"audit_backlog_limit": {
"type": "keyword",
"ignore_above": 1024
},
"audit_backlog_wait_time": {
"type": "keyword",
"ignore_above": 1024
},
"audit_enabled": {
"type": "keyword",
"ignore_above": 1024
},
"audit_failure": {
"type": "keyword",
"ignore_above": 1024
},
"audit_pid": {
"type": "long"
},
"auid": {
"type": "keyword",
"ignore_above": 1024
},
"banners": {
"type": "keyword",
"ignore_above": 1024
},
"bool": {
"type": "keyword",
"ignore_above": 1024
},
"bus": {
"type": "keyword",
"ignore_above": 1024
},
"cap_fe": {
"type": "keyword",
"ignore_above": 1024
},
"cap_fi": {
"type": "keyword",
"ignore_above": 1024
},
"cap_fp": {
"type": "keyword",
"ignore_above": 1024
},
"cap_fver": {
"type": "keyword",
"ignore_above": 1024
},
"cap_pe": {
"type": "keyword",
"ignore_above": 1024
},
"cap_pi": {
"type": "keyword",
"ignore_above": 1024
},
"cap_pp": {
"type": "keyword",
"ignore_above": 1024
},
"capability": {
"type": "keyword",
"ignore_above": 1024
},
"cgroup": {
"type": "keyword",
"ignore_above": 1024
},
"changed": {
"type": "long"
},
"cipher": {
"type": "keyword",
"ignore_above": 1024
},
"class": {
"type": "keyword",
"ignore_above": 1024
},
"cmd": {
"type": "keyword",
"ignore_above": 1024
},
"code": {
"type": "keyword",
"ignore_above": 1024
},
"compat": {
"type": "keyword",
"ignore_above": 1024
},
"daddr": {
"type": "ip"
},
"data": {
"type": "keyword",
"ignore_above": 1024
},
"default_context": {
"type": "keyword",
"ignore_above": 1024
},
"device": {
"type": "keyword",
"ignore_above": 1024
},
"dir": {
"type": "keyword",
"ignore_above": 1024
},
"direction": {
"type": "keyword",
"ignore_above": 1024
},
"dmac": {
"type": "keyword",
"ignore_above": 1024
},
"dport": {
"type": "long"
},
"enforcing": {
"type": "keyword",
"ignore_above": 1024
},
"entries": {
"type": "long"
},
"exit": {
"type": "keyword",
"ignore_above": 1024
},
"fam": {
"type": "keyword",
"ignore_above": 1024
},
"family": {
"type": "keyword",
"ignore_above": 1024
},
"fd": {
"type": "keyword",
"ignore_above": 1024
},
"fe": {
"type": "keyword",
"ignore_above": 1024
},
"feature": {
"type": "keyword",
"ignore_above": 1024
},
"fi": {
"type": "keyword",
"ignore_above": 1024
},
"file": {
"type": "keyword",
"ignore_above": 1024
},
"flags": {
"type": "keyword",
"ignore_above": 1024
},
"format": {
"type": "keyword",
"ignore_above": 1024
},
"fp": {
"type": "keyword",
"ignore_above": 1024
},
"frootid": {
"type": "keyword",
"ignore_above": 1024
},
"fver": {
"type": "keyword",
"ignore_above": 1024
},
"grantors": {
"type": "keyword",
"ignore_above": 1024
},
"grp": {
"type": "keyword",
"ignore_above": 1024
},
"hook": {
"type": "keyword",
"ignore_above": 1024
},
"hostname": {
"type": "keyword",
"ignore_above": 1024
},
"icmp_type": {
"type": "keyword",
"ignore_above": 1024
},
"id": {
"type": "keyword",
"ignore_above": 1024
},
"igid": {
"type": "keyword",
"ignore_above": 1024
},
"img_ctx": {
"type": "keyword",
"ignore_above": 1024
},
"inif": {
"type": "keyword",
"ignore_above": 1024
},
"ino": {
"type": "keyword",
"ignore_above": 1024
},
"inode_gid": {
"type": "keyword",
"ignore_above": 1024
},
"inode_uid": {
"type": "keyword",
"ignore_above": 1024
},
"invalid_context": {
"type": "keyword",
"ignore_above": 1024
},
"ioctlcmd": {
"type": "keyword",
"ignore_above": 1024
},
"ip": {
"type": "ip"
},
"ipid": {
"type": "keyword",
"ignore_above": 1024
},
"ipx_net": {
"type": "keyword",
"ignore_above": 1024
},
"items": {
"type": "long"
},
"iuid": {
"type": "keyword",
"ignore_above": 1024
},
"kernel": {
"type": "keyword",
"ignore_above": 1024
},
"kind": {
"type": "keyword",
"ignore_above": 1024
},
"ksize": {
"type": "keyword",
"ignore_above": 1024
},
"laddr": {
"type": "keyword",
"ignore_above": 1024
},
"len": {
"type": "keyword",
"ignore_above": 1024
},
"list": {
"type": "keyword",
"ignore_above": 1024
},
"lport": {
"type": "long"
},
"mac": {
"type": "keyword",
"ignore_above": 1024
},
"macproto": {
"type": "keyword",
"ignore_above": 1024
},
"maj": {
"type": "keyword",
"ignore_above": 1024
},
"major": {
"type": "keyword",
"ignore_above": 1024
},
"minor": {
"type": "keyword",
"ignore_above": 1024
},
"model": {
"type": "keyword",
"ignore_above": 1024
},
"msg": {
"type": "keyword",
"ignore_above": 1024
},
"nargs": {
"type": "long"
},
"net": {
"type": "keyword",
"ignore_above": 1024
},
"new": {
"type": "keyword",
"ignore_above": 1024
},
"new_chardev": {
"type": "keyword",
"ignore_above": 1024
},
"new_disk": {
"type": "keyword",
"ignore_above": 1024
},
"new_enabled": {
"type": "keyword",
"ignore_above": 1024
},
"new_fs": {
"type": "keyword",
"ignore_above": 1024
},
"new_gid": {
"type": "keyword",
"ignore_above": 1024
},
"new_level": {
"type": "keyword",
"ignore_above": 1024
},
"new_lock": {
"type": "keyword",
"ignore_above": 1024
},
"new_log_passwd": {
"type": "keyword",
"ignore_above": 1024
},
"new_mem": {
"type": "keyword",
"ignore_above": 1024
},
"new_net": {
"type": "keyword",
"ignore_above": 1024
},
"new_pe": {
"type": "keyword",
"ignore_above": 1024
},
"new_pi": {
"type": "keyword",
"ignore_above": 1024
},
"new_pp": {
"type": "keyword",
"ignore_above": 1024
},
"new_range": {
"type": "keyword",
"ignore_above": 1024
},
"new_rng": {
"type": "keyword",
"ignore_above": 1024
},
"new_role": {
"type": "keyword",
"ignore_above": 1024
},
"new_ses": {
"type": "keyword",
"ignore_above": 1024
},
"new_seuser": {
"type": "keyword",
"ignore_above": 1024
},
"new_vcpu": {
"type": "long"
},
"nlnk_fam": {
"type": "keyword",
"ignore_above": 1024
},
"nlnk_grp": {
"type": "keyword",
"ignore_above": 1024
},
"nlnk_pid": {
"type": "long"
},
"oauid": {
"type": "keyword",
"ignore_above": 1024
},
"obj": {
"type": "keyword",
"ignore_above": 1024
},
"obj_gid": {
"type": "keyword",
"ignore_above": 1024
},
"obj_uid": {
"type": "keyword",
"ignore_above": 1024
},
"ocomm": {
"type": "keyword",
"ignore_above": 1024
},
"oflag": {
"type": "keyword",
"ignore_above": 1024
},
"old": {
"type": "keyword",
"ignore_above": 1024
},
"old_auid": {
"type": "keyword",
"ignore_above": 1024
},
"old_chardev": {
"type": "keyword",
"ignore_above": 1024
},
"old_disk": {
"type": "keyword",
"ignore_above": 1024
},
"old_enabled": {
"type": "keyword",
"ignore_above": 1024
},
"old_enforcing": {
"type": "keyword",
"ignore_above": 1024
},
"old_fs": {
"type": "keyword",
"ignore_above": 1024
},
"old_level": {
"type": "keyword",
"ignore_above": 1024
},
"old_lock": {
"type": "keyword",
"ignore_above": 1024
},
"old_log_passwd": {
"type": "keyword",
"ignore_above": 1024
},
"old_mem": {
"type": "keyword",
"ignore_above": 1024
},
"old_net": {
"type": "keyword",
"ignore_above": 1024
},
"old_pa": {
"type": "keyword",
"ignore_above": 1024
},
"old_pe": {
"type": "keyword",
"ignore_above": 1024
},
"old_pi": {
"type": "keyword",
"ignore_above": 1024
},
"old_pp": {
"type": "keyword",
"ignore_above": 1024
},
"old_prom": {
"type": "keyword",
"ignore_above": 1024
},
"old_range": {
"type": "keyword",
"ignore_above": 1024
},
"old_rng": {
"type": "keyword",
"ignore_above": 1024
},
"old_role": {
"type": "keyword",
"ignore_above": 1024
},
"old_ses": {
"type": "keyword",
"ignore_above": 1024
},
"old_seuser": {
"type": "keyword",
"ignore_above": 1024
},
"old_val": {
"type": "keyword",
"ignore_above": 1024
},
"old_vcpu": {
"type": "long"
},
"op": {
"type": "keyword",
"ignore_above": 1024
},
"opid": {
"type": "long"
},
"oses": {
"type": "keyword",
"ignore_above": 1024
},
"outif": {
"type": "keyword",
"ignore_above": 1024
},
"pa": {
"type": "keyword",
"ignore_above": 1024
},
"parent": {
"type": "keyword",
"ignore_above": 1024
},
"pe": {
"type": "keyword",
"ignore_above": 1024
},
"per": {
"type": "keyword",
"ignore_above": 1024
},
"perm": {
"type": "keyword",
"ignore_above": 1024
},
"perm_mask": {
"type": "keyword",
"ignore_above": 1024
},
"permissive": {
"type": "keyword",
"ignore_above": 1024
},
"pfs": {
"type": "keyword",
"ignore_above": 1024
},
"pi": {
"type": "keyword",
"ignore_above": 1024
},
"pp": {
"type": "keyword",
"ignore_above": 1024
},
"printer": {
"type": "keyword",
"ignore_above": 1024
},
"prom": {
"type": "keyword",
"ignore_above": 1024
},
"proto": {
"type": "keyword",
"ignore_above": 1024
},
"qbytes": {
"type": "keyword",
"ignore_above": 1024
},
"range": {
"type": "keyword",
"ignore_above": 1024
},
"reason": {
"type": "keyword",
"ignore_above": 1024
},
"removed": {
"type": "long"
},
"res": {
"type": "keyword",
"ignore_above": 1024
},
"reset": {
"type": "keyword",
"ignore_above": 1024
},
"resrc": {
"type": "keyword",
"ignore_above": 1024
},
"result": {
"type": "keyword",
"ignore_above": 1024
},
"rport": {
"type": "long"
},
"sauid": {
"type": "keyword",
"ignore_above": 1024
},
"scontext": {
"type": "keyword",
"ignore_above": 1024
},
"selected_context": {
"type": "keyword",
"ignore_above": 1024
},
"seperm": {
"type": "keyword",
"ignore_above": 1024
},
"seperms": {
"type": "keyword",
"ignore_above": 1024
},
"seqno": {
"type": "long"
},
"seresult": {
"type": "keyword",
"ignore_above": 1024
},
"ses": {
"type": "keyword",
"ignore_above": 1024
},
"seuser": {
"type": "keyword",
"ignore_above": 1024
},
"sig": {
"type": "keyword",
"ignore_above": 1024
},
"sigev_signo": {
"type": "keyword",
"ignore_above": 1024
},
"smac": {
"type": "keyword",
"ignore_above": 1024
},
"socket": {
"properties": {
"addr": {
"type": "keyword",
"ignore_above": 1024
},
"family": {
"type": "keyword",
"ignore_above": 1024
},
"path": {
"type": "keyword",
"ignore_above": 1024
},
"port": {
"type": "long"
},
"saddr": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"spid": {
"type": "long"
},
"sport": {
"type": "long"
},
"state": {
"type": "keyword",
"ignore_above": 1024
},
"subj": {
"type": "keyword",
"ignore_above": 1024
},
"success": {
"type": "keyword",
"ignore_above": 1024
},
"syscall": {
"type": "keyword",
"ignore_above": 1024
},
"table": {
"type": "keyword",
"ignore_above": 1024
},
"tclass": {
"type": "keyword",
"ignore_above": 1024
},
"tcontext": {
"type": "keyword",
"ignore_above": 1024
},
"terminal": {
"type": "keyword",
"ignore_above": 1024
},
"tty": {
"type": "keyword",
"ignore_above": 1024
},
"unit": {
"type": "keyword",
"ignore_above": 1024
},
"uri": {
"type": "keyword",
"ignore_above": 1024
},
"uuid": {
"type": "keyword",
"ignore_above": 1024
},
"val": {
"type": "keyword",
"ignore_above": 1024
},
"ver": {
"type": "keyword",
"ignore_above": 1024
},
"virt": {
"type": "keyword",
"ignore_above": 1024
},
"vm": {
"type": "keyword",
"ignore_above": 1024
},
"vm_ctx": {
"type": "keyword",
"ignore_above": 1024
},
"vm_pid": {
"type": "long"
},
"watch": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
}Just realized that there are at least another group socket under auditd.data:
- name: auditd.data.socket.port
description: The port number.
type: long
- name: auditd.data.socket.saddr
description: The raw socket address structure.
type: keyword
- name: auditd.data.socket.addr
description: The remote address.
type: keyword
- name: auditd.data.socket.family
description: The socket family (unix, ipv4, ipv6, netlink).
type: keyword
- name: auditd.data.socket.path
description: This is the path associated with a unix socket.
type: keywordCould this auditd.data.socket field cause that it is better then set a dynamic template (keyword) instead for the missing fields (e.g. auditd.data.a0) ?
There was a problem hiding this comment.
Yeah, it is probably not possible to merge a flattened with a group. The fleet code definitely does not support it, it only merges groups, objects and maybe nested. It would probably not make sense to merge a flattened with other types.
So we should probably avoid flattened in this case, using a wildcard and considering subobjects: false.
There was a problem hiding this comment.
Keeping all the definitions present in the file (except auditd.data.a0-N), I think of these two options to update the definition for auditd.data.*:
- Mapping all strings to keyword
- name: auditd.data.*
description: Auditd related data
type: keyword- Mapping everything as a keyword (even fields whose type would be
long)
- name: auditd.data.*
description: Auditd related data
type: object
object_type: keyword
object_type_mapping_type: "*"Maybe better use the first option, and if it is required, add new corresponding definitions in the future, as it happens for auditd.data.socket.port for instance. To have a better control in the fields. WDYT?
There was a problem hiding this comment.
What should it be done for this field definition ? Just remove it?
There was a problem hiding this comment.
Would it work with a auditd.data.a* definition, specially to keep the documentation?
| # this mapping does not generate a dynamic template, and the expected fields do not match | |
| # should it be kept for documentation purposes? | |
| # - name: auditd.data.a0-N | |
| # description: the arguments to a syscall | |
| # type: keyword | |
| - name: auditd.data.a* | |
| description: the arguments to a syscall | |
| type: keyword |
Though this would match also things that are not arguments.
So maybe this can be removed, yes.
There was a problem hiding this comment.
It was required to re-generate the sample_event since this field is now a long (as in the field definition).
🚀 Benchmarks reportTo see the full report comment with |
e32c560 to
f54601d
Compare
1e8b1b3 to
af08a22
Compare
…41bb116f37d13f2d4e04502d0
(cherry picked from commit fa96beb)
6b08b22 to
d2ba547
Compare
This reverts commit d2ba547.
.buildkite/scripts/common.sh
Outdated
|
|
||
| list_all_directories() { | ||
| find . -maxdepth 1 -mindepth 1 -type d | xargs -I {} basename {} | sort | ||
| find . -maxdepth 1 -mindepth 1 -type d | xargs -I {} basename {} | sort |grep auditd_manager |
There was a problem hiding this comment.
To be removed changes in .buildkite/ folder.
|
Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform) |
|
Hi @elastic/sec-linux-platform, Thanks in advance! |
| - name: auditd.data.result | ||
| type: keyword | ||
| - name: auditd.data | ||
| - name: auditd.data.* |
There was a problem hiding this comment.
My interpretation of the original intent with auditd.data.a0-N is to path_match auditd.data.a* and set a dynamic keyword. So I am thinking we should make it auditd.data.a*. Ideally we could use a match regex to only map auditd.data.a\d+ to keyword, this way if there happens be some non-argument field we get alerted to it not having a specific mapping during testing.
There should be specific mappings for the other audit.data fields already. 🤞
There was a problem hiding this comment.
In that case, it would be needed to keep these two field definitions @andrewkroh :
- name: auditd.data.a*
description: the arguments to a syscall
type: keyword
- name: auditd.data.*
description: Auditd related data
type: keywordUpdated in 1aa7413
Currently, the second one auditd.data.* is still needed. If it is not added, there is an error in system tests:
[0] field "auditd.data.subj_user" is undefined
For the auditd.data.* field definition, it cannot be set flattened type as auditd.data (see https://github.com/elastic/integrations/pull/12541/files#r1946774552).
There was a problem hiding this comment.
[0] field "auditd.data.subj_user" is undefined
It does look like we are missing mappings for the SELinux subject properties like seen in this data.
Here are some additions that should cover these missing fields. Then auditd.data.* won't be needed for the tests to pass. 🤞
- name: auditd.data.subj_user
type: keyword
description: >
The SELinux user identity. This represents the SELinux user role that is
assigned to the subject (user or process) performing an action. It's part
of the SELinux security context and is used to enforce policies that
restrict what actions a subject can perform.
- name: auditd.data.subj_role
type: keyword
description: >
The SELinux role associated with the subject. The role determines the
capabilities a subject has within a given SELinux policy. Roles are used
to define higher-level security attributes in the context of the system's
security policies.
- name: auditd.data.subj_domain
type: keyword
description: >
The SELinux domain or type assigned to the subject. The domain specifies
the type of resource or process the subject is interacting with, helping
enforce domain-based access controls, which are crucial in limiting resource
access.
- name: auditd.data.subj_level
type: keyword
description: >
The SELinux sensitivity level for the subject. It indicates the security
classification level, like `s0` or `s2`, that defines how data or processes
are handled based on confidentiality and integrity levels within the system.
- name: auditd.data.subj_category
type: keyword
description: >
The SELinux category associated with the subject. It helps further refine
the level of access by classifying subjects into categories for
multi-level security (MLS). Categories are often used to label data
with additional attributes, like "high" or "low," enhancing granularity.There was a problem hiding this comment.
Added those new fields for SELinux subject properties, thanks!
Just a couple of doubts:
-
Maybe it's expected, but looking at those tests files you linked above, there are some fields that do not have a field definition
auditd.data.exeorauditd.data.proctitle:Are those expected to be ingested too ?
At least running the system tests withelastic-package(whereelastic-agentruns in a docker), those fields are not present in the documents. -
Could you confirm if it should be removed the
auditd.data.*dynamic template after adding those SELinux subject properties?
There was a problem hiding this comment.
The auditd exe and proctitle fields have ECS equivalents which are process.executable and process.title. I would expect that those auditd fields are renamed so that we don't need audit.data mappings for them.
This is a better example of the data sent by auditbeat because it has some ECS mapping applied (whereas the auparse package is only concerned with parsing, not mapping to ECS):
There was a problem hiding this comment.
Could you confirm if it should be removed the auditd.data.* dynamic template after adding those SELinux subject properties?
I think audit.data.* should be removed to encourage developers to explicitly document the fields they are adding in tests. I don't see it impacting users because the dynamic mapping will still exist via all_strings_to_keywords.
andrewkroh
left a comment
There was a problem hiding this comment.
LGTM, but I think you were planning to remove the .buildkite and go.mod changes?
That's right. Those changes have been removed in a5f28a5 |
💚 Build Succeeded
History
cc @mrodm |
|
|
Package auditd_manager - 1.18.3 containing this change is available at https://epr.elastic.co/package/auditd_manager/1.18.3/ |
Update field definitions to be valid found when elastic-package uses the validation based on mappings. Added the missing field definitions for the SELinux subject properties, used a dynamic template for the arguments to the syscall (previously defined as auditd.data.a0-N) and removed the auditd.data (flattened type).
Proposed commit message
Update field definitions to be valid when
elastic-packageuses validation based on mappings.Errors can be found in this PR: https://buildkite.com/elastic/integrations/builds/21222
To solve these issues this PR :
flattenedtype by a dynamic template where all string fields are converted to keywordauditd.data.a0-Nsince this is not taken into account.Checklist
changelog.ymlfile.Author's Checklist
elastic-packageenabled mappings: https://buildkite.com/elastic/integrations/builds/21793.buildkitefolder andgo.mod/go.sumfiles.auditd.data.a0-N).How to test this PR locally
Run elastic-package with mappings validation enabled and run system tests:
Related issues