cscli parsers install crowdsecurity/aws-cloudtrailParser for cloudtrail logs with type aws-cloudtrail.
As cloudtrail logs that are sent to s3 are stored in a json array
object called Records, you may want to use the transform feature of
crowdsec with the following configuration:
map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#)).
Example of acquis.yaml using s3 s3notifications through sqs:
source: s3
polling_method: sqs
sqs_name: <sqs_queue>
sqs_format: s3notification
polling_interval: 30
aws_region: eu-west-1
transform: map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#))
max_buffer_size: 10000000
use_time_machine: true
labels:
type: aws-cloudtrailA direct acquisition method is supported using s3 by directly listing new object in the bucket. In case of high cloudtrail traffic, this is discouraged, because it will require some significant compute resources.
Cloudtrail logs are arriving every few minutes, thus, we can't use the real time feature of crowdsec. That's the reason we are suggesting to use the time machine feature, to take into account the time when they occurred and not when they are sent to CrowdSec.
Please have a look at the documentation https://docs.crowdsec.net/docs/next/data_sources/s3
Cloudtrail logs can be sent to kinesis as well, and crowdsec supports such a source for cloudtrail logs:
source: kinesis
stream_name: cloutrail_stream
aws_region: eu-west-1
from_subscription: true
labels:
type: aws-cloudtrail1onsuccess: next_stage2#debug: true3filter: "evt.Parsed.program == 'aws-cloudtrail'"4name: crowdsecurity/aws-cloudtrail5description: "Parse AWS Cloudtrail logs"6statics:7 - parsed: cloudtrail_parsed8 expression: UnmarshalJSON(evt.Line.Raw, evt.Unmarshaled, 'cloudtrail')9 - target: evt.StrTime10 expression: evt.Unmarshaled.cloudtrail.eventTime11 - meta: user_type12 expression: evt.Unmarshaled.cloudtrail.userIdentity.type13 # see : https://github.com/antonmedv/expr/blob/master/docs/Language-Definition.md#nil-coalescing14 - meta: user_arn15 expression: |16 evt.Unmarshaled.cloudtrail.userIdentity?.arn ?? evt.Unmarshaled.cloudtrail.userIdentity.userName17 - meta: event_name18 expression: evt.Unmarshaled.cloudtrail.eventName19 - meta: event_source20 expression: evt.Unmarshaled.cloudtrail.eventSource21 - meta: region22 expression: evt.Unmarshaled.cloudtrail.awsRegion23 - meta: source_ip24 expression: |25 IsIP(evt.Unmarshaled.cloudtrail.sourceIPAddress) ? evt.Unmarshaled.cloudtrail.sourceIPAddress : ""26 - meta: user_agent27 expression: evt.Unmarshaled.cloudtrail.userAgent28 - meta: error_code29 expression: evt.Unmarshaled.cloudtrail.errorCode30 - meta: event_id31 expression: evt.Unmarshaled.cloudtrail.eventID32 - meta: account_id33 expression: evt.Unmarshaled.cloudtrail.userIdentity.accountId34 - meta: log_type35 value: aws-cloudtrail36