How to send EKS logs to AWS OpenSearch by Fluent-Bit
We know Kubernetes usually does not persist data. We needs to find a place to store application logs. If you are using AWS EKS. CloudWatch is an official suggestion. But recently I got a system it have to store in AWS OpenSearch. Not much information I can find related to it. So, I want to document how I set up the whole app log flow for me or someone’s reference in the future.
Log Flow
The whole app log flow is pod stdout → Fluent-Bit → AWS Kinesis Data Stream → AWS Data Firehose → AWS OpenSearch
As I know Fluent-Bit can send app-logs to OpenSearch directly. But if you have a lot of application and traffic. You should consider using Kinesis Data Streams & Dta Firehose in the middle to be the buffer.
Fluent-Bit
You can follow the AWS document to install Fluent-Bit into your EKS. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html
Please modify the k8s resource template .yaml file according your requirement to deploy it.
ConfigMap fluent-bit-config
application-log.conf & parsers.conf. [OUTPUT] kinesis and [PARSER] regex are important.
For linux
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/name: fluentbit
name: fluent-bit-config
namespace: mts-infra
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Grace 30
Log_Level error
Daemon off
Parsers_File parsers.conf
HTTP_Server ${HTTP_SERVER}
HTTP_Listen 0.0.0.0
HTTP_Port ${HTTP_PORT}
storage.path /var/fluent-bit/state/flb-storage/
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 5M
@INCLUDE application-log.conf
application-log.conf: |
[INPUT]
Name tail
Tag application.*
Exclude_Path /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
Path /var/log/containers/*.log
multiline.parser docker, cri
DB /var/fluent-bit/state/flb_container.db
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Refresh_Interval 10
[FILTER]
Name grep
Match *
Regex log loggly
[FILTER]
Name parser
Match **
Parser loggly
Key_Name log
Reserve_Data On
[FILTER]
Name record_modifier
Match *
Remove_key stream
[OUTPUT]
Name kinesis
Match **
region ${AWS_REGION}
stream ${MY_APP_LOG}
append_newline true
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name loggly
Format regex
Regex ^(?<time>\d{4}-\d{2}-\d{2}\S\d{2}:\d{2}:\d{2}(\.\d*)?\S)([^\|])*(?<logtime>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}(\.\d*)?)\|(?<levelname>[^|].+)\|(?<name>[^|].+)\|(?<filename>[^|].+)\|(?<lineno>[^|].+)\|(?<module>[^|].+)\|(?<funcName>[^|].+)\|(?<processName>[^|].+)\|(?<threadName>[^|].+)\|(?<messageinfo>[^|].+)\|(?<hostname>.*)\|(?<uuid>.*)\|(?<elapsed>.*)
Time_Format %Y-%m-%d %H:%M:%S.%L
Time_Keep On
Time_Key logtimeFor windows
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-windows-config
namespace: mts-infra
labels:
k8s-app: fluent-bit
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Log_Level error
Daemon off
net.dns.resolver LEGACY
Parsers_File parsers.conf
@INCLUDE application-log.conf
application-log.conf: |
[INPUT]
Name tail
Tag application.*
Exclude_Path C:\\var\\log\\containers\\fluent-bit*, C:\\var\\log\\containers\\cloudwatch-agent*
Path C:\\var\\log\\containers\\*.log
Parser docker
DB C:\\var\\fluent-bit\\state\\flb_container.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Rotate_Wait 30
[FILTER]
Name grep
Match *
Regex log loggly
[FILTER]
Name parser
Match **
Parser loggly
Key_Name log
Reserve_Data On
[FILTER]
Name record_modifier
Match *
Remove_key stream
[OUTPUT]
Name kinesis
Match application.*
region ${AWS_REGION}
stream ${MY_APP_LOG}
append_newline true
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %b %d %H:%M:%S
[PARSER]
Name loggly
Format regex
Regex ^(?<time>\d{4}-\d{2}-\d{2}\S\d{2}:\d{2}:\d{2}(\.\d*)?\S)([^\|])*(?<logtime>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}(\.\d*)?)\|(?<levelname>[^|].+)\|(?<name>[^|].+)\|(?<filename>[^|].+)\|(?<lineno>[^|].+)\|(?<module>[^|].+)\|(?<funcName>[^|].+)\|(?<processName>[^|].+)\|(?<threadName>[^|].+)\|(?<messageinfo>[^|].+)\|(?<hostname>.*)\|(?<uuid>.*)\|(?<elapsed>.*)
Time_Format %Y-%m-%d %H:%M:%S.%L
Time_Keep On
Time_Key logtimetime & logtime fields are important index patterns for OpenSearch.
The regex match result of logtime Time_Format needs to be matched with the [PARSER] loggly Time_Key of fluent-bit-config %Y-%m-%d %H:%M:%S.%L
Regex
You can test your regex here https://rubular.com/
Make sure your log can be matched your regex like below.
IAM
Make sure you have attached corresponding AWS resource policy. Otherwise you will see unauthorized error logs in fluent-bit pods.
AWS Kinesis Data Stream
Create a data stream ${MY_APP_LOG} there
AWS Data Firehose
Create Firehose stream and set data source & destination.
AWS OpenSearch
Security / Roles / all_access / Map user
Click Manage mapping to add the backend role in Security/Roles/all_access
If you do not set it right, you will see this Destination error logs in AWS Data Firehose
Stack Management / Index patterns
Create Index patterns for your app log
Discover
If everything set up right, you should see your app log index in Discover
Dev Tools
You can use GET /{your-index}/_search?q= to check your app-logs are stored in OpenSearch or not.
If everything is right, you can see the response like below
If your app-logs are stored in OpenSearch but the Regex set up wrong, the response will be like…
Please double check the Regex of fluent-bit-config configmap.
time & logtime fields are the important index patterns.
