Centralized Logging

Running kubectl logs in your terminal is fine for simple logging checks, but a more sophisticated method is needed to achieve better observability. That is, centralized logging.

Centralized Logging using Fluent Bit and CloudWatch Logs

We will build a centralized logging system according to the reference article.

Create a Namespace for Fluent Bit

Create a yaml file:

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: amazon-cloudwatch
  labels:
    name: amazon-cloudwatch

Then create the namespace:

kubectl apply -f namespace.yaml

Configure an IAM policy

Create an IAM policy with the following permissions:

fluent-bit-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "logs:CreateLogStream",
            "logs:CreateLogGroup",
            "logs:PutLogEvents"
        ],
        "Resource": "*"
    }
    ]
}

You can do this by running the command:

aws iam create-policy --policy-name fluent-bit-policy --policy-document file://fluent-bit-policy.json

If your EKS cluster does not have an IAM OIDC provider, then create it.

eksctl utils associate-iam-oidc-provider  --region us-east-1 --cluster eolh-tutorial --approve 

Then create IAM service account:

# Replace <FLUENT_BIT_POLICY_ARN> with the ARN of fluent-bit-policy
eksctl create iamserviceaccount --cluster eolh-tutorial \
--region us-east-1 \
--attach-policy-arn <FLUENT_BIT_POLICY_ARN> \
--name fluent-bit-windows \
--namespace amazon-cloudwatch \
--approve

Create a Config Maps

ClusterName=eolh-tutorial
RegionName=us-east-1
FluentBitReadFromHead='Off'

kubectl create configmap fluent-bit-cluster-info \
--from-literal=cluster.name=${ClusterName} \
--from-literal=logs.region=${RegionName} \
--from-literal=read.head=${FluentBitReadFromHead} -n amazon-cloudwatch 

Deploy Fluent Bit

Now you can deploy Fluent Bit on your cluster.

Create a manifest file:

fluent-bit-daemon-set.yaml

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-windows-role
  namespace: amazon-cloudwatch
rules:
  - nonResourceURLs:
      - /metrics
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
      - pods/logs
      - nodes
      - nodes/proxy
    verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluent-bit-windows-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-windows-role
subjects:
# Assuming that the Service Account was created earlier with name fluent-bit-windows.
- kind: ServiceAccount
  name: fluent-bit-windows
  namespace: amazon-cloudwatch
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-windows-config
  namespace: amazon-cloudwatch
  labels:
    k8s-app: fluent-bit-windows
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush                       5
        Log_Level                   info
        Daemon                      off
        net.dns.resolver            LEGACY
        Parsers_File                parsers.conf
        
    @INCLUDE application-log.conf
 
  application-log.conf: |
    [INPUT]
        Name                tail
        Tag                 application.*
        Exclude_Path        C:\\var\\log\\containers\\fluent-bit*
        Path                C:\\var\\log\\containers\\*.log
        Docker_Mode         On
        Docker_Mode_Flush   5
        Docker_Mode_Parser  container_firstline
        multiline.parser    docker, cri
        DB                  C:\\var\\fluent-bit\\state\\flb_container.db
        Read_from_Head      ${READ_FROM_HEAD}
 
    [INPUT]
        Name                tail
        Tag                 application.*
        Path                C:\\var\\log\\containers\\fluent-bit*
        multiline.parser    docker, cri
        DB                  C:\\var\\fluent-bit\\state\\flb_log.db
        Read_from_Head      ${READ_FROM_HEAD}
 
    [FILTER]
        Name                kubernetes
        Match               application.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Kube_Tag_Prefix     application.C.var.log.container.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Labels              Off
        Annotations         Off
        Use_Kubelet         Off
        Buffer_Size         0
 
    [OUTPUT]
        Name                cloudwatch_logs
        Match               application.*_default_*
        region              ${AWS_REGION}
        log_group_name      /aws/containerinsights/${CLUSTER_NAME}/default
        log_stream_prefix   ${HOST_NAME}-
        auto_create_group   true
        extra_user_agent    container-insights
 
    [OUTPUT]
        Name                cloudwatch_logs
        Match               application.*_amazon-cloudwatch_*
        region              ${AWS_REGION}
        log_group_name      /aws/containerinsights/${CLUSTER_NAME}/amazon-cloudwatch
        log_stream_prefix   ${HOST_NAME}-
        auto_create_group   true
        extra_user_agent    container-insights
 
  parsers.conf: |
    [PARSER]
        Name                docker
        Format              json
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ
 
    [PARSER]
        Name                container_firstline
        Format              regex
        Regex               (?<log>(?<="log":")\S(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ
 
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit-windows
  namespace: amazon-cloudwatch
  labels:
    k8s-app: fluent-bit-windows
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-windows
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-windows
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: fluent-bit-windows
        image: public.ecr.aws/aws-observability/aws-for-fluent-bit:windowsservercore-latest
        imagePullPolicy: Always
        env:
          - name: AWS_REGION
            valueFrom:
              configMapKeyRef:
                name: fluent-bit-cluster-info
                key: logs.region
          - name: CLUSTER_NAME
            valueFrom:
              configMapKeyRef:
                name: fluent-bit-cluster-info
                key: cluster.name
          - name: HOST_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: READ_FROM_HEAD
            valueFrom:
              configMapKeyRef:
                name: fluent-bit-cluster-info
                key: read.head
        resources:
          limits:
            memory: 600Mi
          requests:
            cpu: 500m
            memory: 600Mi
        volumeMounts:
        # Only read only access to the following mounts is required
        - name: fluentbitstate
          mountPath: C:\var\fluent-bit\state
        - name: varlog
          mountPath: C:\var\log
          readOnly: true
        - name: varlibdockercontainers
          mountPath: C:\ProgramData\docker\containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: C:\fluent-bit\etc\
          readOnly: true          
      terminationGracePeriodSeconds: 30
      volumes:
      - name: fluentbitstate
        hostPath:
          path: C:\var
      - name: varlog
        hostPath:
          path: C:\var\log
      - name: varlibdockercontainers
        hostPath:
          path: C:\ProgramData\docker\containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-windows-config
      nodeSelector:
        kubernetes.io/os: windows          
      serviceAccountName: fluent-bit-windows

Then deploy it:

kubectl apply -f fluent-bit-daemon-set.yaml

Open AWS Management Console and go to the CloudWatch Logs /aws/containerinsights/eolh-tutorial/default group.

Other Solutions

Grafana Loki is often used in the context of observability.

However, to our knowledge, you cannot use Loki with Windows containers/nodes. This is because although the Loki agent needs to work on each Node, it does not work on Windows nodes.

Another possible solution would be to send logs from Fluent Bit to ElasticSearch and visualize them at Kibana or Grafana.

Reference

https://aws.amazon.com/jp/blogs/containers/centralized-logging-for-windows-containers-on-amazon-eks-using-fluent-bit/