Log Aggregation Configuration

Log Aggregation Configuration Properties describes global configuration properties related to deploying and configuring the Fluent Bit data collector to tail MATRIXX log files. For Fluent Bit default values, see the Fluent Bit documentation.

Table 1. Log Aggregation Configuration Properties
Property	Description
global.logging.fluentbit.enabled	If set to `true`, the Fluent Bit aggregation deployment is created. The default value is false.
global.logging.fluentbit.configuration.configmap	Contains any sections of Fluent Bit configuration where there are changes from default values. Note: If a property in a section is overridden, all other subsections of that section must be provided, even though they are unchanged.
global.logging.fluentbit.configuration.configmap.service	(Optional) Overrides Fluent Bit service configuration.
global.logging.fluentbit.configuration.configmap.inputs	(Optional) Overrides Fluent Bit input plug-in. The default is using the tail input.
global.logging.fluentbit.configuration.configmap.filters	(Optional) Overrides Fluent Bit filter configuration.
global.logging.fluentbit.configuration.configmap.filters.logs	(Optional) Has log-specific filters only that might not require modification for every deployment.
global.logging.fluentbit.configuration.configmap.filters.rewriteTag	(Optional) Updates the rewrite tag with the correct namespace of the MATRIXX deployment. A tag with a namespace is needed to fetch Kubernetes metadata from the cluster. Configure this for each deployment.
global.logging.fluentbit.configuration.configmap.filters.kubernetes	(Optional) Updates the API server endpoint for the cluster. Configure this for each deployment.
global.logging.fluentbit.configuration.sources[x].refName	The name of an entry in the `global.configurationSources` map. For more information about configuration sources, see the discussion about configuration sources for other components.
global.logging.fluentbit.serviceAccount.create	When set to `true`, the a ServiceAccount is created for use with the pod. The default value is `true`.
global.logging.fluentbit.serviceAccount.name	The name of ServiceAccount for the pod. When `global.logging.fluentbit.serviceAccount.create` is set to `true`, This is the name of the created ServiceAccount. The default value is `fluentbit-release_name`.
global.logging.fluentbit.serviceAccount.annotations	Any annotations to add to the ServiceAccount. For more information, see the discussion about annotations in Kubernetes documentation.

Fluent Bit is typically configured in a Kubernetes environment using a ConfigMap. This overrides the configuration file in the image with custom configuration. Fluent Bit Configuration describes the Fluent Bit configuration parameters.

Table 2. Fluent Bit Configuration
Section	Description
SERVICE	Defines a single instance of the Fluent Bit aggregator. Has references to other configuration files and configuration for remote connections to Fluent Bit.
INPUT	Configuration of a single input source. Includes details of source files and the name of the parser to use with the file.
FILTER	Configures any filter plug-ins to be used. The example following this table shows the filter used to enrich log messages with additional Kubernetes metadata.
OUTPUT	The destination configuration for aggregated logs. The example following this table uses Elasticsearch.
PARSER	Defines a parser that can be used for multiple inputs. The example following this table shows the configuration for a multi-line parser.
PLUGIN	Allows additional plug-ins to be used with Fluent Bit.

The required configurations create a Fluent Bit pod only, configured by default to send to port 9200 at an Elasticsearch address. Use the optional properties to override the default values for service, inputs, and filter configuration. For example the following Helm values file excerpt includes specification of port 2020 at address 0.0.0.0, followed by inputs and filters sections containing Fluent Bit configuration:

logging:
 output:
   persistentVolume: true
 fluentbit:
   enabled: false
   configuration:
     configmap:
       service: |
         [SERVICE]
           Flush         1
           Log_Level     error
           Daemon        off
           Parsers_File  parsers.conf
           Plugins_File  plugins.conf
           HTTP_Server   On
           HTTP_Listen   0.0.0.0
           HTTP_Port     2020
 
       inputs: |
         [INPUT]
           Name              tail
           Tag               kube.engine.*
           Path              /var/log/mtx/ckpt*/mtx_debug.log,/var/log/mtx/proc*/mtx_debug.log,/var/log/mtx/publ*/mtx_debug.log,/var/log/mtx/tra*/mtx_debug.log,/var/log/mtx/tralb-proc*/tra_process_control.log,/var/log/mtx/tralb-publ*/tra_process_control.log,/var/log/mtx/tralb-dr*/tra_process_control.log,/var/log/mtx/ne-*/mtx_debug.log
           Multiline         On
           Parser_Firstline  engine-multiline-mtx-debug-format
           DB                /var/log/mtx/flb_kube_engine.db
           Skip_Long_Lines   On
           Refresh_Interval  60
           Rotate_Wait       30
           Inotify_Watcher   false
 
         [INPUT]
           Name              tail
           Tag               kube.engine.*
           Path              /var/log/mtx/tralb-proc*/tra_util.log,/var/log/mtx/tralb-publ*/tra_util.log,/var/log/mtx/tralb-dr*/tra_util.log
           Multiline         On
           Parser_Firstline  engine-multiline-tra-format
           DB                /var/log/mtx/flb_kube_tra.db
           Skip_Long_Lines   On
           Refresh_Interval  60
           Rotate_Wait       30
           Inotify_Watcher   false
 
         [INPUT]
           Name              tail
           Tag               kube.engine.*
           Path              /var/log/mtx/ckpt*/*.log,/var/log/mtx/proc*/*.log,/var/log/mtx/publ*/*.log,/var/log/mtx/tra*/*.log
           Exclude_Path      /var/log/mtx/ckpt*/mtx_debug.log,/var/log/mtx/proc*/mtx_debug.log,/var/log/mtx/publ*/mtx_debug.log,/var/log/mtx/tra*/mtx_debug.log,/var/log/mtx/ne-*/mtx_debug.log,/var/log/mtx/tralb-proc*/tra_process_control.log,/var/log/mtx/tralb-publ*/tra_process_control.log,/var/log/mtx/tralb-dr*/tra_process_control.log,/var/log/mtx/tralb-proc*/tra_util.log,/var/log/mtx/tralb-publ*/tra_util.log,/var/log/mtx/tralb-dr*/tra_util.log
           Multiline         On
           Parser_Firstline  engine-multiline
           DB                /var/log/mtx/flb_kube_engine_other.db
           Skip_Long_Lines   On
           Refresh_Interval  60
           Rotate_Wait       30
           Inotify_Watcher   false
 
         [INPUT]
           Name              tail
           Tag               kube.engine.*
           Path              /var/log/mtx/mgr-engine-*/mtx-engine-mgr.log
           Multiline         On
           Parser_Firstline  engine-mgr
           DB                /var/log/mtx/flb_kube_engine-mgr.db
           Skip_Long_Lines   On
           Refresh_Interval  60
           Rotate_Wait       30
           Inotify_Watcher   false
 
         [INPUT]
           Name              tail
           Tag               kube.engine.*
           Path              /var/log/mtx/engine-operator/mtx-engine-opr.log
           Multiline         On
           Parser_Firstline  engine-opr
           DB                /var/log/mtx/flb_kube_engine-opr.db
           Skip_Long_Lines   On
           Refresh_Interval  60
           Rotate_Wait       30
           Inotify_Watcher   false
 
         [INPUT]
           Name              tail
           Tag               kube.bizapps.*
           Path              /var/log/mtx/mymatrixx-*/mymatrixx.log, /var/log/mtx/gateway-proxy-*/gateway_proxy.log, /var/log/mtx/rsgateway-*/rsgateway.log, /var/log/mtx/matrixxbct-*/matrixxbct.log, /var/log/mtx/notifier-*/mtx_notifier_camel.log, /var/log/mtx/activemq-*/activemq.log, /var/log/mtx/ldap-gateway-*/ldap-gateway-service.log
           Multiline         On
           Parser_Firstline  bizapps-multiline
           DB                /var/log/mtx/flb_kube_bizapps.db
           Skip_Long_Lines   On
           Refresh_Interval  60
           Rotate_Wait       30
           Inotify_Watcher   false
 
         [INPUT]
           Name              tail
           Tag               kube.bizapps.*
           Path              /var/log/mtx/pricing-loader-*/pricing-loader.log, /var/log/mtx/pricing-controller-*/pricing-controller.log
           Multiline         On
           Parser_Firstline  multiline-pricing
           DB                /var/log/mtx/flb_kube_pricing.db
           Skip_Long_Lines   On
           Refresh_Interval  60
           Rotate_Wait       30
           Inotify_Watcher   false
 
         [INPUT]
           Name              tail
           Tag               kube.chf.*
           Path              /var/log/mtx/sba-5gc-chf-*/nf.log
           Multiline         On
           Parser_Firstline  chf-multiline
           DB                /var/log/mtx/flb_kube_chf.db
           Skip_Long_Lines   On
           Refresh_Interval  60
           Rotate_Wait       30
           Inotify_Watcher   false
 
       filters:
         logs: |
           [FILTER]
             Name modify
             Match kube.engine.*
             Condition Key_value_matches type LM_TRACE
             Remove type
             Add type TRACE
 
           [FILTER]
             Name modify
             Match kube.engine.*
             Condition Key_value_matches type LM_INFO
             Remove type
             Add type INFO
 
           [FILTER]
             Name modify
             Match kube.engine.*
             Condition Key_value_matches type LM_DEBUG
             Remove type
             Add type DEBUG
 
           [FILTER]
             Name modify
             Match kube.engine.*
             Condition Key_value_matches type LM_WARN
             Remove type
             Add type WARN
 
           [FILTER]
             Name modify
             Match kube.engine.*
             Condition Key_value_matches type LM_CRITI
             Remove type
             Add type CRITICAL
 
           [FILTER]
             Name modify
             Match kube.engine.*
             Condition Key_value_matches type LM_ERROR
             Remove type
             Add type ERROR
 
         rewriteTag: |
           [FILTER]
             Name              rewrite_tag
             Match             kube.*
             # TODO: How to Render namespace in conf?? Tag needs to have namespace in order to query metadata when using kubernetes filter.
             Rule              $type ^(\S+)$ from.$TAG[0].$TAG[2].$TAG[3].$TAG[4].matrixx.$TAG[5].$TAG[6].log false
 
         kubernetes: |
           [FILTER]
             Name                kubernetes
             Match               from.*
             # Note: if default API url as suggested by kubernetes dose not work then
             # Replace the Kube_URL endpoint with specific API server endpoint for the cluster.
             Kube_URL            https://kubernetes.default.svc:443
             Kube_Tag_Prefix     from.kube.var.log.mtx.
             Regex_Parser        kubernetes-tag
             K8S-Logging.Exclude On
             Labels              On
             Annotations         Off
 
           [FILTER]
             Name nest
             Match from.kube.var.log.mtx.*
             Operation lift
             Nested_under kubernetes
 
           [FILTER]
             Name nest
             Match from.kube.var.log.mtx.*
             Operation lift
             Nested_under labels
 
           [FILTER]
             Name record_modifier
             Match from.kube.var.log.mtx.*
             Remove_key mtx-version
 
 
       outputs: |
         [OUTPUT]
           Name              es
           Match             *
           # TODO: Set the elasticsearch host value correctly
           Host              <elasticearch host Ip Or fully qualified service name>
           Port              9200
           Logstash_Format   On
           # uncomment bellow and provide valid elastic user id and password
           # needed only if elastic is deployed with basic security
           #HTTP_User elastic
           #HTTP_Passwd test
           Replace_Dots      On
           Include_Tag_Key   On
           Tag_Key           flb_tag
           Suppress_Type_Name On
           Trace_Output      Off
           Trace_Error       Off
 
       parsers: |
         [PARSER]
           Name        engine-multiline-mtx-debug-format
           Format      regex
           Regex       ^(?<type>LM_[^ ]*)\s+(?<pId>0*[0-9]+)[|](?<threadId>0*[0-9]+) (?<time>[^ ]*\s[^ ]*)\s+\[(?<source>[^\]]*)\](?<requestId>.*?)\|\s+(?<message>.*)$
           Time_Key    time
           Time_Format %Y-%m-%d %H:%M:%S.%L
 
         [PARSER]
           Name             engine-multiline-tra-format
           Format           regex
           Regex            ^(?<type>LM_[^ ]*)\s+(?<pId>0*[0-9]+)[|](?<threadId>0*[0-9]+) (?<time>[^ ]*\s[^ ]*)\s+\[(?<source>[^\]]*)\](?<requestId>.*?)\|\s+(?<message>.*)$
           Time_Key         time
           Time_Format      %Y-%m-%d %H:%M:%S,%L
 
         [PARSER]
           Name        engine-multiline
           Format      regex
           Regex       ^(?<type>LM_[^ ]*)\s+(?<time>[^ ]*\s[^ ]*)\s(?<pId>0*[0-9]+)[|](?<threadId>0*[0-9]+)\s+\[(?<source>[^\]]*)\](?<requestId>.*?)\|\s+(?<message>.*)$
           Time_Key    time
           Time_Format %Y-%m-%d %H:%M:%S.%L
 
         [PARSER]
           Name              bizapps-multiline
           Format            regex
           Regex             ^(?<time>\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}.\d{3})\s+\|\s+(?<type>[^ ]*)\s+\|\s+(?<requestId>[^ ]*)\s+\|\s+(?<threadId>[^ ]*)\s+\|\s+(?<source>[^ ]*)\s+\|\s+(?<message>.*)$
           Time_Key          time
           Time_Format       %Y-%m-%d %H:%M:%S.%L
          
         [PARSER]
           Name              chf-multiline
           Format            regex
           Regex             ^(?<time>\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}.\d{3})\s+\|\s+(?<type>[^| ]*)\s+\|\s+(?<requestId>[^|.]*)\s+\|\s+(?<threadId>[^| ]*)\s+\|\s+(?<source>[^| ]*)\s+\|((.*\s+\|\s+.*)?\s+(?<subscriberId>[^| ]*)\s+\||)\s+(?<message>.*)$
           Time_Key          time
           Time_Format       %Y-%m-%d %H:%M:%S.%L
 
         [PARSER]
           Name              engine-opr
           Format            regex
           Regex             ^(?<time>\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2})[\t](?<type>[^ ]*)[\t](?<source>[^ ]*)[\t](?<message>.*)$
           Time_Key          time
           Time_Format       %Y-%m-%d %H:%M:%S
 
         [PARSER]
           Name              engine-mgr
           Format            regex
           Regex             ^(?<time>\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2})\s+\|\s+(?<type>[^ ]*)\s+\| (?<source>[^ ]*)\s+\|\s+(?<message>.*)$
           Time_Key          time
           Time_Format       %Y-%m-%d %H:%M:%S
 
         [PARSER]
           Name              multiline-pricing
           Format            regex
           Regex             ^(?<time>\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2},\d{3})(?:[.]\d{6})\s+\|\s+(?<type>[^ ]*)\s+\|\s+(?<pid>[^ ]*)\s+\|\s+(?<threadId>[^ ]*)\s+\|\s+(?<source>[^ ]*)\s+\|\s+\|\s+(?<message>.*)$
           Time_Key          time
           Time_Format       %Y-%m-%d %H:%M:%S,%L
 
         [PARSER]
           Name              kubernetes-tag
           Format            regex
           Regex             ^(?<namespace_name>[^.]*)\.(?<pod_name>[^.]*)\.(?<component_name>[^.]*).log$
 
       plugins: |
         [PLUGINS]
           # Path /path/to/out_gstdout.so

Cloud native MATRIXX deployments are configured to output all logging to files on a PV by default. This is useful in a development or test environment where a log aggregator or other logging infrastructure is not available. Output to STDOUT might be preferable.

You can enable Fluent Bit log aggregation of logs that are stored in the shared PV. This feature can be enabled in your Helm values file by setting global.logging.fluentbit.enabled to true. Enabling this option creates a Fluent Bit pod in the installation namespace that connects to the logging PV, allowing all application log files to be aggregated.

Fluent Bit Configuration for Log Aggregation for External Diameter Gateway

The External Diameter Gateway supports logging to STDOUT by default. To aggregate the External Diameter Gateway logs, you can use the open source Fluent Bit. Example installation and removal scripts and an example Helm values file are provided in mtx_kubernetes_deploy-version-build.tar

Alerting

Fluent Bit supports multiple plug-ins for sending alerts in the event of a particular issue being logged by the application. Filters can be defined to intercept events that are tagged with specific values. The intercepted events can be used to trigger sending counts to Prometheus, calling a REST API, or sending an email.

Obtaining a File From Shared Storage

Copy log files from a PV with the kubectl cp command. For example, copy all log files in a particular pod to the local /tmp/mtx directory with the following command:

kubectl cp namespace/pod-name:/var/log/mtx/<pod-name>/*.log /tmp/mtx/

All pods write their logs to the same shared storage, so it is possible to copy any pod log file from the same pod. A Fluent Bit pod is provided to allow configurable log aggregation of files in the shard logging storage, so it would be possible to copy files using that pod. For example:

kubectl cp matrixx/matrixx-fluentbit-ag1-64d775bf8-xwp8s:/var/log/mtx/proc-s1e1-0/ /tmp/mtx/proc-s1e1-0/