What is Fluentd ?

Fluentd is a open source log shipping framework developed using c and ruby. it converts each log line into an event and they can be processed and enriched in a fluentd pipe line.

Advantages:

1) Setup is pretty straight forward for basic log shipping.
2) Better at complex routing , due to its declarative approach.
3) Supports multiple sources ( eg Apache logs , Syslogs etc ) and destinations ( MangoDB, Elasticsearch etc )

Disadvantages:

1) Decentralized plugin repository.
2) Has built in reliability , however requires tuning of more configuration parameters.

Fluentd Vs Beats:

Fluentd
1) Decentralized library ( https://www.fluentd.org/plugins)
2) Takes more memory compared to beats, fluentbit can be used to have more lightweight shippers (http://fluentbit.org/ )
3) Rely on other plugins for additional features.

Beats
1) Centralized library ( https://www.elastic.co/products/beats )
2) Compared to fluentd , beats are lightweight.
3) Each Beats has built in modules features, which covers different use cases, for example in the case of filebeats we have different submodules like system, auditd etc.

Deployment into K8 cluster

Setup:
You can refer to the following link for fluentd setup on K8.
https://docs.fluentd.org/v0.12/articles/kubernetes-fluentd

Git repo
https://github.com/fluent/fluentd-kubernetes-daemonset

Yaml files in the above repo contains the configuration of the DaemonSet and a ConfigMap. The configmap holds the fluentd configuration.

Configuration to be modified in the above Yaml files :

The following two parameters need to be configured for fluentd to ship the logs o elasticsearch.

FLUENT_ELASTICSEARCH_HOST ,

FLUENT_ELASTICSEARCH_PORT

Once the above two parameters are configured , yaml file is ready to be deployed into K8 cluster. Once fluentd is deployed , it sits as an agent and starts shipping the logs to elasticsearch server. A pattern can be created in kibana for the fluentd logs and the same can be visualized from kibana dashboard.

References:

https://www.fluentd.org/faqs
https://logz.io/blog/fluentd-logstash/

Following link Talks about “logging-stackdriver” which internally uses fluentd as an agent in K8 world.
https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/