What is Fluentd ?

Fluentd is a open source log shipping framework developed using c and ruby. it converts each log line into an event and they can be processed and enriched in a fluentd pipe line.

Advantages:

1) Setup is pretty straight forward for basic log shipping.
2) Better at complex routing , due to its declarative approach.
3) Supports multiple sources ( eg Apache logs , Syslogs etc ) and destinations ( MangoDB, Elasticsearch etc )

Disadvantages:

1) Decentralized plugin repository.
2) Has built in reliability , however requires tuning of more configuration parameters.

Fluentd Vs Beats:

Fluentd
1) Decentralized library ( https://www.fluentd.org/plugins)
2) Takes more memory compared to beats, fluentbit can be used to have more lightweight shippers (http://fluentbit.org/ )
3) Rely on other plugins for additional features.

Beats
1) Centralized library ( https://www.elastic.co/products/beats )
2) Compared to fluentd , beats are lightweight.
3) Each Beats has built in modules features, which covers different use cases, for example in the case of filebeats we have different submodules like system, auditd etc.

Deployment into K8 cluster

Setup:
You can refer to the following link for fluentd setup on K8.
https://docs.fluentd.org/v0.12/articles/kubernetes-fluentd

Git repo
https://github.com/fluent/fluentd-kubernetes-daemonset

Yaml files in the above repo contains the configuration of the DaemonSet and a ConfigMap. The configmap holds the fluentd configuration.

Configuration to be modified in the above Yaml files :

The following two parameters need to be configured for fluentd to ship the logs o elasticsearch.

FLUENT_ELASTICSEARCH_HOST ,

FLUENT_ELASTICSEARCH_PORT

Once the above two parameters are configured , yaml file is ready to be deployed into K8 cluster. Once fluentd is deployed , it sits as an agent and starts shipping the logs to elasticsearch server. A pattern can be created in kibana for the fluentd logs and the same can be visualized from kibana dashboard.

References:

https://www.fluentd.org/faqs
https://logz.io/blog/fluentd-logstash/

Following link Talks about “logging-stackdriver” which internally uses fluentd as an agent in K8 world.
https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/

What is Teamcity personal build ?

Its an ability to test your changes on a build server before actually committing anything. This concept is also refereed to as remote build.

Why do we need Personal/remote builds ?

Personal builds solves two major problems in a traditional software development.

1) Generally before committing the code , as a developer we do complete testing on our local machine to make sure that we are not introducing any problems into the build. However this is a lengthy process as we need wait for a significant time.

2) If we don’t want to wait for significant time , committing everything into build server and testing there is other approach , But this is at high risk as the developer can break the whole repo.

We can address the above two problems,if we had a flexibility of testing on build server and at the same time, no code is committed into Repo until the build is passed. Personal build is one such concept which will help us to solve this problem. It helps you to test your changes on your build server without committing anything. Once personal build is fired, developer can start working again in parallel, this can increase the productivity of the developer. if build is successful developer can push the code or else he can fire the personal build build again with the new set of changes.

As a Developer , How can i run my build using Personal build ?

1) To make use of personal builds , you are supposed to install Teamcityplugin.
( https://confluence.jetbrains.com/display/TCD9/IntelliJ+Platform+Plugin ).
2) Once plugin is installed , you need to connect to Teamcity ( Teamcity–>Login ), using team city registered user.
3) Once connected , you can trigger remote Run ( Screen-1-Remote-Run ).
4) As a next step you will see different projects and builds available with Teamcity, select the required build configuration.
5) Along with that, in the same screen, click on customize icon present on the top, this options helps to set environment variables.
6) Then click on ok, with all the local changes a patch is generated and pushed to team city ,build ( docker image ) then gets triggered at Teamcity along with all the latest changes on Git.
7) Plugin provides clear build status at the bottom of the IDE. (Screen-4-IDE_Console)
8) If build is succeeded, you can see success message in IDE console.
9) If build is failed, you can see the message in IDE console. Along with this,Teamcity will send a email notification to the team city logged in user used at step-2

DockerFile present in a particular project is considered for this use case .As a part of this docker file , proxy settings are present in two places, one is with env and the other is with run command and both of them has to be configurable and should not be hard coded in the docker file,so that docker container can be used in any environment.

Basically to fix this we can make use of –build-arg and ARG options provided by Docker as follows

Following is one way of sending the proxy settings at runtime.

docker build -t images.dockhub-atp.abc.com/teamcity:v0.2 . –build-arg http_proxy=http://www-proxy-xyz.us.abc.com:80 –build-arg https_proxy=http://www-proxy-abc.us.xyz.com:80 –build-arg no_proxy=abccorp.com,localhost,127.0.0.1

with the above step, Following ENV can be removed from the DockerFile.

ENV http_proxy http://www-proxy-atp.us.xyz.com:80
ENV https_proxy http://www-proxy-ran.us.xyz.com:80
ENV no_proxy .abc.com,.abccorp.com,localhost,127.0.0.1

Secondly, With respect to the following hard coded proxy settings we can make use of ARG.

RUN echo ‘export http_proxy=http://www-proxy-abc.us.abc.com:80’ >> /etc/default/docker \
&& echo ‘export https_proxy=http://www-proxy-abc.us.abc.com:80’ >> /etc/default/docker \
&& echo ‘export no_proxy=.abc.com,.xyzcorp.com,localhost,127.0.0.1’ >> /etc/default/docker \
&& echo ‘export NO_PROXY=.abc.com,.zyaccorp.com,localhost,127.0.0.1’ >> /etc/default/docker \
&& echo ‘export HTTPS_PROXY=http://www-proxy-hqdc.us.abc.com:80’ >> /etc/default/docker \
&& echo ‘export HTTP_PROXY=http://www-proxy-hqdc.us.abc.com:80’ >> /etc/default/docker \

Replaced above with the following.

ARG http_proxy
ARG no_proxy

#Add proxy conf for docker within teamcity agent
RUN echo ‘export http_proxy=$http_proxy’ >> /etc/default/docker \
&& echo ‘export https_proxy=$http_proxy’ >> /etc/default/docker \
&& echo ‘export no_proxy=$no_proxy’ >> /etc/default/docker \
&& echo ‘export NO_PROXY=$no_proxy’ >> /etc/default/docker \
&& echo ‘export HTTPS_PROXY=$http_proxy’ >> /etc/default/docker \
&& echo ‘export HTTP_PROXY=$http_proxy’ >> /etc/default/docker \

http_proxy and no_proxy are passed at runtime while building the docker image.

To measure the time taken by any task in Gradle , we can use the option “profile” as follows

./gradlew –profile

The above command generates a detailed HTML report with the following Tabs.

Summary: Gives the overall time of the task executed.
Configuration: Time taken by each project present in the given task.
Dependency resolution: Shows the times for resolving all the dependencies
Task Execution: shows time taken by each task, this can help you to trace which task is taking more time and act accordingly.

Therefore as part of optimizing build task , –profile helps in generating the detailed report of which task is taking which time.

Following tasks are identified as taking more time for build task

distTar, distZip, bootDistTar, bootStartScripts, bootDistZip

Next step is how to ignore these tasks if not required ?
Gradle provide the option ./gradlew -x

Usage example:

./gradlew build –info -x distTar -x distZip -x bootDistTar -x bootDistZip

In addition we can also use the option “–build-cache” to improve the performance of any build target.
build cache is a cache mechanism that aims to save time by reusing outputs produced by other builds.

More inforation about build cache:
https://docs.gradle.org/current/userguide/build_cache.html

Therefore Going forward, we can use the same approach for optimizing/analysing any Gradle task.

Go to .git folder present in your repository. then edit config file.

.git/config , try to have the user name before @symbol as follows

url = https://ravisha@github.com/ravisha/android_server.git

 

PUTTY:

This utility is used to connect to remote linux or unix machines, it provides CLI ( Command Line Interface ) to the remote machine. Once putty is opened , type the Hostname and port( port not mandatory, default port is taken )  in the respective text fields and click open , then it will ask for credentials to connect. Putty can be used to do SSH, Telnet and other operations.

Downloadable Link: https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

Installable : putty.exe ( Release 0.70 )

VNC ( Virtual Network Computing )

This utility is also used to connect to the remote linux or unix machines, It provides GUI ( Graphic User interface ) to the remote machine. Open VNC  and then go to File→ New Connection and then provide the remote HostName to connect, Once you click on host name icon, it will ask for the credentials to connect.

Downloadable Link: https://www.realvnc.com/en/connect/download/vnc/

Installable : VNC-Viewer-6.17.731-Windows.exe

MSTSC ( Microsoft Terminal Services )

This Utility is used for connecting to the remote windows machines. This utility is provided by default in windows machine, you can go to run and type mstsc, then a dialog box will be popped up , which will ask for the remote host name along with credentials, once you enter the required details , remote windows desktop will be opened in a new window. This tool provides GUI.

How to Check Memory ?

1) watch -n 3 cat /proc/meminfo
2) watch -n 3 free -m

How to free Memory ?

1. Freeing Up the Page Cache
echo 1 > /proc/sys/vm/drop_caches
2. Freeing Up the Dentries and Inodes
echo 2 > /proc/sys/vm/drop_caches
3. Freeing Up the Page Cache, Dentries and Inodes
echo 3 > /proc/sys/vm/drop_caches
4. Flushing the File System Buffers

1) Run the following command (where validity is the number of days before the certificate will expire):
keytool -genkey -keyalg RSA -alias selfsigned -keystore keystore.jks -storepass password -validity 361 -keysize 2048
Fill in the prompts for your organization information. When it asks for your first and last name, enter the host/domain name  of the server that users will be entering to connect to your application adc01djc.us.oracle.com, this will be filled in the certificate with an attribute called cname, any mismatch here can result in following type of exceptions when making HTTP client calls.

Example Exception.

Exception in thread “main” javax.net.ssl.SSLException: hostname in certificate didn’t match: <abc01> != <vbauth>

 

This will create a keystore.jks file containing a private key and your  fresh self signed certificate.

2) if you want certificate out of the key store use the following command
keytool -export -alias selfsigned -keystore keystore.jks -rfc -file noejb.cer
3) To see the contents of the certificate
keytool -printcert -file noejb.cer

While debugging through the code without IDE , if you want to get the call hierarchy of any particular method to see the flow of the calling methods, adding the following piece of code in the method you are interested in can help you.

StringWriter sw = new StringWriter();
new Throwable(“”).printStackTrace(new PrintWriter(sw));
String stackTrace = sw.toString();
System.out.println(stackTrace);

Passing -verbose:class to java command, will print each class loaded and from which jar it is loaded from.

example: java -verbose:class Sample