Prometheus

Prometheus is a monitoring solution that gathers time-series based numerical data.

Why Prometheus for Teamcity ?

Teamcity provides provides a variety of diagnostic tools and indicators to monitor and troubleshoot the server, which are accessible from the Administration | Diagnostics page.Along with this, we would like to have Prometheus as one stop for health lookup for the completed stack of components present in the Dev cluster.Therefore as a part of this POC, we would like to investigate the methods of getting teamcity metrics into Prometheus.

How can Teamcity or any Third Application send metrics to Prometheus?

Prometheus refers this as as instrumentation of an application . Any application that want to expose metrics need to expose a HTTP/s endpoint with URL as /metrics. Prometheus will use this endpoint to scrape metrics at regular intervals. Application that exposes the metrics is referred to as exporter.

Readily available Prometheus Exporters: https://prometheus.io/docs/instrumenting/exporters/

Prometheus Metrics Types: https://prometheus.io/docs/concepts/metric_types/

Teamcity and Prometheus :

Teamcity doesn’t provide any exporter for Prometheus, Therefore to pull metrics from Teamcity, we need to write a custom exporter. Exporters can be developed using different languages like GO, Python,Ruby or Java.

Repos Explored for Teamcity custom exporter.

https://github.com/allistera/teamcity-exporter

https://github.com/leominov/teamcity-exporter

https://github.com/Guidewire/teamcity_exporter

https://github.com/m4h/prometheus.

POC: ( Used Python )

As a part of this POC , following metrics are exposed as an end point for Prometheus to scrape.

Teamcity Metrics Exposed:

1) Checking Build Queue over the time. ( Rest API used – /app/rest/buildQueue )

2) Number of Agents over the time. ( Rest API used – /rest/agents ).

3) Projects aggregated status: Go through all the projects in teamcity,For each project get status of each build and print aggregated build status for each project ( SUCCESS=0, RUNNING=1, ERROR=2, FAILURE=3, UNKNOWN=4′ )

( Rest API used – app/rest/buildTypes )

Prometheus URL:

http://:9090/graph

Use the following metrics and click on the button execute in the above link

1)teamcity_project_aggregated_status

2)teamcity_agents_num

3)teamcity_build_queue_len

( Attached screen shots for reference )

Teamcity Rest API https://confluence.jetbrains.com/display/TCD18/REST+API can be explored more to see what metrics can be useful.