Portrait of Patrice GADEGBEPatrice GADEGBE

Monitoring

Two self-hosted services accompany my projects: GlitchTip, which collects the errors of the applications, and SonarQube, which analyses the code and the coverage of four Sanu repositories on every run on their main branch.

Role
Sole author of the two installation repositories
Period
September 2026
Status
In service; SonarQube connected to four repositories, GlitchTip to one site
Technologies
GlitchTip 6, SonarQube Community, PostgreSQL 17, Docker Compose, nginx, GitLab CI
Repositories
Two
Monitoring and code qualityThe GitLab pipelines send their analyses to SonarQube and the applications send their errors to GlitchTip, through the host's nginx, which terminates the encrypted connection. Each service has its own PostgreSQL database.GitLab pipelinesfour Sanu repositoriesApplicationsSentry protocolnginxencrypted connectionSonarQubeanalysis and coverageGlitchTiperrors, uptimeHTTPSPostgreSQL 17one database per service
Figure 1. The two services share the server of the staging environments, which explains most of the choices described here.

Context

The errors of a running application and the quality of its code are usually tracked through online services. GlitchTip and SonarQube Community offer the equivalent to host oneself: the first speaks the Sentry protocol, so that any library that accepts a Sentry address sends it its reports without modification, and the second receives the analyses launched by the pipelines.

Problem

These two services are installed on a server that already carries the staging environments and two pipeline runners, with no swap memory. SonarQube, which runs three Java virtual machines and a search engine, can on its own exhaust the memory and cause a neighbouring service to be stopped. The installation also handles tokens and an administration password, which a carelessly written script leaves in the process list or in a log.

Answer

A memory budget and a designated victim

SonarQube is capped at three gigabytes and two processors. Its out-of-memory score is also raised, so that the kernel, should memory run out all the same, stops SonarQube rather than a staging environment.

# The server also carries the staging stacks and two docker runners allowed 3 GB per
# job, with no swap. The cap keeps SonarQube (three JVMs at 512 MB heap each, plus
# Elasticsearch off-heap) inside a known budget, and the OOM score makes the kernel
# pick SonarQube rather than a staging container if memory still runs out.
mem_limit: 3g
cpus: 2
oom_score_adj: 500
# Lets Elasticsearch close its indices cleanly on stop.
stop_grace_period: 2m
Listing 1. Limits imposed on the SonarQube container, with their reason.

Secrets that go through neither the command line nor a URL

The installation scripts pass the credentials to curl through a configuration file read from standard input: they appear neither among the arguments, which the process list would display, nor in a URL, which the access log would record. The administration password is generated on the server and is never displayed there, and each analysed repository receives its own token, which the script replaces on every run.

A re-runnable deployment

The deployment of SonarQube can be run again without precaution, including after a version change: the health check treats a pending migration as a healthy state, which lets the script trigger it itself. The Dart analysis plugin is pinned by its version and by its checksum, then copied into the service’s volume and not into the image, a named volume being filled from the image only when it is created.

A measured workaround

GlitchTip’s uptime checks were being refused by a site placed behind a content delivery network. Seven compared requests showed that the response depended on the presence of a set of headers and not on their content. The fix takes the original settings and removes only one key from them, for the one process concerned, so that a version upgrade keeps everything else; the file itself states under what condition it can be deleted.

Limits and outlook

GlitchTip today has only one site as a client, the Sanu interfaces having planned their connection without having made it. No backup is scheduled for either service: it is done by hand before a version upgrade, and stays on the same server.

SonarQube Community analyses only one branch per project, which rules out the analysis of merge requests, and the Dart analysis depends on a community continuation of a plugin that its author no longer maintains. Finally, since September 2026, a quality gate that is not met blocks the staging deployment of the four repositories: the service that the kernel must sacrifice first now stands on the path of those deliveries, a contradiction that will have to be resolved by giving it its own machine.

Back to the other projects