[tor-project] TPA-RFC-74: GitLab CI retention policy

[Jérôme Charaoui <lavamind@xxxxxxxxxxxxxx>]

Summary: a proposal to limit the retention of GitLab CI data to 1 year

# Background

As more and more Tor projects moved to GitLab and embraced its 
continuous integration features, managing the ensuing storage 
requirements has been a challenge.

We regularly deal with near filesystem saturation incidents on the 
GitLab server, especially involving CI artifact storage, such as 
tpo/tpa/team#41402 and recently, tpo/tpa/team#41861

Previously, [TPA-RFC-14][] was implemented to reduce the default 
artifact retention period from 30 to 14 days. This, and CI optimization 
of individual projects has provided relief, but the long-term issue has 
not been definitively addressed since the retention period doesn't apply 
to some artifacts such as job logs, which are kept indefinitely by default.

[TPA-RFC-14]: 
https://gitlab.torproject.org/tpo/tpa/team/-/wikis/tpa-rfc-14-gitlab-artifacts

# Proposal

Implement a daily GitLab maintenance task to delete CI pipelines older 
than 1 year in *all* projects hosted on our instance. This will:

 * Purge old CI pipeline and job records for the GitLab database
 * Delete associated CI job artifacts, even those "kept" either:
   * When [manually prevented from expiring][] ("Keep" button on CI job 
pages)
   * When they're the [latest successful pipeline artifact][]
 * Delete old CI job log artifacts

[manually prevented from expiring]: 
https://gitlab.torproject.org/help/ci/jobs/job_artifacts#with-an-expiry
[latest successful pipeline artifact]: 
https://gitlab.torproject.org/help/ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs

## Goals

This is expected to significantly reduce the growth rate of CI-related 
storage usage, and of the GitLab service in general.

## Affected users

All users of GitLab CI will be impacted by this change.

But more specifically, some projects have "kept" artifacts, which were 
manually set not to expire. We'll ensure the concerned users and 
projects will be notified of this proposal. GitLab's documentation has 
the [instructions to extract this list of non-expiring 
artifacts](https://docs.gitlab.com/ee/administration/cicd/job_artifacts_troubleshooting.html#list-projects-and-builds-with-artifacts-with-a-specific-expiration-or-no-expiration).

## Timeline

Barring the need to further discussion, this will be implemented on 
Monday, December 9th.

## Costs estimates

### Hardware

This is expected to reduce future requirements in terms of storage hardware.

### Staff

This will reduce the amount of TPA labor needed to deal with filesystem
saturation incidents.

# Alternatives considered

A "CI housekeeping" script is already in place, which scrubs job logs 
daily in a hard-coded list of key projects such as c-tor packaging, 
which runs an elaborate CI pipeline on a daily basis, and triage-bot, 
which runs it CI pipeline on a schedule, every 15 minutes.

Although it has helped up until now, this approach is not able to deal 
with the increasing use of personal fork projects which are used for 
development.

It's possible to define a different retention policy based on a 
project's namespace. For example, projects under the `tpo` namespace 
could have a longer retention period, while others (personal projects) 
could have a shorter one. This isn't part of the proposal currently as 
it could violate the principle of least surprise.

# References

 * Discussion ticket: tpo/tpa/team#41874
 * [Make It Ephemeral: Software Should Decay and Lose 
Data](https://lucumr.pocoo.org/2024/10/30/make-it-ephemeral/)
_______________________________________________
tor-project mailing list -- tor-project@xxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to tor-project-leave@xxxxxxxxxxxxxxxxxxxx