Introduction

Welcome to InfraDocs. This is the manual for Void systems management. The manual is split up into infrastructure managed by different systems. Knowledge of Void or base Linux administration is assumed.

Terraform

Not all infrastructure owned by the Void project is hosted on our infrastructure or integrated into our systems. For some infrastructure we need to mirror data out to 3rd party systems. This is done with HashiCorp Terraform.

Files for terraform end in .tf and live in the terraform subdirectory of the infrastructure repo. There is currently no automation that pushes terraform state to remote systems.

Important!

It is VERY IMPORTANT that only one Terraform push is in progress at a time. It is additionally VERY IMPORTANT that anyone pushing terraform has up to date terraform state, and that this state is synced back to other people who have it. The state should be treated as secure since terraform has a bad habbit of putting things in there it shouldn't.

We are currently evaluating Atlantis to push terraform state from a central location.

Setting Up

Terraform is configured to use remote state. One-time configuration is required to access this state:

Ensure that your netauth user is a member of netauth/terraform, without access to this group you will not be able to access the terraform state.

Change the terraform directory and run the following command:

$ terraform init --backend-config "username=<username>" --backend-config "password=<password>"

NOTE: This will cache your password to local disk. This is a known defect that will be resolved in Terraform 2. Until then, either remove your password from the .terraform/ cache files, or more preferably, store the entire void-infrastructure repo on an EncFS, as there is other sensitive content that winds up in gitignored subdirectories of this repository.

Obtaining Control Authority

Having access to state isn't sufficient. You will also need a Personal Access Token from GitHub, and access to the cloud service account. Obtain a token for your account, and contact another member of the infrastructure team to obtain the service principal. Place the service principal in terraform/account.json, and the token in the environment variable GITHUB_TOKEN.

GitHub

GitHub only provides an interface to sync data from LDAP, and even then only if using the enterprise version. Since Void is an open source project and isn't using this option, we don't sync data. The organization at github.com/void-linux has very little state, primarily users and groups.

Groups

There are currently two groups that gate access into GitHub resources:

pkg-committers

Members of this group have broad commit access and can generally push to any Void owned repo. The primary reason for people to gain access to this group is to be able to push package templates. Access to this group should be assumed to contain the ability to trigger builds that will eventually be signed for inclusion in the main repo.

void-ops

Membership into this group is highly restricted and should generally not be authorized without a signoff from an infrastructure lead or maldridge@. This group gates access to the infrastructure repo itself, and is restricted to prevent accidental breakage from pushing something that is later pushed by automation that performs change detection against the state of the repo.

Adding and Removing Members

Adding and removing members takes place in github_members.tf. This file contains a stanza for every user and every group they are in. To change membership of a group add or remove a stanza, then apply the state transformation to GitHub.

This file is manually formatted, take care to maintain lexical sort ordering and indentation. For example if a new committer with username voidfu was to be added, a new stanza as follows would be added to the file:

resource "github_team_membership" "pkg-committers_voidfu" {
  team_id = "${github_team.pkg-committers.id}"
  role = "maintainer"
  username = "voidfu"
}

The name placed in the resource line should always be lower case. The name that appears in the username should be an exact match for the username shown on the user's profile page.

Pushing state changes

Pushing a state change can only be done by organization owners. To request a push of terraform state, request action from one of:

* the-maldridge
* gottox
* duncaen

It is very important that only one push be in progress at a time. To this end, anyone making a push should endeavor to determine no other changes are in motion, manual or terraformed.

Authenticating to GitHub for Push

Github needs authentication to authorize the push. This takes the format of a personal access token. The token must contain sufficient permissions to add and remove people from the organization, add and remove repositories, and add and remove groups. The token should be stored in the environment variable GITHUB_TOKEN.

Pushing the Changes

Pushing the changes is done in two phases. The first phase is a planning phase. In this phase call terraform as shown:

$ terraform plan

Verify that the output is sane, it will provide a diff of any action that terraform wants to take. This should be very simple to understand what is going to happen because you shouldn't push large changes, instead prefer to push incremental changes in succession.

When you are satisfied with the planned actions, apply them:

$ terraform apply

You'll be asked to confirm the application of state. If you're satisfied, apply the state. Terraform is not like Ansible, be careful that you don't remove people from the organization or clear permissions that you can't put back without assistance.

Google Cloud Platform

We use some resources on Google Cloud Platform. Most prominently the primary DNS zone for voidlinux.org is hosted on GCP.

To minimize the number of people with access to the cloud account, we use a service account. The service account should be given only the minimum of permissions and its key should be stored as 'account.json' in the terraform directory (which is gitignored).

GCP DNS

Void's primary DNS servers are provided through Google Cloud Platform. To add a new record, make a copy of an existing record in the right section in google-dns.tf and send a PR. The state will be applied after the PR is accepted.

Ansible

Ansible is the primary configuration system for Void's hardware. Ansible is a standard technology currently owned by RedHat, and complete information can be found at the ansible website.

Operation of Ansible is beyond the scope of this manual, but a short introduction is nonethelss provided.

Installation

To apply the Ansible playbooks you need to install Ansible. To do this make sure you have python and virtualenv available. Void's configuration expects python from the 3.x branch on the control host. Any version of python is acceptable on the target machines, but for consistency reasons the system python is 2.7 series.

The first step is to install and check Ansible. Within the ansible/ directory run the following commands. The virtual environment must live in a subdirectory called 'venv' as this path is referenced by ansible.cfg.

$ virtualenv venv
$ source venv/bin/activate
$ pip install -r requirements.txt

Once installed, verify that you have ansible available within path:

$ ansible --version

The version should match exactly:

$ ansible --version
ansible 2.5.5

Secrets

Almost all of Void's configuration data is public. Things that are not public are referred to as "secrets". This includes information such as the buildbot login file, signing keys, various tokens that authenticate services and so on. This data lives in ansible/secret and you must obtain a copy of this directory before trying to push to any machine.

The format of files in the secret directory should be plain text for files that are string secrets, or the native file format of the secret in question. Secret names should be of the form ROLENAME_SECRET so a token signing key for the netauth service should be named netauth_token.key in the secret directory and its file format should be PEM encoded key data.

Storing Secrets

Secrets should be encrypted when at rest. It is advised to store secrets in an EncFS directory which is mounted into the appropriate location as needed. Any encrypted system is acceptable here assuming that it provides a normal filesystem view and supports strong cryptography.

Obtaining Secrets

Secrets should be held by as few people as possible, but no fewer than 2 at any given time. Secrets should only be exchanged after positive identity confirmation has been confirmed. Transfer should then be done via secure means, such as a copy made to a Void host and then made visible via Unix file permissions.

Deploying a Playbook

Deploying a playbook to the Ansible managed infrastructure is done with the ansible-playbook command. An example invocation to update the buildmaster is shown below:

$ ansible-playbook -DK build.yml --limit vm1.a-lej-de.m.voidlinux.org

Breaking down the above command line:

  • -D: Provide a diff of the changes that are made.
  • -K: Prompt for the sudo password
  • build.yml: The playbook that we want to run
  • --limit: Restrict running of this playbook to the following host(s)
  • vm1.a-lej-de.m.voidlinux.org: The hostname of the specific server that runs the buildmaster.

Here's what a full run of this command looks like:

$ ansible-playbook -DK build.yml --limit vm1.a-lej-de.m.voidlinux.org
SUDO password: 

PLAY [buildmaster] ***************************************************************************

TASK [Gathering Facts] ***********************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Configure hosts] *************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Configure hostname] **********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Install iptables] ************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Install iptables-reload command] *********************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Configure dhcpcd] ************************************************************
--- before: /etc/dhcpcd.conf
+++ after: /home/maldridge/.ansible/tmp/ansible-local-3289rle0c9_z/tmp_37j0_hr/dhcpcd.conf.j2
@@ -10,7 +10,7 @@
 
    noipv6
 interface eth1
-	nopipv4
+	noipv4
 
    static ip6_address=2a01:4f8:212:34cc::01d:b/64
    static domain_name_servers=2a01:4f8:0:a0a1::add:1010

changed: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Enable dhcpcd] ***************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Enable wpa_supplicant hook] **************************************************
skipping: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Add dhcpcd iptables hook] ****************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Enable dhcpcd iptables hook] *************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Make iptables.d] *************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Configure base rules for IPv4 firewall] **************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Make ip6tables.d] ************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [network : Configure base rules for IPv6 firewall] **************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [acmetool : Install acmetool] ***********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [acmetool : Create acmetool data root] **************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [acmetool : Create acmetool directories] ************************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=accounts)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=certs)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=conf)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=desired)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=keys)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=live)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=tmp)

TASK [acmetool : Install acmetool responses file] ********************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [acmetool : Check for quickstart flag] **************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [acmetool : Run quickstart] *************************************************************
skipping: [vm1.a-lej-de.m.voidlinux.org]

TASK [acmetool : Install acmetool configuration] *********************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [acmetool : Configure wanted certificates] **********************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=build.voidlinux.eu)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=sources.voidlinux.eu)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=repo.voidlinux.eu)

TASK [acmetool : Ensure cron.d exists] *******************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [acmetool : Install renewal crontab] ****************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [acmetool : Install acmetool firewall rules] ********************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Install nginx] *****************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Configure nginx] ***************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Install dhparam.pem] ***********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Create the webroot] ************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Create sites-available] ********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Create sites-enabled] **********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Enable nginx] ******************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Configure nginx firewall rules] ************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Configure nginx firewall rules] ************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [crond : Install cronie] ****************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [crond : Enable cronie] *****************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Create the void-repo group] **********************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install the buildmaster firewall rules] **********************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install the buildmaster firewall rules (v6)] *****************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install virtualenv & deps] ***********************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Create the BuildBot Master user] *****************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Create the BuildMaster Root Directory] ***********************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install Buildbot] ********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Make Buildbot More Terse] ************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Create BuildMaster Subdirectories] ***************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=scripts)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=public_html)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=templates)

TASK [buildmaster : Copy un-inheritable Buildbot Assets] *************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=bg_gradient.jpg)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=default.css)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=favicon.ico)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=robots.txt)

TASK [buildmaster : Copy Buildbot Bootstrap Database] ****************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install GitHub Webhook Password] *****************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Configure BuildMaster] ***************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install Static Scripts] **************************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=__init__.py)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=ShellCommandChangeList.py)

TASK [buildmaster : Install Buildbot Master Configuration] ***********************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : include_vars] ************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : include_vars] ************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Configure BuildSlave References] *****************************************
--- before: //home/void-buildmaster//buildmaster/scripts/user_settings.py
+++ after: /home/maldridge/.ansible/tmp/ansible-local-3289rle0c9_z/tmpuaj4n11l/user_settings.py.j2
@@ -9,7 +9,7 @@
         'BootstrapArgs': '-N',
         'slave_name': 'x86_64_void',
         'slave_pass': 'REDACTED',
-        'admin': 'xtraeme'
+        'admin': 'gottox'
     },
     {
         'name': 'i686-primary',
@@ -21,7 +21,7 @@
         'BootstrapArgs': '-N',
         'slave_name': 'i686_void',
         'slave_pass': 'REDACTED',
-        'admin': 'xtraeme'
+        'admin': 'gottox'
     },
     {
         'name': 'armv6l-primary',
@@ -33,7 +33,7 @@
         'BootstrapArgs': '-N',
         'slave_name': 'cross-rpi_void',
         'slave_pass': 'REDACTED',
-        'admin': 'xtraeme'
+        'admin': 'gottox'
     },
     {
         'name': 'armv7l-primary',
@@ -45,7 +45,7 @@
         'BootstrapArgs': '-N',
         'slave_name': 'cross-armv7l_void',
         'slave_pass': 'REDACTED',
-        'admin': 'xtraeme'
+        'admin': 'gottox'
     },
     {
         'name': 'x86_64-musl-primary',

changed: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install BuildBot Service (1/2)] ******************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install BuildBot Service (2/2)] ******************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Enable BuildBot Service] *************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Configure webserver] *****************************************************

TASK [nginx : Create folder for external nginx locations] ************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Install site descriptor] *******************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Enable site] *******************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Install firewall rules for resolvers] ******************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Install firewall v6 rules for resolvers] ***************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install root location block] *********************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Create the Signing User] *************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Create .ssh for void-repomaster] *****************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install Signing Key] *****************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Create bin/ directory for void-repomaster] *******************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install Signing and Repo-Management Scripts] *****************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=xbps-sign-repos)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=xbps-clean-repos)

TASK [buildmaster : Install Signing Cronjob] *************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install rsync] ***********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildmaster : Install Sync Keys] *******************************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=None)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=None)
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [root-mirror-shim : Create Repo Directory] **********************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [root-mirror-shim : Create xlocate group] ***********************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [root-mirror-shim : Create Static Mirror Directories] ***********************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=distfiles)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=live)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=logos)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=static)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=current)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=xlocate)

TASK [root-mirror-shim : Mount the package filesystem into the mirror] ***********************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [void-updates : Install void-updates] ***************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [void-updates : Create the voidupdates user] ********************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [void-updates : Install Update Check Cron Job] ******************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [void-updates : Link Results] ***********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [mirror-base : Create the reposync group] ***********************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [mirror-base : Create the reposync user] ************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [rsyncd : Install rsync] ****************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [rsyncd : Install rsync firewall rules] *************************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=iptables.d)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=ip6tables.d)

TASK [rsyncd : Create rsyncd.conf.d] *********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [rsyncd : Template rsyncd.conf] *********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [rsyncd : Enable rsyncd] ****************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [live-mirror : Install Prerequisites] ***************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [live-mirror : Create the mirror dataroot directory] ************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [live-mirror : Configure firewall rules] ************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [live-mirror : Configure webserver] *****************************************************

TASK [nginx : Create folder for external nginx locations] ************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Install site descriptor] *******************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Enable site] *******************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Install firewall rules for resolvers] ******************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Install firewall v6 rules for resolvers] ***************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [live-mirror : Include rsyncd user secrets] *********************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [live-mirror : Configure rsyncd] ********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [live-mirror : Configure rsyncd secrets] ************************************************
skipping: [vm1.a-lej-de.m.voidlinux.org]

TASK [live-mirror : Install sync service secret] *********************************************
skipping: [vm1.a-lej-de.m.voidlinux.org]

TASK [live-mirror : Install mirror sync service (1/4)] ***************************************
skipping: [vm1.a-lej-de.m.voidlinux.org]

TASK [live-mirror : Install mirror sync service (2/4)] ***************************************
skipping: [vm1.a-lej-de.m.voidlinux.org]

TASK [live-mirror : Install mirror sync service (3/4)] ***************************************
skipping: [vm1.a-lej-de.m.voidlinux.org]

TASK [live-mirror : Install mirror sync service (4/4)] ***************************************
skipping: [vm1.a-lej-de.m.voidlinux.org]

TASK [sources_site : Create sources link] ****************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [sources_site : Configure webserver] ****************************************************

TASK [nginx : Create folder for external nginx locations] ************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Install site descriptor] *******************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Enable site] *******************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Install firewall rules for resolvers] ******************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [nginx : Install firewall v6 rules for resolvers] ***************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

RUNNING HANDLER [network : dhcpcd] ***********************************************************
changed: [vm1.a-lej-de.m.voidlinux.org]

PLAY [buildslave] ****************************************************************************

TASK [Gathering Facts] ***********************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildslave : Install BuildBot Slave and Dependencies] **********************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildslave : Create Buildslave user (void-buildslave)] *********************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildslave : Create Buildsync user (void-buildsync)] ***********************************
skipping: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildslave : Create void-buildsync .ssh] ***********************************************
skipping: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildslave : Install sync key] *********************************************************
skipping: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildslave : Create Builder Directories] ***********************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=x86_64)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=i686)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv6l)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv7l)

TASK [buildslave : Enforce permissions on hostdir] *******************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=DE-1)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=DE-1)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=DE-1)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=DE-1)

TASK [buildslave : include_vars] *************************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildslave : Configure buildbot-slave] *************************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=x86_64)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=i686)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv6l)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv7l)

TASK [buildslave : Create buildbot-slave info directories] ***********************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=x86_64)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=i686)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv6l)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv7l)

TASK [buildslave : Configure buildbot host description] **************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=x86_64)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=i686)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv6l)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv7l)

TASK [buildslave : Configure buildbot admin description] *************************************
--- before: //home/void-buildslave//void-builder-x86_64/info/admin
+++ after: /home/maldridge/.ansible/tmp/ansible-local-3289rle0c9_z/tmpq5hidygh/admin.j2
@@ -1 +1 @@
-Juan RP <xtraeme@voidlinux.eu>
+Enno Boland <gottox@voidlinux.eu>

changed: [vm1.a-lej-de.m.voidlinux.org] => (item=x86_64)
--- before: //home/void-buildslave//void-builder-i686/info/admin
+++ after: /home/maldridge/.ansible/tmp/ansible-local-3289rle0c9_z/tmpny2jz3zs/admin.j2
@@ -1 +1 @@
-Juan RP <xtraeme@voidlinux.eu>
+Enno Boland <gottox@voidlinux.eu>

changed: [vm1.a-lej-de.m.voidlinux.org] => (item=i686)
--- before: //home/void-buildslave//void-builder-armv6l/info/admin
+++ after: /home/maldridge/.ansible/tmp/ansible-local-3289rle0c9_z/tmpb6hfxdu2/admin.j2
@@ -1 +1 @@
-Juan RP <xtraeme@voidlinux.eu>
+Enno Boland <gottox@voidlinux.eu>

changed: [vm1.a-lej-de.m.voidlinux.org] => (item=armv6l)
--- before: //home/void-buildslave//void-builder-armv7l/info/admin
+++ after: /home/maldridge/.ansible/tmp/ansible-local-3289rle0c9_z/tmpaw5ppogt/admin.j2
@@ -1 +1 @@
-Juan RP <xtraeme@voidlinux.eu>
+Enno Boland <gottox@voidlinux.eu>

changed: [vm1.a-lej-de.m.voidlinux.org] => (item=armv7l)

TASK [buildslave : Configure xbps-src] *******************************************************
ok: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildslave : Configure local build mirror] *********************************************
skipping: [vm1.a-lej-de.m.voidlinux.org]

TASK [buildslave : Create Service Directories] ***********************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=x86_64)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=i686)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv6l)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv7l)

TASK [buildslave : Configure Runit] **********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=x86_64)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=i686)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv6l)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv7l)

TASK [buildslave : Enable BuildSlave] ********************************************************
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=x86_64)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=i686)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv6l)
ok: [vm1.a-lej-de.m.voidlinux.org] => (item=armv7l)

PLAY RECAP ***********************************************************************************
vm1.a-lej-de.m.voidlinux.org : ok=115  changed=4    unreachable=0    failed=0

The end of the play will always have a "Play Recap" which will show what hosts finished in what state. Always check and if necessary re-apply for hosts that are in failed state.

Applying Ansible playbooks requires unrestricted root or access to a service user on each node. Members of netauth/dante can run playbooks manually.

Certificate Authority

Void operates a private certificate authority based on CloudFlare's cfssl tool. The configuration data for this CA lives in the CA/ directory of the infrastructure repo.

The certificates can be generated using the bin/gencerts.sh script. They should be copied to the appropriate location in the ansible/secret directory after being generated. Once copied, use bin/shred.sh in the CA/ directory to clean up.

LetsEncrypt vs Void CA

When the option exists to obtain a certificate dynamically from LetsEncrypt, this option should be used. Additionally any time a certificate will be visible to an end user this certificate must have a valid trust-root. Since Void's CA isn't trusted by anything automatically user facing certificates MUST be issued by an external CA.

Void's CA should be used for infrastructure needs that require certificates for authentication, or long lived certificates for channel integrity. Void Operations should be consulted before adding any new certificate configurations or adjusting the CA configuration.

Services

Void operates a number of services across the managed fleet. This section documents that various services and appropriate care and feeding.

Services are mapped onto physical or virtual hosts by Ansible configuration. This mapping is encapsulated in the ansible/inventory file. Some services are replicated or distributed. In many cases, services take additional configuration values which are stored in either the host_vars or the group_vars depending on the appropriate variable scope.

acmetool

All SSL certificates for Void are provided by LetsEncrypt via acmetool. Configuration for names requested in acmetool certs are done through various host variables.

Acmetool is configured to run under snooze, and should attempt to renew certificates once a day. Certificates that have more than 30 days remaining will not be renewed. Acmetool does not automatically restart services that consume certificates. In the case of web services, it is assumed that there will be a push that restarts services frequently enough that this will not be an issue.

In the future we may include certificate checks to restart services that do not support dynamic certificate reloading.

BuildBot

BuildBot is our legacy build scheduler.

The buildbot master runs at build.voidlinux.org and provides unified scheduling to all other build tasks in the fleet. BuildBot also exposes a web interface.

The current status of the build infrastructure can be found on the build waterfall. This view shows what each of the buildslaves is doing right now, and uses traffic light colors for build state. A purple builder is usually a reason to contact void-ops and figure out what's wrong with the build host.

Authenticated users of the buildbot can restart builds that have failed without needing to push a new commit. Not all committers have access to restart failed builds this way. If you believe that you should have this access, contact maldridge@.

Moving a buildslave

Don't.

In the event that this is unavoidable, all builds need to be paused until the move can be completed. In the even the builder that needs to be moved is on the musl cluster, all musl builders will need to be moved with it. Similarly, the aarch64 builders must always move as a pair.

EOL

BuildBot is slated for replacement this fall/winter. The system will be replaced by the Distributed XBPS Package Builder (dxpb) which will resolve many of the long standing problems in the buildbot.

Mirrorbits

Mirrorbits is a geographically aware IP based mirror redirection system. We use mirrorbits to dynamically route repository traffic to the nearest mirror that has the correct file to service the request. Mirrorbits is a VLC project with more information available on GitHub. Mirrorbits needs either rsync (preferred) or ftp access to all mirrors to run periodic scans and health checks. Mirrorbits can also handle dynamic handoff between mirrors and is configured to do so by issuing per-file redirects.

Mirrorbits serves traffic under the virtual hostname auto.voidlinux.org.

Useful Tricks

Mirrorbits keeps a log of traffic that has been routed for the last 24 hours. Since mirrorbits knows how large each file it is directing is, the bandwidth cost of each mirror is known to mirrorbits. To see this data visit auto.voidlinux.org?mirrorstats.

The mirrorstats page also has a handy map that shows you where all the servers currently mirroring Void are located. Leads with shell access to the mirrorbits host can summon further information from the mirrordb by using the mirrorbits CLI. For example:

$ mirrorbits --config /etc/mirrorbits/config.yml show a-lej-de
Mirror: vm1-a-lej-de
HttpURL: http://vm1.a-lej-de.m.voidlinux.org/
RsyncURL: rsync://localhost/voidmirror/
FtpURL: ""
SponsorName: Void Linux
SponsorURL: https://www.voidlinux.org
SponsorLogoURL: https://voidlinux.github.io/assets/img/void_fg.png
AdminName: Michael Aldridge
AdminEmail: maldridge@voidlinux.eu
CustomData: ""
ContinentOnly: false
CountryOnly: false
ASOnly: false
Score: 1
Latitude: 49.11594
Longitude: 10.7534
ContinentCode: EU
CountryCodes: DE
ExcludedCountryCodes: ""
ASNum: 0
Enabled: true
AllowRedirects: null

Comment:

If you think that mirrorbits has made the wrong call in where a file was served from, you can append ?mirrorlist to any mirror URL and see why you're being routed to a particular mirror.

An example URL to try: http://auto.voidlinux.org/live/current/sha256sums.txt?mirrorlist

No SSL

Mirrorbits runs without SSL because it needs to be able to issue redirects to other servers. Not all of these mirrors offer SSL connection options and protocol downgrade during a 302 redirect is not well supported. Users that wish to use SSL at all times should be advised to manually configure a preferred mirror.

NetAuth

NetAuth provides all the authentication and authorization information to systems within Void's managed fleet. NetAuth is an open source project with a website at https://netauth.org.

Full documentation and usage information for NetAuth can be found at docs.netauth.org.

Architecture

Void's deployment has a NetAuth server hosted on a dedicated VM which uses certificates from the Void CA for transport security. The server is configured to use the ProtoDB storage engine and is backed up regularly by manual action. Automatic backups are not deemed necessary at this time since the information changes infrequently.

The primary NetAuth server can be reached at netauth.voidlinux.org on port 8443 and uses TLS for all connections.

Remote Linux Systems

Linux systems that need to derive authentication and authorization information are configured to use a combination of pam_netauth and nsscache to provide required services. The authentication information is cached to local systems on use by the PAM Policycache and refreshed periodically. The grooup and authorization information is cached every 30 minutes to disk on all machines. Keys for systems such as SSH are requested on-demand via a helper binary netkeys which does not perform any caching.

While less than ideal, Void could operate for an extended period of time without the primary NetAuth server running.

Basic Administration

NetAuth uses a capability based system for administration of itself. Members of group dante have permissions to make changes on behalf of other users and generally should be the only people making changes to the directory.

Adding a New User

When adding a new user make sure to specify the username and number to ensure the number is in the range that will be cached by nsscached.

$ netauth new-entity --ID <username> --number <number>

Making an entity a valid shell user

Shell users have additional required attributes, these can be set seperately:

$ netauth modify-meta --ID <username> --primary-group netusers --shell /bin/bash

For all users the primary group should be netusers and the shell should generally be /bin/bash. Additional fields may be set as needed.

Adding an entity to a group

Groups are used to gate access to all resources across the fleet. For example to add a new build operator who can unwedge the buildslaves, the following command sets the appropriate groups:

$ netauth entity-membership --ID <username> --group build-ops --action add

Adding and removing SSH keys

Adding and removing SSH keys is done with the netauth command. The default type of key is SSH. When adding and removing keys the key content needs to be quoted to avoid splitting by the shell. When removing keys the server will match keys on substrings, so technically the key comment should be sufficient to remove it if it is unique.

$ netauth modify-keys --ID <username> --mode ADD --key "<key>"

nginx

Void's preferred webserver is nginx using drop in config fragments. All nginx instances are managed by Ansible and have an Apache2 style sites-available and sites-enabled directory structure in /etc/nginx/. Additionally an /etc/nginx/locations.d/ exists for each site to provide location {} fragments that may not be owned by the same task that created the original site.

When possible, it is preferable to proxy web services through nginx to do TLS termination and to abstract certificate handling away from backend services. Services that communicate via protocols that use HTTP as a transport such as gRPC services do not need to use nginx as a proxy.

PopCorn

PopCorn is Void's package popularity system, similar to Debian's popularity contest system popcon, from which the name was inspired.

Information in depth about PopCorn can be found at the project's GitHub repository.

Querying Stats

Stats from PopCorn are available for anyone who wishes to query the system. The server is live at popcorn.voidlinux.org with reports services on port 8000 and the stats repository available on port 8003.

Getting the day's stats

You can download the current raw stats at any time. These are the stats that are written to the per-day files at the PopCorn site.

$ popcornctl --server popcorn.voidlinux.org --port 8001 report

If no file is specified then output.json will be written to the current directory. If a file is specified by passing --file <path> to report, then the output will be written to the named file.

Finding out versions of a package

The versions for a package can be queried from the statsrepo with popcornctl. By default the stats are queried over the most recent 30 day interval. To get known versions us the following query:

$ popcornctl --server popcorn.voidlinux.org --port 8003 pkgstats --pkg <pkg>

Additional formatting options are available by specifying --format. Useful alternate formats are date and csv which provide information about versions seen over time.

rsyncd

All managed mirrors provide unauthenticated rsync. Like nginx rsyncd is configured to use drop in files read from /etc/rsync.conf.d.

rsync is the preferred way to mirror large amounts of package data between two locations, even for ad-hoc migrations. For persistent sync the rsync protocol (rsync://) is preferred.

void-updates

The void-updates site provides a text file every day that shows all package maintainers and all packages with updates known.

Because of the mechanism by which void-updates works, care must be taken to not let it run unthrottled. We have configured it to scrape for updates once per day, and this seems to be infrequent enough to keep most webmasters happy.

While a manual run of void-updates can be triggered, be aware that this can cause instability in the output data and is discouraged.

xlocate

The xlocate service provides the data source for consumption by the xlocate command from the xtools package. This task is responsible for once a day regenerating the search index that is used for all package files.

Required Colocation

Because the xlocate indexing task requires running xbps-query over all packages in x86_64, this task must be colocated with a package mirror. The mirror must also be configured locally so that xlocate does not unecessarily load a webserver on the same host.

Organization

Void Linux is a controlled anarchy. This is working as intended, and we like it this way. We've decided that it's better to have flexible workflows that can adapt to new situations as they arise rather than needing to consult detailed documentation or request authorization in advance.

Rather than determining processes for every action, we instead choose to trust our members to think on their feet and come up with reasonable solutions.

We still need some processes though, and we need consistency in the way that people think about problems. The processes described in this section aim to keep the organization running.

Onboarding

This section explains how new members are proposed, approved, and new permissions are assigned.

Proposing a New Member

Any existing member of the organization may propose a candidate. Candidates are expected to have been an active part of the organization for some time prior to proposing.

The proposal should take the form of written notice to the existing team via some private channel. Email is a good option for this. The mail should include the following information:

  • Candidate's name or well-known username
  • Candidate's contributions to Void, ideally as bullet points
  • A deadline by which comments should be provided, at least 1 week

After the comment period expires, the comments and final statements should be reviewed. Organization owners will have the final ability to approve a proposal and to dismiss comments or objections to the proposal.

If the proposal is approved, the candidate should be contacted, if they agree to join the organization, proceed to the next section.

Onboarding an Approved Candidate

Onboarding the candidate requires at minimum a member of netauth/terraform to run the final commands that will add the new candidate. During the review approval process, a Void Ops lead should have agreed to handle the proposals, they will ensure the following steps are done.

A patch should be created for the void-infrastructure repo which adds the candidate to the github_members.tf file. This patch should be sent to the repo as a pull request. The pull request should contain the same information as the email to internal team members, including a comment period of at least one week.

At the conclusion of the comment period, a final decision will be made to submit the patch or to rescind the invitation. This decision is made by agreement of the organization owners. The default is to approve.

Unless the new member objects, a post should be made to voidlinux.org welcoming them to the organization (a new member may not wish to have this kind of attention immediately).

Additional Powers

Once added to the organization, additional powers may need to be delegated. This should be discussed in the PR that added the individual to the organization.

Additional powers include:

  • Forum Moderator status
  • NetAuth system account
  • NetAuth mail account
  • NetAuth group memberships

Consult the service specific documentation for how to apply additional permissions as needed.

Offboarding

Sometimes people leave the project, either of their own volition or with a helping hand. Its very important to ensure that things are done right and according to this process when someone leaves, so please read it in full.

Cause for following these offboarding guidelines can take the form of any of the following non-exhaustive list:

  • acted maliciously
  • have become inactive
  • wish to be removed
  • or any other reason that justifies the removal

Proposing a Removal

Any current member of the Void Linux Organization can propose a removal. There are two ways this process may go depending on if the removal candidate is also the requesting entity.

Removing Yourself

If you'd like to remove yourself, contact an administrator of the organization and state your request plainly.

  1. Contact an administrator via a private channel.
  2. The administrator will confirm the request via your registered email address.
  3. The administrator will file a PR to the void-infrastructure repo to be processed by void-ops.

If you're leaving the project permanently, attempting to find a successor to maintain your packages is greatly appreciated.

Removing Someone Else

Removing someone else is a more involved process and may require more discussion.

If you're proposing a security removal, escalate to void-ops directly, these are processed immediately.

  1. Send an email to an administrator via their registered email address. This email should contain the person you believe should be removed and why.
  2. The administrator should contact the named individual and notify them of a removal request. This should be done via the registered email. This email must also contain a deadline of at least one week for the individual to make a statement.
  3. Any statement should be considered and discussed amongst the admins of the organization. This may require pushing the deadline back.
  4. Once an agreement has been reached the removal will either proceed or be dropped. In the case no agreement can be reached after a reasonable amount of time has passed and a sincere effort from both admins and the individual, dropping membership is the default resolution.
  5. A ticket is created for void-ops to process the removal.

Ops Removal Checklist

Removing a member of the organization follows the following checklist which can be copy/pasted into a markdown aware ticket.

  • Remove access from the GitHub organization via TerraForm
  • Remove all memberships in NetAuth, removing the NetAuth principal is optional, but discouraged.
  • Remove any moderator bits in the forum
  • Remove any moderator bits in the wiki
  • Remove any existing manually provisioned mail aliases

Preserving NetAuth entities seems a bit unusual, but it prevents accidentally re-provisioning the name later, and the default groups grant no non-public access.

Forum Moderation

The forum is a managed communications space running at https://forum.voidlinux.org. The senior team rarely frequents the forum, and so we delegate the powers of moderation to selected members of the forum.

Things we generally expect forum moderators to do:

  • Keep the peace on the forums
  • Keep the spam down

Things to keep an eye out for:

  • Links to random user repositories (which are not allowed), with the exception of repositories that are testing things scheduled to be merged to master
  • Spam of any kind
  • Uncivil discourse

We expect moderators to exercise good judgement when determining what needs action from a moderator, and in cases of question to consult senior members of the team.