🌻:Infrastructure And Operations/Inventory/Matrix

To be able to log into the matrix server, you need to have a user account created for you by Ansible.

That is managed by these files in the ansible playbook:

1. Add your public key file to
git.cyberia.club/cyberia/ops-handbook/src/branch/main/ansible/files/keys
2. add yourself as an operator
git.cyberia.club/cyberia/ops-handbook/src/branch/main/ansible/group_vars/all#L3
3. log into leckie.cyberia.club and invoke ansible (become root with sudo -i and then look in the shell history for examples)

Example of logging in and becoming root:

forest@debian:~$ ssh matrix.cyberia.club
Last login: Tue Dec  2 03:13:01 2025 from 198.74.6.203
(>'-')>  ┻━┻  cy83RiA  k4f3  &   clu8
$ sudo -i
root@matrix:~# pwd
/root
root@matrix:~# 

ℹ️ INFO: my username on my laptop here is the same as my username on the matrix server (forest), so I didn't have to provide the username in the ssh command. Otherwise I would have had to type ssh forest@matrix.cyberia.club

Table of Contents

• List of admin Scripts that are in roots home directory /root

• List of places where Software is Configured

• List of places where Data is Stored

• Example of how to discover this stuff yourself

• Upgrade Procedures

• Backup info

• how to set retention period on matrix rooms

List of scripts that are in roots home directory /root:

-rwxr-xr-x  1 root     root      423 Jan 23  2022 delete-account
-rwxr-xr-x  1 root     root      251 Jan  1  2023 get-delete-room-status
-rwxr-xr-x  1 root     root      285 Mar 23  2024 get-user-last-login
-rwxr-xr-x  1 postgres postgres 3.3K Jan  1  2023 iterate-kick-users-and-purge-room
-rwxr-xr-x  1 root     root      499 Feb 22  2025 join_user_to_room
-rwxr-xr-x  1 root     root     1.5K Oct  8  2022 kick-users-and-purge-ban-room
-rwxr-xr-x  1 root     root      763 Jan  1  2023 kick-users-and-temp-delete-room
-rwxr-xr-x  1 root     root      421 Nov 18 23:25 make-room-admin
-rwxr-xr-x  1 root     root      357 Aug 12  2022 purge-remote-media
-rwxr-xr-x  1 root     root      365 Oct  8  2022 purge-room
-rwxr-xr-x  1 root     root       99 Dec 29  2022 purge_from_file.sh
-rwxr-xr-x  1 root     root      351 Feb 22  2025 reactivate-account
-rwxr-xr-x  1 root     root     1.6K Aug  7  2021 register-user
-rwxr-xr-x  1 root     root      601 Sep 23 22:51 reset-password

This was generated w/ ls -lah | grep -v drwx | grep xr-x.

list all files with details, filter out folders (drwx) and only include executable files (xr-r)

I would recommend reading each script to learn how it works before trying to use it.

Here is an example:

root@matrix:~# cat make-room-admin 
#!/bin/sh

roomid="$1"
userid="$2"

[ -z "$roomid" ] && echo 'first arg (roomid) missing' && exit 1
[ -z "$userid" ] && echo 'second arg (userid) missing' && exit 1

echo "making $userid admin in $roomid..."

curl -v -H "Content-Type: application/json" -X POST "localhost:8008/_synapse/admin/v1/rooms/$roomid/make_room_admin?access_token=[redacted]" \
        --data '{ "user_id": "'"$userid"'" }'

⚠️ EEP! the admin's access_token is hard coded into the script here!

This is an artifact of laziness. It's mostly fine, but just something to be aware of before you go sharing the contents of these scripts!! 😱

Anyways, from this script, we can see that the room id is the first argument $1 and the userid is the second argument $2.

So in order to run this, you would run a command like:

./make-room-admin '!qTIhdNcMzGhdFLohBS:cyberia.club' '@forestjohnson:cyberia.club'

ℹ️ INFO: the room id MUST be single quoted in shell commands, because ! is a syntax character in the shell, just like $. If it is inside a double quoted string, it will be interpreted as syntax rather than just plain text.

You can get the room id from Settings --> Advanced in element-web:

Places where Software is Configured

matrix-synapse, this is the main "matrix homeserver" implementation

postgres, the database used by matrix-synapse

matrix-appservice-discord, this is discord bridge

matrix-diskspace-janitor, custom webservice forest made to track disk usage and delete rooms that were victims of spam attacks and have polluted the disk.

mjolnir? TODO

Places where Data is Stored

Example of how to find the configs yourself

The matrix server is based on debian and as such it uses systemd to define services that run automatically, such as matrix-synapse itself.

We have a cheat sheet for systemd

But for convenience, I'll put an example in-line here:

root@matrix:~# systemctl status matrix-synapse
● matrix-synapse.service - Synapse Matrix homeserver
     Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2025-11-18 20:23:16 UTC; 1 months 4 days ago
   Main PID: 3227424 (python)
      Tasks: 47 (limit: 9527)
     Memory: 1.7G
        CPU: 1w 17h 20min 46.764s
     CGroup: /system.slice/matrix-synapse.service
             └─3227424 /opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/

See where it says Loaded: loaded (/lib/systemd/system/matrix-synapse.service;?

That's the file path of the "systemd service unit" config file that defines when matrix starts, how its configured, etc.

So we can just go look at that file:

root@matrix:~# cat /lib/systemd/system/matrix-synapse.service
[Unit]
Description=Synapse Matrix homeserver

[Service]
Type=notify
User=matrix-synapse
WorkingDirectory=/var/lib/matrix-synapse
ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys
ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=3
SyslogIdentifier=matrix-synapse

[Install]
WantedBy=multi-user.target

This is helpful because it tells us where the matrix-synapse code is located (/opt/venvs/matrix-synapse) and it also gives us two more places to look for config info:

• /etc/matrix-synapse/homeserver.yaml

• /etc/matrix-synapse/conf.d/

Upgrade Procedures

TODO

Backup info

The entire VM including all the data is backed up via the normal capsul.org backup system

Remote backup for the matrix server DB is configured by:

root@matrix:~# cat /etc/cron.daily/pgbackup 
#!/bin/sh

/opt/postgres-backup/run-backup.sh 2>&1 >> /var/log/pgbackup.log

root@matrix:~# cat /opt/postgres-backup/run-backup.sh
#!/usr/bin/env bash

set -o pipefail

echo "pg-backup is starting up!"
echo "this script is run by cron.daily"
date

cd /opt/postgres-backup
sudo -u postgres pg_dump synapse | pv --interval 5 | zstd -fo "synapse.zst"

backup_exitcode="$?"

echo "pg_dump | pv | zstd -fo 'synapse.zst' exited with status code $backup_exitcode"
date

if [[ $backup_exitcode -eq 0 ]]; then
  set -e
  echo "transferring backup offsite.."
  rsync --whole-file --recursive --delete-after --progress --stats --human-readable \
    -e "ssh -p 2022 -i /opt/postgres-backup/id_ed25519 -o UserKnownHostsFile=/opt/postgres-backup/known_hosts -o IdentitiesOnly=yes" \
    /opt/postgres-backup/synapse.zst backups@layerze.ro:/tank/backups/postgres/synapse.zst
  date
  echo "running prom-collect..."
  prom-collect postgresql_backup
  echo "success!"
fi

how to set retention period on matrix rooms

SEE: github.com/matrix-org/matrix-spec-proposals/blob/matthew/msc1763/proposals/1763-configurable-retention-periods.md

TBH i'm struggling a bit to set the retention in an automated fashion at all. We don't have a tool for it and it requires having admin in the room.
The best I was able to figure out so far

1. go to the cyberia space with the network tab of the dev tools open

2. look for the _matrix/client/v1/rooms/!XJhdXteYnRMldzWFyX%3Acyberia.club/hierarchy?suggested_only=true&max_depth=1&limit=20 GET request and copy its json response

3. extract the room ids with jq
4. ssh into the matrix server and run ./reset-password ed-209:cyberia.club
5. log in as ed-209
6. go to a room then type /devtools to open the devtools, and make sure the browser network tab is open
7. send a custom room state event with event type m.room.retention and body {"max_lifetime":86400000000}
8. find the PUT _matrix/client/r0/rooms/!roqAKCndauLBfhLdGs%3Acyberia.club/state/m.room.retention/ request in the network tab and right click -> copy as curl
9. write a script that runs this curl command for every room ID that you collected

(()=>{

  const list = `!JFYMcOkcWtccikJlkA%3Acyberia.club
!SfvWHaBPcmgzTWdvZB%3Acyberia.club
!hYrELkZnhpqlzZoQZZ%3Acyberia.club
blahbalahblahbah....`.split("\n");

const space = `


`;
  console.log(space + 
  list.map(roomid => {
    return `curl 'https://matrix.cyberia.club/_matrix/client/r0/rooms/${roomid}/state/m.room.retention/' -X PUT  AUTH HEADER GOES HERE!!!  --data-raw '{"max_lifetime":86400000000}'`
  }).join('\n\n\n')+space);

})()

so far its been slightly hit and miss but I'll check a few rooms and make sure it at least appears to have worked

( you can check a room by typing /devtools in the message box and then going to the explore room state section, there should be an entry for m.room.retention )