[[]]---
title: Inventory
---

Last Updated: Mar 2023


## Loose Notes

Have something you want to document, but not sure where to put it or how to format it ? Please go right ahead and dump it into the pre-formatted text block below!!

```
resize qemu image:
  # first, shut down the capsul, then:
  $ cd /tank/vm
  $ qemu-img resize cvm-lqj2x9nxic.qcow2 +50G
  $ virsh start cvm-lqj2x9nxic.qcow2

you may have to resize the partition by hand internally, depending on the distro.
```


-----------------

## Overview


img {
   infrastructure_and_operations/inventory/overview.jpg
}


This diagram was created with https://app.diagrams.net/.
To edit it, download the [[infrastructure_and_operations/inventory/overview.drawio | diagram file]] and edit it with the https://app.diagrams.net/ web application, or you may run the application from [[https://github.com/jgraph/drawio | source]] if you wish.

## Physical Hosts

*  baikal
**  69.61.110.118
**  cyberia's first rack server, installed in  CyberWurx datacenter in Atlanta Georgia
**  j3s and forest are the only authorized support contacts on the CyberWurx portal right now

* beet
**  beet was recently rebuilt with a new motherboard and CPU. it has 4x 2GB  HDDs
**  it's currently located at layer zero - `ssh layerze.ro`

## Cloud Service Accounts

* namecheap.com
**  we use j3s's to manage DNS entries for:
***   cyberia.club
***   nullhex.com
***   capsul.org
**  all DNS updates are automated via https://git.cyberia.club/cyberia/dns
**  conventions:
***   A records are named after hostnames & point to VMs / physical hosts
***   CNAMEs are named after the service & point to the A record of the host the service runs on

* CyberWurx portal
**  Allows us to add reverse DNS entries for Capsuls
**  View metrics, get datacenter information, support tickets, etc
**  Right now j3s/forest are the only one who can log in / be authorized for support. Can add others though!

*  Capsul.org
** Cyberia has an internal capsul account that we use. If you want access to this account, talk to j3s, vvesley, or forest.

* njal.la
** For redundancy and random domains, such as:
***   cyberia.top
***   cyberia.tube
***   cyberia.social
** conventions: None. DNS updates are currently not automated. To update dns or to add / manage domains log into https://njal.la with treasurer@cyberia.club or use the [[https://wiki.cyberia.club/hypha/infrastructure_and_operations/njalla | api]].


## Capsul

Most of cyberia's services run on [[https://capsul.org | Capsul]], our Virtual Machine Management tool & service.

```
Ansible Managed Capsuls:
capsul-ay3yh10q2q  f1-xs  69.61.2.230  alpine311  Jun 20 2020 domechild.cyberia.club  (email server)
capsul-c04bbf593b  f1-s   69.61.2.246  alpine311  Jun 01 2020 raaz.cyberia.club       (NSHC / North Star Health Collective) 
capsul-pfgy2tthx9  f1-xs  69.61.2.167  alpine311  May 10 2020 legion.cyberia.club     (postgres database)
capsul-t6tfb2dh5p  f1-m   69.61.2.183  alpine311  May 10 2020 mothership.cyberia.club (prometheus & grafana & future log agg)
capsul-w6hsx09r7v  f1-xs  69.61.2.213  alpine311  Aug 20 2020 leckie.cyberia.club     (ansible bastion + build submitter)
capsul-f6crtfzx5c  f1-xs  69.61.2.218  alpine313  Mar 01 2021 comet.cyberia.club      (owncast server)
cvm-lqj2x9nxic	   f1-l   69.61.2.190  debian10   Mar 07 2020 matrix.cyberia.club     (cyberia matrix) 
cvm-m1tjv0lljd	   f1-xs  69.61.2.178  debian10   Mar 10 2020 elliot.cyberia.club     (websites, this wiki, goatcounter)
capsul-sbsmrkpgx7  f1-xs  69.61.38.199 debian10   Aug 01 2021 paimon.cyberia.club     (git.cyberia.club)
capsul-nnzryhg9df  f1-xs  69.61.2.198  alpine315  Jun 15 2022 zicocapsul.cyberia.club (gancio server)
```

The Ansible Managed servers should have a user account for each user. 

Manually Managed Capsuls:

```
capsul-le2l50mbln  f1-x  69.61.2.225  Debian 11 (bullseye)   Apr 26 2022 pigsy.cyberia.club   (code-server for helloworld labs)

NOTE: currently capsul-le2l50mbln exists under forests capsul account 

capsul-hcdyq2fgdu  f1-xs  69.61.38.214  archlinux (nyaaori custom manjaro installation)  Sep 21 2021 nyanjaro.cyberia.club  (matrix bridges)

NOTE: the ssh username for nyanjaro is "user" not "cyberian". Only nyaaori, forest, and j3s have access so far.
```

#### Host Key Fingerprints

**NOTE:** you can control what kind of host key your ssh client will use like this: 

`ssh -o HostKeyAlgorithms=ssh-ed25519 example.cyberia.club`

```
rathouse.layerze.ro
  RSA      SHA256:gqFzujf1ar0GONYLzJ6zLIHeLbaNocZYaLxM25R/Jx0  
  ECDSA    SHA256:cCo0LKZyGV2vJSdd5ePqCxhanDqeQHQvhgzUJkPn1qg 
  ED25519  SHA256:exR3rca77jgHeDx2VocoGmLvDMRfJQ4mRgGOrA59WOQ 

baikal.cyberia.club
  ECDSA    SHA256:85GTFfUpDDefcNcIROtFpuTiHC1j3iNU74aaKFO03+0
  ED25519  SHA256:v9MEa97wnmA75CyzQC5lW8nOI56LJ4jTmD2f68udK80

beet.cyberia.club
  NOTE:    SSH to sequentialread.com at port 3791 while this server is at forests house
  ECDSA    SHA256:kPOBn03CH176zrTlFDVmjFJpWi1OGHhkNCiK6stNn/0
  ED25519  SHA256:7M8ppVJ534Axz1ZXt6NheBxYkqY9UJ3AAmb9BmY9bYk

dredd.cyberia.club
  NOTE:    dredd uses port 3217 for ssh. connect with ssh -p 3217 dredd.cyberia.club
  ECDSA    SHA256:5157aYG7PT8Y0I4sTzlpQ5i/E3bq4aPF9T1P+xj+l9Q
  ED25519  SHA256:w6F0NXBoLCXG60yXoI3QhYGiLlPCr6YrK/OUSSDcmAw

mothership.cyberia.club
  ECDSA    SHA256:3XJG2fyaPDJWjnEOW3q2KiWg5qLV6hmEPczvp8GqhE0
  ED25519  SHA256:njIT2k1t6hHuOO0VjBNmHW1QSGN4GEqQQMj/BGpnBa0

domechild.cyberia.club
  ECDSA    SHA256:IQqTPv14u3dG62hS0q2Mr6pef6KwpjPKM2uVP+SK+qA
  ED25519  SHA256:3z5BI2ZEZjzDEh0B7a2GxgMa4faqA3Y6bQdGcQp4G88

legion.cyberia.club
  ECDSA    SHA256:EW9ydcgLg/pwoA0GPsI0VVeIBpnSi7aIHhvXOQBa+Xg
  ED25519  SHA256:cWLBFESOHrmVFrLRLjxrY4tcPmVRerJe1SB/+6tXSxAv

leckie.cyberia.club
  ECDSA    SHA256:KbzxzEKP21B0S3A/SKqqGmjiymnkk7byvoc6W4SxEwM
  ED25519  SHA256:M1QPflfIrsbhVlMaomvGQsr5AZS5YRkBHv+pnyI7bg4

raaz.cyberia.club
  ECDSA    SHA256:AJb0bZN2PTTm83zf5zI1IOEIVfeXUxQl/vTode/88jA
  ED25519  SHA256:zJv6E6lG4dAsqNmDHTO/qFVlTESKYq/KD29e8Nt/6j4

matrix.cyberia.club
  ECDSA    SHA256:VlRPAqLGxY4JUVhYirOVlfuDFtgTbaiw3x29xYizEeU
  ED25519  SHA256:BExhsVPNTp49jyJ6ezRf+Nn4TxPj8D9VZMhnjMABq6g

elliot.cyberia.club
  ECDSA    SHA256:/tsASDZ+MX519DC/Y7mHbV2CYCPnyMAbX1e0GHBOin0
  ED25519  SHA256:B9QNCnz57agsI40tMVU8UwyvZqMbz/p1ZNH5E1gL3io

comet.cyberia.club
  ECDSA    SHA256:UcDUCFd/U3F8ECG/RKxLbJRAAiMBSRKVKqDM0hmjwJ8
  ED25519  SHA256:SoOuSzKmpUd4x8Y8G32EAfQTY15agz1z7zJJCWdI8Tw

paimon.cyberia.club
  ECDSA    SHA256:IILubNkDwqzi1/6l5UCm24MvoxyxB6Y/m0BGWSmePZ0
  ED25519  SHA256:UsYrMq7nqxAND81Qzpgpzqz5ZxC/or6T0KIikM0tY9Q

nyanjaro.cyberia.club
  ECDSA    SHA256:hCpko+C9zSVNBC/76Ji6sjfMrj7f0+xnpLpqQEVK3oY
  ED25519  SHA256:AzT2aBvAJfD4JEq062I+NhPd5tVE0fi+m5ixnu5SnLs
  RSA      SHA256:taJdsRqtL2D80buYxcRgDbqitZ7tbuoX469Du6dN7yI    

pigsy.cyberia.club
  ECDSA    SHA256:fLOy8ZtOJg7SaQUMvpDI/33CYHKMJYyaq+53Q2kytEQ
  ED25519  SHA256:jbAnwc9nYJ95mjK3GS3mak9TM7hvACA19OGb/WciqyE
  RSA      SHA256:iDynajywy4D1pCrvIhG/i9WEHLgJh5CgaCqQTXUZfEw

zicocapsul.cyberia.club
  ECDSA    SHA256:GJZzDRtDZY5Bz4XTshWKn6GUb0BEjCQwyEBlr/Zg23c
  ED25519  SHA256:xUXIdvkFAhBtURyTLvoMBjYtuwRkpZHkEgGPyx0c9KI
  RSA      SHA256:fEOqNEy6/tNiYxtaRXg1I9YO0JPn4YRhqgqvAvJz8xI
```


## Automation (Ansible)

The [Ops Handbook](https://git.cyberia.club/services/ops-handbook/) is still on the old git server, it is the main repo with the ansible inventory & playbooks.

Ansible bastion host/automation is on leckie.cyberia.club


## Service Inventory

table {
!  User-oriented Name   !        URL        !  Developer-oriented Name  !        Host           !    Deployment Code !  Application Code !
|cyberia's matrix server| [[https://matrix.cyberia.club/]] |         synapse         |  matrix.cyberia.club  | [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/synapse | ansible/roles/synapse ]] | [[ https://github.com/matrix-org/synapse | matrix-org/synapse ]] |
|cyberia's matrix server| [[https://riot.cyberia.club/]] | element (used to be called riot) |  matrix.cyberia.club  | [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/riot | ansible/roles/riot ]] | [[ https://github.com/vector-im/element-web | vector-im/element-web ]] |
|cyberia's matrix server| N/A                       |  postgres |  matrix.cyberia.club  | [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/postgresql | ansible/roles/postgresql ]] |  [[ https://git.postgresql.org/git/postgresql.git | git.postgresql.org ]] |
|cyberia's matrix server| [[https://matrix.cyberia.club/_synapse/metrics]] | matrix prometheus exporter |  matrix.cyberia.club  |     TBD    | [[ https://github.com/matrix-org/synapse/tree/develop/synapse/metrics | matrix-org/synapse/metrics ]] |
|ed-209 (mod bot)| N/A                       |  mjolnir |  beet  | TBD |  [[ https://github.com/matrix-org/mjolnir | matrix-org/mjolnir ]] |
|wiki-update-bot | N/A                       |  rssToMatrix |  elliot  | TBD |  [[ https://git.cyberia.club/cyberia/rssToMatrix | rssToMatrix ]] |
| matrix irc bridge  | [[https://web.libera.chat/#cyberia.club-cyberia]]            | heisenbridge   | nyanjaro.cyberia.club | TBD | [[ https://github.com/hifi/heisenbridge | hifi/heisenbridge]] |
| matrix irc bridge  | bridges.cyberia.club            | catalyst   | nyanjaro.cyberia.club | TBD | TBD (ask nyaaori) |
|nullhex email          | [[https://nullhex.com/]]      | alps |  elliot.cyberia.club  |     TBD    | [[ https://sr.ht/~emersion/alps/ | ~emersion/alps/ ]] |
|nullhex email          | nullhex.com ports 25 & 587 (STARTTLS) | opensmtpd |  domechild.cyberia.club  |     [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/opensmtpd | ansible/roles/opensmtpd ]]    | [[ https://github.com/OpenSMTPD/OpenSMTPD/ | OpenSMTPD/OpenSMTPD ]] |
|nullhex email          | nullhex.com:993 (imap)   | dovecot |  domechild.cyberia.club  |     [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/dovecot | ansible/roles/dovecot ]]    | [[ https://github.com/dovecot/core | dovecot/core ]] |
|nullhex email          | N/A                       | rspamd  |  domechild.cyberia.club  |     TBD    | [[ https://github.com/rspamd/rspamd | rspamd/rspamd ]] |
| capsul                | [[https://capsul.org]]       | capsul | baikal.cyberia.club | TBD       | [[ https://giit.cyberia.club/~forest/capsul-flask/ | ~forest/capsul-flask/ ]] |
| cyberia's git server | [[https://git.cyberia.club/]] | gitea ( | paimon.cyberia.club |   TBD       | [[ https://gitea.com/ | gitea ]] |
| capsul / mailing lists | [[https://lists.cyberia.club/]] | postgres | legion.cyberia.club |     TBD    | [[ https://git.postgresql.org/git/postgresql.git | git.postgresql.org ]] |
| concourse (the new build server) | [[https://concourse.cyberia.club/]] | concourse | rosewater.cyberia.club | TBD | [[ https://github.com/concourse/concourse | concourse/concourse ]] |
| vault (build secrets manager) | N/A | vault | rosewater.cyberia.club | [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/concourse-vault/files | ansible/roles/concourse-vault ]] | [[ https://github.com/hashicorp/vault | hashicorp/vault ]] |
| cyberia's website     | [[https://cyberia.club/]]    | caddy static site   | elliot.cyberia.club |     TBD    | [[ https://git.cyberia.club/services/website/ | services/website ]] |
| prometheus   | [[https://prometheus.cyberia.club/]] | prometheus   | mothership.cyberia.club | [[ https://git.cyberia.club/services/ops-handbook/tree/rules | rules ]] & [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/prometheus | ansible/roles/prometheus ]] | [[ https://github.com/prometheus/prometheus | prometheus/prometheus ]] |
| alertmanager   | N/A                            | alertmanager   | mothership.cyberia.club | same as prometheus | [[ https://github.com/prometheus/alertmanager | prometheus/alertmanager ]] |
| grafana          | [[https://grafana.cyberia.club/]] | grafana   | mothership.cyberia.club | [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/grafana | ansible/roles/grafana ]] | [[ https://github.com/grafana/grafana | grafana/grafana ]] |
| Jackal           | [[https://bot.j3s.sh]]            |   go-neb (matrix bot)  | mothership.cyberia.club | TBD | [[ https://giit.cyberia.club/~forest/go-neb/log/forest-feature-rebase-2 | matrix-org/go-neb (forest's fork) ]] |
| web analytics   | [[ https://goatcounter.cyberia.club ]]            | goatcounter   | elliot.cyberia.club | [[ https://git.cyberia.club/cyberia/ops-handbook/src/branch/master/ansible/roles/goatcounter | ansible/roles/goatcounter ]] | [[ https://github.com/arp242/goatcounter/ | arp242/goatcounter/]] |
| web analytics  | [[ https://goatcounter.cyberia.club/ ]]            | goatcounter-caddy-log-tailer   | elliot.cyberia.club | [[ https://git.cyberia.club/cyberia/ops-handbook/src/branch/master/ansible/roles/goatcounter-caddy-log-tailer | ansible/roles/goatcounter-caddy-log-tailer ]] | [[ https://git.sequentialread.com/forest/sequentialread-caddy-config/src/branch/master/dockerbuild_goatcounter/goatcounter-caddy-log-adapter | forest/.../goatcounter-caddy-log-adapter ]] |
| web analytics   | [[ https://goatcounter-capsul.cyberia.club ]]            | goatcounter-nginx-log-tailer   | baikal.cyberia.club | TBD (roughly based on [[ https://git.cyberia.club/cyberia/ops-handbook/src/branch/master/ansible/roles/goatcounter-nginx-log-tailer | ansible/roles/goatcounter-nginx-log-tailer ]]) | [[ https://github.com/arp242/goatcounter/ | arp242/goatcounter/]] |
| Stream           | [[https://stream.cyberia.club]]            | owncast   | comet.cyberia.club | TBD | [[ https://github.com/owncast/owncast | owncast/owncast ]] |
| wiki / cyberdex   | [[https://wiki.cyberia.club]]            | Mycorrhiza   | elliot.cyberia.club | TBD | [[ https://github.com/bouncepaw/mycorrhiza | bouncepaw/mycorrhiza]] |
| layer zero calendar   | [[https://https://calendar.layerze.ro/]]            | Gancio   | zicocapsul.cyberia.club | [[ https://git.cyberia.club/cyberia/ops-handbook/src/branch/master/ansible/roles/gancio | ansible/roles/gancio ]] | [[ https://framagit.org/les/gancio | framagit.org/les/gancio]] |

}

## Metrics Inventory

We use Prometheus and Grafana to capture, store, and display metrics related to our services.

There are four main dashboards in grafana:

*  [[  https://grafana.cyberia.club/d/rYdddlPWk/node-exporter-full?orgId=1 | Node Exporter ]]
** The prometheus node exporter runs on all of our servers and is [[ https://git.cyberia.club/cyberia/ops-handbook/src/branch/master/ansible/roles/common/tasks/main.yml#L66 | managed by ansible ]]. 
*  [[ https://grafana.cyberia.club/d/wGgaPlciz/postgres-overview?orgId=1 | Postgres Overview ]]
** These metrics are reported by a postgres exporter which was installed manually on matrix.cyberia.club and legion.cyberia.club
*  [[  https://grafana.cyberia.club/d/000000026/cyberia-synapse?orgId=1 | Cyberia Synapse ]]
** These metrics are reported by synapse itself. If they are not working, it may be an issue with the reverse proxy in front of synapse. 
*  [[ https://grafana.cyberia.club/d/jMw9xSRMz/capsul-stats?orgId=1 |  Capsul Stats ]]
** these metrics are reported by https://git.cyberia.club/cyberia/libvirt_exporter (running on baikal). They are used to generate the graphs in the capsul.org web interface

## LetsEncrypt Certificate Inventory

For information on certificates which are managed by uacme automatically, see 
https://git.cyberia.club/services/ops-handbook/tree/ansible/hosts
and the `tls_certs` variable in https://git.cyberia.club/services/ops-handbook/tree/ansible/group_vars

Certificates which are exceptions to the rule: 

```
btcpay.cyberia.club
 - btcpay.cyberia.club certificate is automatically managed by the caddy server running on elliot

baikal.cyberia.club
elliot.cyberia.club
The following are managed by a script called acme.sh located at `/root/.acme.sh/`
 - capsul.org
 - www.capsul.org
 - nullhex.com
 - cyberia.club
 - git.cyberia.club

matrix.cyberia.club
The following are managed by a script called acme.sh located at `/root/.acme.sh/`
  - matrix.cyberia.club
  - riot.cyberia.club

beet.cyberia.club
The following are managed by Caddy on the router which sits in front of beet. Ask j3s or fack about this.
  - cafe.cyberia.club

The following are managed by acme.sh
  - mumble.cyberia.club

systemctl stop mumble-server nginx
acme.sh renew mumble.cyberia.club
cp /root/.acme.sh/mumble.cyberia.club/mumble.cyberia.club.cer /etc/murmur/cert.pem
cp /root/.acme.sh/mumble.cyberia.club/mumble.cyberia.club.key /etc/murmur/key.pem
systemctl start mumble-server nginx

paimon.cyberia.club
The following are managed by something that sanine set up. Ask sanine about this.
  - git.cyberia.club

nyanjaro.cyberia.club
The following are updated by a cron job that nyaaori made. it calls /etc/letsencrypt/renew.sh
  - bridges.cyberia.club


```

### How to use acme.sh:

```
systemctl stop nginx ; acme.sh --renew --domain git.cyberia.club; systemctl start nginx ;
```

If you get an error like 
```
Please specify at least one validation method: '--webroot', '--standalone', '--apache', '--nginx' or '--dns' etc.
```

Then you must edit the config file, for example 

```
nano root@elliot:~/.acme.sh# nano cyberia.club/cyberia.club.conf
```

and change `Le_Webroot=''` to `Le_Webroot='no'`  inside the `<domain-name>/<domain-name.conf>` file.  [see: github issue](https://github.com/acmesh-official/acme.sh/issues/1172)


### certificate expiry alerts

The certificate expiry alerts are defined here: https://git.cyberia.club/services/ops-handbook/tree/rules/alerts.yml#n112

The `probe_ssl_earliest_cert_expiry` metric is written by the blackbox exporter, configured here: https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/prometheus/templates/prometheus.yml.j2#n82

## Notes

https://poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/