🌻:Infrastructure And Operations/Inventory

---
title: Inventory
---

Last Updated: May 5, 2023

Loose Notes

Have something you want to document, but not sure where to put it or how to format it ? Please go right ahead and dump it into the pre-formatted text block below!!

Overview

This diagram was created with app.diagrams.net/.
To edit it, download the diagram file and edit it with the app.diagrams.net/ web application, or you may run the application from source if you wish.

Physical Hosts

• baikal

  • 69.61.110.118

  • cyberia's first rack server, installed in CyberWurx datacenter in Atlanta Georgia

  • j3s and forest are the only authorized support contacts on the CyberWurx portal right now

• beet

  • beet was recently rebuilt with a new motherboard and CPU. it has 4x 2GB HDDs

  • it's currently located at layer zero - ssh layerze.ro

Cloud Service Accounts

• namecheap.com

  • we use j3s's to manage DNS entries for:

    • cyberia.club

    • nullhex.com

    • capsul.org

  • all DNS updates are automated via git.cyberia.club/cyberia/dns

  • conventions:

    • A records are named after hostnames & point to VMs / physical hosts

    • CNAMEs are named after the service & point to the A record of the host the service runs on

• CyberWurx portal

  • Allows us to add reverse DNS entries for Capsuls

  • View metrics, get datacenter information, support tickets, etc

  • Right now j3s/forest are the only one who can log in / be authorized for support. Can add others though!

• Capsul.org

  • Cyberia has an internal capsul account that we use. If you want access to this account, talk to j3s, vvesley, or forest.

• njal.la

  • For redundancy and random domains, such as:

    • cyberia.top

    • cyberia.tube

    • cyberia.social

  • conventions: None. DNS updates are currently not automated. To update dns or to add / manage domains log into njal.la with treasurer@cyberia.club or use the api.

Capsul

Most of cyberia's services run on Capsul, our Virtual Machine Management tool & service.

Ansible Managed Capsuls:
capsul-ay3yh10q2q  f1-xs  69.61.2.230  alpine311  Jun 20 2020 domechild.cyberia.club  (email server)
capsul-c04bbf593b  f1-s   69.61.2.246  alpine311  Jun 01 2020 raaz.cyberia.club       (NSHC / North Star Health Collective) 
capsul-pfgy2tthx9  f1-xs  69.61.2.167  alpine311  May 10 2020 legion.cyberia.club     (postgres database)
capsul-t6tfb2dh5p  f1-m   69.61.2.183  alpine311  May 10 2020 mothership.cyberia.club (prometheus & grafana & future log agg)
capsul-w6hsx09r7v  f1-xs  69.61.2.213  alpine311  Aug 20 2020 leckie.cyberia.club     (ansible bastion + build submitter)
capsul-f6crtfzx5c  f1-xs  69.61.2.218  alpine313  Mar 01 2021 comet.cyberia.club      (owncast server)
cvm-lqj2x9nxic	   f1-l   69.61.2.190  debian10   Mar 07 2020 matrix.cyberia.club     (cyberia matrix) 
cvm-m1tjv0lljd	   f1-xs  69.61.2.178  debian10   Mar 10 2020 elliot.cyberia.club     (websites, this wiki, goatcounter)
capsul-sbsmrkpgx7  f1-xs  69.61.38.199 debian10   Aug 01 2021 paimon.cyberia.club     (git.cyberia.club)
capsul-nnzryhg9df  f1-xs  69.61.2.198  alpine315  Jun 15 2022 zicocapsul.cyberia.club (gancio server)
capsul-rat13644xq         69.61.38.209  alpine                stratus.cyberia.club    (vaultwarden)

The Ansible Managed servers should have a user account for each user.

Manually Managed Capsuls:

capsul-le2l50mbln  f1-x  69.61.2.225  Debian 11 (bullseye)   Apr 26 2022 pigsy.cyberia.club   (code-server for helloworld labs)

NOTE: currently capsul-le2l50mbln exists under forests capsul account 

capsul-hcdyq2fgdu  f1-xs  69.61.38.214  archlinux (nyaaori custom manjaro installation)  Sep 21 2021 nyanjaro.cyberia.club  (matrix bridges)

NOTE: the ssh username for nyanjaro is "user" not "cyberian". Only nyaaori, forest, and j3s have access so far.

Host Key Fingerprints

TO PRINT THIS INFO: ssh-keyscan localhost 2>/dev/null | ssh-keygen -l -f -

NOTE: you can control what kind of host key your ssh client will use like this:

ssh -o HostKeyAlgorithms=ssh-ed25519 example.cyberia.club

rathouse.layerze.ro
  RSA      SHA256:gqFzujf1ar0GONYLzJ6zLIHeLbaNocZYaLxM25R/Jx0  
  ECDSA    SHA256:cCo0LKZyGV2vJSdd5ePqCxhanDqeQHQvhgzUJkPn1qg 
  ED25519  SHA256:exR3rca77jgHeDx2VocoGmLvDMRfJQ4mRgGOrA59WOQ 

baikal.cyberia.club
  ECDSA    SHA256:85GTFfUpDDefcNcIROtFpuTiHC1j3iNU74aaKFO03+0
  ED25519  SHA256:v9MEa97wnmA75CyzQC5lW8nOI56LJ4jTmD2f68udK80

stratus.cyberia.club
2048 SHA256:b9a+191bE3yanyXefx1kchracNRXu/PWMXshXbxM5Vg stratus.cyberia.club (RSA)
256 SHA256:89yWa1Lc4pU48PLHQcaIiVwBT5u3GCQ72gPeXNAdqsk stratus.cyberia.club (ECDSA)
256 SHA256:PuhKpvdAz9xo8SZ8qY91IzESfKrTXEaIPkCttbUKPUI stratus.cyberia.club (ED25519)

lzwyse.layerze.ro
  RSA     SHA256:SgR01kDAJUNhIJY++0D93wR29fEIC/euBvQFzMl50VE
  ECDSA   SHA256:/jyQTwSfRAN1nIGeDpWmpXdZszPkSxn98ty2WR7ncoQ 
  ED25519 SHA256:4ua293kaTnW308/No42RxjnYxYeTPKiuYRPkYnuzwCA

ssh -p 2022 layerze.ro
beet.cyberia.club
  ECDSA    SHA256:kPOBn03CH176zrTlFDVmjFJpWi1OGHhkNCiK6stNn/0
  ED25519  SHA256:7M8ppVJ534Axz1ZXt6NheBxYkqY9UJ3AAmb9BmY9bYk

dredd.cyberia.club
  NOTE:    dredd uses port 3217 for ssh. connect with ssh -p 3217 dredd.cyberia.club
  ECDSA    SHA256:5157aYG7PT8Y0I4sTzlpQ5i/E3bq4aPF9T1P+xj+l9Q
  ED25519  SHA256:w6F0NXBoLCXG60yXoI3QhYGiLlPCr6YrK/OUSSDcmAw

mothership.cyberia.club
  ECDSA    SHA256:3XJG2fyaPDJWjnEOW3q2KiWg5qLV6hmEPczvp8GqhE0
  ED25519  SHA256:njIT2k1t6hHuOO0VjBNmHW1QSGN4GEqQQMj/BGpnBa0

domechild.cyberia.club
  ECDSA    SHA256:IQqTPv14u3dG62hS0q2Mr6pef6KwpjPKM2uVP+SK+qA
  ED25519  SHA256:3z5BI2ZEZjzDEh0B7a2GxgMa4faqA3Y6bQdGcQp4G88

legion.cyberia.club
  ECDSA    SHA256:EW9ydcgLg/pwoA0GPsI0VVeIBpnSi7aIHhvXOQBa+Xg
  ED25519  SHA256:cWLBFESOHrmVFrLRLjxrY4tcPmVRerJe1SB/+6tXSxAv

leckie.cyberia.club
  ECDSA    SHA256:KbzxzEKP21B0S3A/SKqqGmjiymnkk7byvoc6W4SxEwM
  ED25519  SHA256:M1QPflfIrsbhVlMaomvGQsr5AZS5YRkBHv+pnyI7bg4

raaz.cyberia.club
  ECDSA    SHA256:AJb0bZN2PTTm83zf5zI1IOEIVfeXUxQl/vTode/88jA
  ED25519  SHA256:zJv6E6lG4dAsqNmDHTO/qFVlTESKYq/KD29e8Nt/6j4

matrix.cyberia.club
  ECDSA    SHA256:VlRPAqLGxY4JUVhYirOVlfuDFtgTbaiw3x29xYizEeU
  ED25519  SHA256:BExhsVPNTp49jyJ6ezRf+Nn4TxPj8D9VZMhnjMABq6g

elliot.cyberia.club
  ECDSA    SHA256:/tsASDZ+MX519DC/Y7mHbV2CYCPnyMAbX1e0GHBOin0
  ED25519  SHA256:B9QNCnz57agsI40tMVU8UwyvZqMbz/p1ZNH5E1gL3io

comet.cyberia.club
  ECDSA    SHA256:UcDUCFd/U3F8ECG/RKxLbJRAAiMBSRKVKqDM0hmjwJ8
  ED25519  SHA256:SoOuSzKmpUd4x8Y8G32EAfQTY15agz1z7zJJCWdI8Tw

paimon.cyberia.club
  ECDSA    SHA256:IILubNkDwqzi1/6l5UCm24MvoxyxB6Y/m0BGWSmePZ0
  ED25519  SHA256:UsYrMq7nqxAND81Qzpgpzqz5ZxC/or6T0KIikM0tY9Q

nyanjaro.cyberia.club
  ECDSA    SHA256:hCpko+C9zSVNBC/76Ji6sjfMrj7f0+xnpLpqQEVK3oY
  ED25519  SHA256:AzT2aBvAJfD4JEq062I+NhPd5tVE0fi+m5ixnu5SnLs
  RSA      SHA256:taJdsRqtL2D80buYxcRgDbqitZ7tbuoX469Du6dN7yI    

pigsy.cyberia.club
  ECDSA    SHA256:fLOy8ZtOJg7SaQUMvpDI/33CYHKMJYyaq+53Q2kytEQ
  ED25519  SHA256:jbAnwc9nYJ95mjK3GS3mak9TM7hvACA19OGb/WciqyE
  RSA      SHA256:iDynajywy4D1pCrvIhG/i9WEHLgJh5CgaCqQTXUZfEw

zicocapsul.cyberia.club
  ECDSA    SHA256:GJZzDRtDZY5Bz4XTshWKn6GUb0BEjCQwyEBlr/Zg23c
  ED25519  SHA256:xUXIdvkFAhBtURyTLvoMBjYtuwRkpZHkEgGPyx0c9KI
  RSA      SHA256:fEOqNEy6/tNiYxtaRXg1I9YO0JPn4YRhqgqvAvJz8xI

Automation (Ansible)

The [Ops Handbook](git.cyberia.club/services/ops-handbook/) is still on the old git server, it is the main repo with the ansible inventory & playbooks.

Ansible bastion host/automation is on leckie.cyberia.club

Service Inventory

Metrics Inventory

We use Prometheus and Grafana to capture, store, and display metrics related to our services.

There are four main dashboards in grafana:

• Node Exporter

  • The prometheus node exporter runs on all of our servers and is managed by ansible.

• Postgres Overview

  • These metrics are reported by a postgres exporter which was installed manually on matrix.cyberia.club and legion.cyberia.club

• Cyberia Synapse

  • These metrics are reported by synapse itself. If they are not working, it may be an issue with the reverse proxy in front of synapse.

• Capsul Stats

  • these metrics are reported by git.cyberia.club/cyberia/libvirt_exporter (running on baikal). They are used to generate the graphs in the capsul.org web interface

LetsEncrypt Certificate Inventory

For information on certificates which are managed by uacme automatically, see
git.cyberia.club/services/ops-handbook/tree/ansible/hosts
and the tls_certs variable in git.cyberia.club/services/ops-handbook/tree/ansible/group_vars

Certificates which are exceptions to the rule:

btcpay.cyberia.club
 - btcpay.cyberia.club certificate is automatically managed by the caddy server running on elliot

baikal.cyberia.club
elliot.cyberia.club
The following are managed by a script called acme.sh located at `/root/.acme.sh/`
 - capsul.org
 - www.capsul.org
 - nullhex.com
 - cyberia.club
 - git.cyberia.club

matrix.cyberia.club
The following are managed by a script called acme.sh located at `/root/.acme.sh/`
  - matrix.cyberia.club
  - riot.cyberia.club

beet.cyberia.club
The following are managed by Caddy on the router which sits in front of beet. Ask j3s or fack about this.
  - cafe.cyberia.club

The following are managed by acme.sh
  - mumble.cyberia.club

systemctl stop mumble-server nginx
acme.sh renew mumble.cyberia.club
cp /root/.acme.sh/mumble.cyberia.club/mumble.cyberia.club.cer /etc/murmur/cert.pem
cp /root/.acme.sh/mumble.cyberia.club/mumble.cyberia.club.key /etc/murmur/key.pem
systemctl start mumble-server nginx

paimon.cyberia.club
The following are managed by something that sanine set up. Ask sanine about this.
  - git.cyberia.club

nyanjaro.cyberia.club
The following are updated by a cron job that nyaaori made. it calls /etc/letsencrypt/renew.sh
  - bridges.cyberia.club

How to use acme.sh:

systemctl stop nginx ; acme.sh --renew --domain git.cyberia.club; systemctl start nginx ;

If you get an error like

Please specify at least one validation method: '--webroot', '--standalone', '--apache', '--nginx' or '--dns' etc.

Then you must edit the config file, for example

nano root@elliot:~/.acme.sh# nano cyberia.club/cyberia.club.conf

and change Le_Webroot='' to Le_Webroot='no' inside the <domain-name>/<domain-name.conf> file. [see: github issue](github.com/acmesh-official/acme.sh/issues/1172)

certificate expiry alerts

The certificate expiry alerts are defined here: git.cyberia.club/services/ops-handbook/tree/rules/alerts.yml#n112

The probe_ssl_earliest_cert_expiry metric is written by the blackbox exporter, configured here: git.cyberia.club/services/ops-handbook/tree/ansible/roles/prometheus/templates/prometheus.yml.j2#n82

Notes

How to resize a capsul disk:

resize qemu image:
  # first, shut down the capsul, then:
  $ cd /tank/vm
  $ qemu-img resize cvm-lqj2x9nxic.qcow2 +50G
  $ virsh start cvm-lqj2x9nxic

you may have to resize the partition by hand internally, depending on the distro.

For alpine:
mothership:~# apk add e2fsprogs e2fsprogs-extra
...
mothership:~# resize2fs /dev/vda
resize2fs 1.46.6 (1-Feb-2023)
Filesystem at /dev/vda is mounted on /; on-line resizing required
old_desc_blocks = 2, new_desc_blocks = 7
The filesystem on /dev/vda is now 25690112 (4k) blocks long.

mothership:~# df -h
Filesystem                Size      Used Available Use% Mounted on
...
/dev/vda                 96.4G      6.3G     86.1G   7% /

poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/