Please note that viewing attachments of hyphae is not supported in history for now. Get Mycomarkup source of this revision

🌻Infrastructure And Operations Inventory

---
title: Inventory
---

Last Updated: March 2021

Overview

Hypha infrastructure_and_operations/inventory/overview.png does not exist

This diagram was created with app.diagrams.net/.
To edit it, download the diagram file and edit it with the app.diagrams.net/ web application, or you may run the application from source if you wish.

Physical Hosts

- baikal
- 69.61.110.118
- cyberia's first rack server, installed in CyberWurx datacenter in Atlanta Georgia
- j3s and forest are the only authorized support contacts on the CyberWurx portal right now

- dredd
- dynamic(ish) ip address
- NOTE: dredd uses port 3217 for ssh. connect with ssh -p 3217 dredd.cyberia.club
- olde desktop located in j3s's house

- ~~magnataur~~
- ~~dynamic(ish) ip address~~
- ~~NOTE: magnataur uses port 3216 for ssh. connect with ssh -p 3216 magnataur.cyberia.club~~
- ~~olde desktop located in j3s's house~~

Cloud Service Accounts

- namecheap.com
- fack's namecheap account is currently being used to manage DNS entires for:
- cyberia.club
- nullhex.com
- capsul.org
- all DNS updates are being done manually by j3s/forest.
- conventions:
- A records are named after hostnames & point to VMs / physical hosts
- CNAMEs are named after the service & point to the A record of the host the service runs on

- CyberWurx portal
- Allows us to add reverse DNS entries for Capsuls
- View metrics, get datacenter information, support tickets, etc
- Right now j3s/forest are the only one who can log in / be authorized for support. Can add others though!

- Capsul.org
- Cyberia has an internal capsul account that we use. If you want access to this account, talk to j3s, vvesley, or forest.

Capsul

Most of cyberia's services run on Capsul, our Virtual Machine Management tool & service.

Ansible Managed Capsuls:
capsul-ay3yh10q2q  f1-xs  69.61.2.230  alpine311  Jun 20 2020 domechild.cyberia.club  (email server)
capsul-c04bbf593b  f1-s   69.61.2.246  alpine311  Jun 01 2020 raaz.cyberia.club       (NSHC / North Star Health Collective) 
capsul-pfgy2tthx9  f1-xs  69.61.2.167  alpine311  May 10 2020 legion.cyberia.club     (postgres for forge & others in the future)
capsul-id502edkg0  f1-xs  69.61.2.170  alpine311  Apr 01 2020 rosewater.cyberia.club  (cyberia forge, concourse server)
capsul-t6tfb2dh5p  f1-m   69.61.2.183  alpine311  May 10 2020 mothership.cyberia.club (prometheus & grafana & future log agg)
capsul-w6hsx09r7v  f1-xs  69.61.2.213  alpine311  Aug 20 2020 leckie.cyberia.club     (ansible bastion + build submitter)
capsul-f6crtfzx5c  f1-xs  69.61.2.218  alpine313  Mar 01 2021 comet.cyberia.club      (owncast server)
capsul-e1tfrw0637  f1-xs  69.61.2.201  alpine313  Mar 13 2021 kindred.cyberia.club    (mastodon server)

Legacy Capsuls:
capsul-yi9ffqbjly  f1-x   69.61.2.188  debian10   Apr 15 2020 btcpay.cyberia.club     (btcpay) 
cvm-lqj2x9nxic	   f1-l   69.61.2.190  debian10   Mar 07 2020 matrix.cyberia.club     (cyberia matrix) 
cvm-m1tjv0lljd	   f1-xs  69.61.2.178  debian10   Mar 10 2020 elliot.cyberia.club     (websites, this wiki, nullhex.com)
capsul-sbsmrkpgx7  f1-xs  69.61.38.199 debian10   Aug 01 2021 paimon.cyberia.club     (git.cyberia.club)

The Ansible Managed servers should have a user account for each user. The Legacy servers & baikal only have one user named cyberian, with everyone's keys authorized for that server.

Host Key Fingerprints

NOTE: you can control what kind of host key your ssh client will use like this:

ssh -o HostKeyAlgorithms=ssh-ed25519 example.cyberia.club

baikal.cyberia.club
  ECDSA    SHA256:85GTFfUpDDefcNcIROtFpuTiHC1j3iNU74aaKFO03+0
  ED25519  SHA256:v9MEa97wnmA75CyzQC5lW8nOI56LJ4jTmD2f68udK80

magnataur.cyberia.club
  NOTE:    magnataur uses port 3216 for ssh. connect with ssh -p 3216 magnataur.cyberia.club
  ECDSA    SHA256:kPOBn03CH176zrTlFDVmjFJpWi1OGHhkNCiK6stNn/0
  ED25519  SHA256:7M8ppVJ534Axz1ZXt6NheBxYkqY9UJ3AAmb9BmY9bYk

dredd.cyberia.club
  NOTE:    dredd uses port 3217 for ssh. connect with ssh -p 3217 dredd.cyberia.club
  ECDSA    SHA256:5157aYG7PT8Y0I4sTzlpQ5i/E3bq4aPF9T1P+xj+l9Q
  ED25519  SHA256:w6F0NXBoLCXG60yXoI3QhYGiLlPCr6YrK/OUSSDcmAw

mothership.cyberia.club
  ECDSA    SHA256:3XJG2fyaPDJWjnEOW3q2KiWg5qLV6hmEPczvp8GqhE0
  ED25519  SHA256:njIT2k1t6hHuOO0VjBNmHW1QSGN4GEqQQMj/BGpnBa0

domechild.cyberia.club
  ECDSA    SHA256:IQqTPv14u3dG62hS0q2Mr6pef6KwpjPKM2uVP+SK+qA
  ED25519  SHA256:3z5BI2ZEZjzDEh0B7a2GxgMa4faqA3Y6bQdGcQp4G88

rosewater.cyberia.club
  ECDSA    SHA256:dAbABreDUpV9AG7kChcx9S6+6f+fmnhqwwInqYoxcwU
  ED25519  SHA256:nT+ISIGV95MBKkIpcHTKo30lx4qRQ0Cpu1iM3w6+Sh0

legion.cyberia.club
  ECDSA    SHA256:EW9ydcgLg/pwoA0GPsI0VVeIBpnSi7aIHhvXOQBa+Xg
  ED25519  SHA256:cWLBFESOHrmVFrLRLjxrY4tcPmVRerJe1SB/+6tXSxAv

leckie.cyberia.club
  ECDSA    SHA256:KbzxzEKP21B0S3A/SKqqGmjiymnkk7byvoc6W4SxEwM
  ED25519  SHA256:M1QPflfIrsbhVlMaomvGQsr5AZS5YRkBHv+pnyI7bg4

raaz.cyberia.club
  ECDSA    SHA256:AJb0bZN2PTTm83zf5zI1IOEIVfeXUxQl/vTode/88jA
  ED25519  SHA256:zJv6E6lG4dAsqNmDHTO/qFVlTESKYq/KD29e8Nt/6j4

matrix.cyberia.club
  ECDSA    SHA256:VlRPAqLGxY4JUVhYirOVlfuDFtgTbaiw3x29xYizEeU
  ED25519  SHA256:BExhsVPNTp49jyJ6ezRf+Nn4TxPj8D9VZMhnjMABq6g

elliot.cyberia.club
  ECDSA    SHA256:/tsASDZ+MX519DC/Y7mHbV2CYCPnyMAbX1e0GHBOin0
  ED25519  SHA256:B9QNCnz57agsI40tMVU8UwyvZqMbz/p1ZNH5E1gL3io

comet.cyberia.club
  ECDSA    SHA256:UcDUCFd/U3F8ECG/RKxLbJRAAiMBSRKVKqDM0hmjwJ8
  ED25519  SHA256:SoOuSzKmpUd4x8Y8G32EAfQTY15agz1z7zJJCWdI8Tw

kindred.cyberia.club
  ECDSA    SHA256:M2oWKPgOqynag2nXrxnideac+r4Vb2tAsEz5ddEh/EM
  ED25519  SHA256:wCyMJYgoPAwlFKTXw41v/q8kypuand4fmhY4zsWdGlc

paimon.cyberia.club
  ECDSA    SHA256:IILubNkDwqzi1/6l5UCm24MvoxyxB6Y/m0BGWSmePZ0
  ED25519  SHA256:UsYrMq7nqxAND81Qzpgpzqz5ZxC/or6T0KIikM0tY9Q

Automation (Ansible)

The [Ops Handbook](git.cyberia.club/services/ops-handbook/) is still on the old git server, it is the main repo with the ansible inventory & playbooks.

Ansible bastion host/automation is on leckie.cyberia.club

Service Inventory

| User-oriented Name | URL | Developer-oriented Name | Host | Deployment Code | Application Code |
|-----------------------|-------------------------|-------------------------|-----------------------|--------------------|-------------------|
|cyberia's matrix server|matrix.cyberia.club/| synapse | matrix.cyberia.club | ansible/roles/synapse | matrix-org/synapse |
|cyberia's matrix server|riot.cyberia.club/| element (used to be called riot) | matrix.cyberia.club | ansible/roles/riot | vector-im/element-web |
|cyberia's matrix server| N/A | postgres | matrix.cyberia.club | ansible/roles/postgresql | git.postgresql.org |
|cyberia's matrix server| N/A | irc bridge to freenode | matrix.cyberia.club | TBD | matrix-org/matrix-appservice-irc |
|cyberia's matrix server| matrix.cyberia.club/_synapse/metrics | matrix prometheus exporter | matrix.cyberia.club | TBD | matrix-org/synapse/metrics |
|nullhex email | nullhex.com/ | alps | elliot.cyberia.club | TBD | ~emersion/alps/ |
|nullhex email | nullhex.com ports 25 & 587 (STARTTLS) | opensmtpd | domechild.cyberia.club | ansible/roles/opensmtpd | OpenSMTPD/OpenSMTPD |
|nullhex email | nullhex.com:993 (imap) | dovecot | domechild.cyberia.club | ansible/roles/dovecot | dovecot/core |
|nullhex email | N/A | rspamd | domechild.cyberia.club | TBD | rspamd/rspamd |
| capsul | capsul.org | capsul | baikal.cyberia.club | TBD | ~forest/capsul-flask/ |
| cyberia's git server | git.cyberia.club/ | gitea ( | paimon.cyberia.club | TBD | gitea |
| cyberias mailing lists | lists.cyberia.club/ | postgres | legion.cyberia.club | TBD | git.postgresql.org |
| concourse (the new build server) | concourse.cyberia.club/ | concourse | rosewater.cyberia.club | TBD | concourse/concourse |
| vault (build secrets manager) | N/A | vault | rosewater.cyberia.club | ansible/roles/concourse-vault | hashicorp/vault |
| cyberia's website | cyberia.club/ | nginx static site | elliot.cyberia.club | TBD | services/website |
| prometheus | prometheus.cyberia.club/ | prometheus | mothership.cyberia.club | rules & ansible/roles/prometheus | prometheus/prometheus |
| alertmanager | N/A | alertmanager | mothership.cyberia.club | same as prometheus | prometheus/alertmanager |
| grafana | grafana.cyberia.club/ | grafana | mothership.cyberia.club | ansible/roles/grafana | grafana/grafana |
| Jackal | bot.j3s.sh | go-neb (matrix bot) | mothership.cyberia.club | TBD | matrix-org/go-neb (forest's fork) |
| Stream | stream.cyberia.club | owncast | comet.cyberia.club | TBD | owncast/owncast |
| Mastodon | social.cyberia.club | hometown | kindred.cyberia.club | TBD | hometown-fork/hometown |
| wiki / cyberdex | wiki.cyberia.club | Mycorrhiza | elliot.cyberia.club | TBD | bouncepaw/mycorrhiza |

Metrics Inventory

We use Prometheus and Grafana to capture, store, and display metrics related to our services.

There are four main dashboards in grafana:

- https://grafana.cyberia.club/d/rYdddlPWk/node-exporter-full?orgId=1
- The prometheus node exporter runs on all of our servers and is managed by ansible.
- https://grafana.cyberia.club/d/wGgaPlciz/postgres-overview?orgId=1
- These metrics are reported by a postgres exporter which was installed manually on matrix.cyberia.club and legion.cyberia.club
- https://grafana.cyberia.club/d/000000026/cyberia-synapse?orgId=1
- These metrics are reported by synapse itself. If they are not working, it may be an issue with the reverse proxy in front of synapse.
- https://grafana.cyberia.club/d/jMw9xSRMz/capsul-stats?orgId=1
- these metrics are reported by git.cyberia.club/cyberia/libvirt_exporter (running on baikal). They are used to generate the graphs in the capsul.org web interface

LetsEncrypt Certificate Inventory

For information on certificates which are managed by uacme automatically, see
git.cyberia.club/services/ops-handbook/tree/ansible/hosts
and the tls_certs variable in git.cyberia.club/services/ops-handbook/tree/ansible/group_vars

Certificates which are exceptions to the rule:

btcpay.cyberia.club
 - btcpay.cyberia.club certificate is automatically managed by btcpayserver-docker

elliot.cyberia.club
The following are managed by a script called acme.sh located at `/root/.acme.sh/`
 - capsul.org
 - www.capsul.org
 - nullhex.com
 - cyberia.club
 - git.cyberia.club

matrix.cyberia.club
The following are managed by a script called acme.sh located at `/root/.acme.sh/`
  - matrix.cyberia.club
  - riot.cyberia.club

magnataur.cyberia.club
The following are managed by Caddy on the router which sits in front of magnataur. Ask j3s or fack about this.
  - cafe.cyberia.club
  - mumble.cyberia.club

paimon.cyberia.club
The following are managed by something that sanine set up. Ask sanine about this.
  - git.cyberia.club

How to use acme.sh:

systemctl stop nginx ; acme.sh --renew --domain git.cyberia.club; systemctl start nginx ;

If you get an error like

Please specify at least one validation method: '--webroot', '--standalone', '--apache', '--nginx' or '--dns' etc.

Then you must edit the config file, for example

nano root@elliot:~/.acme.sh# nano cyberia.club/cyberia.club.conf

and change Le_Webroot='' to Le_Webroot='no' inside the <domain-name>/<domain-name.conf> file. [see: github issue](github.com/acmesh-official/acme.sh/issues/1172)