🌻Infrastructure And OperationsInventory

---
title: Inventory
---

Last Updated: March 2021

Overview

This diagram was created with app.diagrams.net/.
To edit it, download the diagram file and edit it with the app.diagrams.net/ web application, or you may run the application from source if you wish.

Physical Hosts

  • baikal

    • 69.61.110.118

    • cyberia's first rack server, installed in CyberWurx datacenter in Atlanta Georgia

    • j3s and forest are the only authorized support contacts on the CyberWurx portal right now

  • dredd

    • dynamic(ish) ip address

    • NOTE: dredd uses port 3217 for ssh. connect with ssh -p 3217 dredd.cyberia.club

    • olde desktop located in j3s's house

  • magnataur

    • dynamic(ish) ip address

    • NOTE: magnataur uses port 3216 for ssh. connect with ssh -p 3216 magnataur.cyberia.club

    • olde desktop located in j3s's house

Cloud Service Accounts

  • namecheap.com

    • fack's namecheap account is currently being used to manage DNS entires for:

      • cyberia.club

      • nullhex.com

      • capsul.org

    • all DNS updates are being done manually by j3s/forest.

    • conventions:

      • A records are named after hostnames & point to VMs / physical hosts

      • CNAMEs are named after the service & point to the A record of the host the service runs on

  • CyberWurx portal

    • Allows us to add reverse DNS entries for Capsuls

    • View metrics, get datacenter information, support tickets, etc

    • Right now j3s/forest are the only one who can log in / be authorized for support. Can add others though!

  • Capsul.org

    • Cyberia has an internal capsul account that we use. If you want access to this account, talk to j3s, vvesley, or forest.

Capsul

Most of cyberia's services run on Capsul, our Virtual Machine Management tool & service.

Ansible Managed Capsuls:
capsul-ay3yh10q2q  f1-xs  69.61.2.230  alpine311  Jun 20 2020 domechild.cyberia.club  (email server)
capsul-c04bbf593b  f1-s   69.61.2.246  alpine311  Jun 01 2020 raaz.cyberia.club       (NSHC / North Star Health Collective) 
capsul-pfgy2tthx9  f1-xs  69.61.2.167  alpine311  May 10 2020 legion.cyberia.club     (postgres for forge & others in the future)
capsul-id502edkg0  f1-xs  69.61.2.170  alpine311  Apr 01 2020 rosewater.cyberia.club  (cyberia forge, concourse server)
capsul-t6tfb2dh5p  f1-m   69.61.2.183  alpine311  May 10 2020 mothership.cyberia.club (prometheus & grafana & future log agg)
capsul-w6hsx09r7v  f1-xs  69.61.2.213  alpine311  Aug 20 2020 leckie.cyberia.club     (ansible bastion + build submitter)
capsul-f6crtfzx5c  f1-xs  69.61.2.218  alpine313  Mar 01 2021 comet.cyberia.club      (owncast server)
capsul-e1tfrw0637  f1-xs  69.61.2.201  alpine313  Mar 13 2021 kindred.cyberia.club    (mastodon server)

Legacy Capsuls:
capsul-yi9ffqbjly  f1-x   69.61.2.188  debian10   Apr 15 2020 btcpay.cyberia.club     (btcpay) 
cvm-lqj2x9nxic	   f1-l   69.61.2.190  debian10   Mar 07 2020 matrix.cyberia.club     (cyberia matrix) 
cvm-m1tjv0lljd	   f1-xs  69.61.2.178  debian10   Mar 10 2020 elliot.cyberia.club     (websites, this wiki, nullhex.com)
capsul-sbsmrkpgx7  f1-xs  69.61.38.199 debian10   Aug 01 2021 paimon.cyberia.club     (git.cyberia.club)

The Ansible Managed servers should have a user account for each user. The Legacy servers & baikal only have one user named cyberian, with everyone's keys authorized for that server.

Host Key Fingerprints

NOTE: you can control what kind of host key your ssh client will use like this:

ssh -o HostKeyAlgorithms=ssh-ed25519 example.cyberia.club

baikal.cyberia.club
  ECDSA    SHA256:85GTFfUpDDefcNcIROtFpuTiHC1j3iNU74aaKFO03+0
  ED25519  SHA256:v9MEa97wnmA75CyzQC5lW8nOI56LJ4jTmD2f68udK80

magnataur.cyberia.club
  NOTE:    magnataur uses port 3216 for ssh. connect with ssh -p 3216 magnataur.cyberia.club
  ECDSA    SHA256:kPOBn03CH176zrTlFDVmjFJpWi1OGHhkNCiK6stNn/0
  ED25519  SHA256:7M8ppVJ534Axz1ZXt6NheBxYkqY9UJ3AAmb9BmY9bYk

dredd.cyberia.club
  NOTE:    dredd uses port 3217 for ssh. connect with ssh -p 3217 dredd.cyberia.club
  ECDSA    SHA256:5157aYG7PT8Y0I4sTzlpQ5i/E3bq4aPF9T1P+xj+l9Q
  ED25519  SHA256:w6F0NXBoLCXG60yXoI3QhYGiLlPCr6YrK/OUSSDcmAw

mothership.cyberia.club
  ECDSA    SHA256:3XJG2fyaPDJWjnEOW3q2KiWg5qLV6hmEPczvp8GqhE0
  ED25519  SHA256:njIT2k1t6hHuOO0VjBNmHW1QSGN4GEqQQMj/BGpnBa0

domechild.cyberia.club
  ECDSA    SHA256:IQqTPv14u3dG62hS0q2Mr6pef6KwpjPKM2uVP+SK+qA
  ED25519  SHA256:3z5BI2ZEZjzDEh0B7a2GxgMa4faqA3Y6bQdGcQp4G88

rosewater.cyberia.club
  ECDSA    SHA256:dAbABreDUpV9AG7kChcx9S6+6f+fmnhqwwInqYoxcwU
  ED25519  SHA256:nT+ISIGV95MBKkIpcHTKo30lx4qRQ0Cpu1iM3w6+Sh0

legion.cyberia.club
  ECDSA    SHA256:EW9ydcgLg/pwoA0GPsI0VVeIBpnSi7aIHhvXOQBa+Xg
  ED25519  SHA256:cWLBFESOHrmVFrLRLjxrY4tcPmVRerJe1SB/+6tXSxAv

leckie.cyberia.club
  ECDSA    SHA256:KbzxzEKP21B0S3A/SKqqGmjiymnkk7byvoc6W4SxEwM
  ED25519  SHA256:M1QPflfIrsbhVlMaomvGQsr5AZS5YRkBHv+pnyI7bg4

raaz.cyberia.club
  ECDSA    SHA256:AJb0bZN2PTTm83zf5zI1IOEIVfeXUxQl/vTode/88jA
  ED25519  SHA256:zJv6E6lG4dAsqNmDHTO/qFVlTESKYq/KD29e8Nt/6j4

matrix.cyberia.club
  ECDSA    SHA256:VlRPAqLGxY4JUVhYirOVlfuDFtgTbaiw3x29xYizEeU
  ED25519  SHA256:BExhsVPNTp49jyJ6ezRf+Nn4TxPj8D9VZMhnjMABq6g

elliot.cyberia.club
  ECDSA    SHA256:/tsASDZ+MX519DC/Y7mHbV2CYCPnyMAbX1e0GHBOin0
  ED25519  SHA256:B9QNCnz57agsI40tMVU8UwyvZqMbz/p1ZNH5E1gL3io

comet.cyberia.club
  ECDSA    SHA256:UcDUCFd/U3F8ECG/RKxLbJRAAiMBSRKVKqDM0hmjwJ8
  ED25519  SHA256:SoOuSzKmpUd4x8Y8G32EAfQTY15agz1z7zJJCWdI8Tw

kindred.cyberia.club
  ECDSA    SHA256:M2oWKPgOqynag2nXrxnideac+r4Vb2tAsEz5ddEh/EM
  ED25519  SHA256:wCyMJYgoPAwlFKTXw41v/q8kypuand4fmhY4zsWdGlc

paimon.cyberia.club
  ECDSA    SHA256:IILubNkDwqzi1/6l5UCm24MvoxyxB6Y/m0BGWSmePZ0
  ED25519  SHA256:UsYrMq7nqxAND81Qzpgpzqz5ZxC/or6T0KIikM0tY9Q

Automation (Ansible)

The [Ops Handbook](git.cyberia.club/services/ops-handbook/) is still on the old git server, it is the main repo with the ansible inventory & playbooks.

Ansible bastion host/automation is on leckie.cyberia.club

Service Inventory

User-oriented Name

URL

Developer-oriented Name

Host

Deployment Code

Application Code

cyberia's matrix server

matrix.cyberia.club/

synapse

matrix.cyberia.club

ansible/roles/synapse

matrix-org/synapse

cyberia's matrix server

riot.cyberia.club/

element (used to be called riot)

matrix.cyberia.club

ansible/roles/riot

vector-im/element-web

cyberia's matrix server

N/A

postgres

matrix.cyberia.club

ansible/roles/postgresql

git.postgresql.org

cyberia's matrix server

matrix.cyberia.club/_synapse/metrics

matrix prometheus exporter

matrix.cyberia.club

TBD

matrix-org/synapse/metrics

nullhex email

nullhex.com/

alps

elliot.cyberia.club

TBD

~emersion/alps/

nullhex email

nullhex.com ports 25 & 587 (STARTTLS)

opensmtpd

domechild.cyberia.club

ansible/roles/opensmtpd

OpenSMTPD/OpenSMTPD

nullhex email

nullhex.com:993 (imap)

dovecot

domechild.cyberia.club

ansible/roles/dovecot

dovecot/core

nullhex email

N/A

rspamd

domechild.cyberia.club

TBD

rspamd/rspamd

capsul

capsul.org

capsul

baikal.cyberia.club

TBD

~forest/capsul-flask/

cyberia's git server

git.cyberia.club/

gitea (

paimon.cyberia.club

TBD

gitea

capsul / mailing lists

lists.cyberia.club/

postgres

legion.cyberia.club

TBD

git.postgresql.org

concourse (the new build server)

concourse.cyberia.club/

concourse

rosewater.cyberia.club

TBD

concourse/concourse

vault (build secrets manager)

N/A

vault

rosewater.cyberia.club

ansible/roles/concourse-vault

hashicorp/vault

cyberia's website

cyberia.club/

nginx static site

elliot.cyberia.club

TBD

services/website

prometheus

prometheus.cyberia.club/

prometheus

mothership.cyberia.club

rules & ansible/roles/prometheus

prometheus/prometheus

alertmanager

N/A

alertmanager

mothership.cyberia.club

same as prometheus

prometheus/alertmanager

grafana

grafana.cyberia.club/

grafana

mothership.cyberia.club

ansible/roles/grafana

grafana/grafana

Jackal

bot.j3s.sh

go-neb (matrix bot)

mothership.cyberia.club

TBD

matrix-org/go-neb (forest's fork)

Stream

stream.cyberia.club

owncast

comet.cyberia.club

TBD

owncast/owncast

Mastodon

social.cyberia.club

hometown

kindred.cyberia.club

TBD

hometown-fork/hometown

wiki / cyberdex

wiki.cyberia.club

Mycorrhiza

elliot.cyberia.club

TBD

bouncepaw/mycorrhiza

Metrics Inventory

We use Prometheus and Grafana to capture, store, and display metrics related to our services.

There are four main dashboards in grafana:

LetsEncrypt Certificate Inventory

For information on certificates which are managed by uacme automatically, see
git.cyberia.club/services/ops-handbook/tree/ansible/hosts
and the tls_certs variable in git.cyberia.club/services/ops-handbook/tree/ansible/group_vars

Certificates which are exceptions to the rule:

btcpay.cyberia.club
 - btcpay.cyberia.club certificate is automatically managed by btcpayserver-docker

elliot.cyberia.club
The following are managed by a script called acme.sh located at `/root/.acme.sh/`
 - capsul.org
 - www.capsul.org
 - nullhex.com
 - cyberia.club
 - git.cyberia.club

matrix.cyberia.club
The following are managed by a script called acme.sh located at `/root/.acme.sh/`
  - matrix.cyberia.club
  - riot.cyberia.club

magnataur.cyberia.club
The following are managed by Caddy on the router which sits in front of magnataur. Ask j3s or fack about this.
  - cafe.cyberia.club
  - mumble.cyberia.club

paimon.cyberia.club
The following are managed by something that sanine set up. Ask sanine about this.
  - git.cyberia.club

How to use acme.sh:

systemctl stop nginx ; acme.sh --renew --domain git.cyberia.club; systemctl start nginx ;

If you get an error like

Please specify at least one validation method: '--webroot', '--standalone', '--apache', '--nginx' or '--dns' etc.

Then you must edit the config file, for example

nano root@elliot:~/.acme.sh# nano cyberia.club/cyberia.club.conf

and change Le_Webroot='' to Le_Webroot='no' inside the <domain-name>/<domain-name.conf> file. [see: github issue](github.com/acmesh-official/acme.sh/issues/1172)

certificate expiry alerts

The certificate expiry alerts are defined here: git.cyberia.club/services/ops-handbook/tree/rules/alerts.yml#n112

The probe_ssl_earliest_cert_expiry metric is written by the blackbox exporter, configured here: git.cyberia.club/services/ops-handbook/tree/ansible/roles/prometheus/templates/prometheus.yml.j2#n82

Notes

poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/

Subhyphae