[[]]--- title: Inventory --- Last Updated: March 2021 ## Overview img { infrastructure_and_operations/inventory/overview.png } This diagram was created with https://app.diagrams.net/. To edit it, download the [[infrastructure_and_operations/inventory/overview.drawio | diagram file]] and edit it with the https://app.diagrams.net/ web application, or you may run the application from [[https://github.com/jgraph/drawio | source]] if you wish. ## Physical Hosts * baikal ** 69.61.110.118 ** cyberia's first rack server, installed in CyberWurx datacenter in Atlanta Georgia ** j3s and forest are the only authorized support contacts on the CyberWurx portal right now * dredd ** dynamic(ish) ip address ** NOTE: dredd uses port 3217 for ssh. connect with `ssh -p 3217 dredd.cyberia.club` ** olde desktop located in j3s's house * ~~magnataur~~ ** ~~dynamic(ish) ip address~~ ** ~~NOTE: magnataur uses port 3216 for ssh. connect with `ssh -p 3216 magnataur.cyberia.club`~~ ** ~~olde desktop located in j3s's house~~ ## Cloud Service Accounts * namecheap.com ** fack's namecheap account is currently being used to manage DNS entires for: *** cyberia.club *** nullhex.com *** capsul.org ** all DNS updates are being done manually by j3s/forest. ** conventions: *** A records are named after hostnames & point to VMs / physical hosts *** CNAMEs are named after the service & point to the A record of the host the service runs on * CyberWurx portal ** Allows us to add reverse DNS entries for Capsuls ** View metrics, get datacenter information, support tickets, etc ** Right now j3s/forest are the only one who can log in / be authorized for support. Can add others though! * Capsul.org ** Cyberia has an internal capsul account that we use. If you want access to this account, talk to j3s, vvesley, or forest. ## Capsul Most of cyberia's services run on [[https://capsul.org | Capsul]], our Virtual Machine Management tool & service. ``` Ansible Managed Capsuls: capsul-ay3yh10q2q f1-xs 69.61.2.230 alpine311 Jun 20 2020 domechild.cyberia.club (email server) capsul-c04bbf593b f1-s 69.61.2.246 alpine311 Jun 01 2020 raaz.cyberia.club (NSHC / North Star Health Collective) capsul-pfgy2tthx9 f1-xs 69.61.2.167 alpine311 May 10 2020 legion.cyberia.club (postgres for forge & others in the future) capsul-id502edkg0 f1-xs 69.61.2.170 alpine311 Apr 01 2020 rosewater.cyberia.club (cyberia forge, concourse server) capsul-t6tfb2dh5p f1-m 69.61.2.183 alpine311 May 10 2020 mothership.cyberia.club (prometheus & grafana & future log agg) capsul-w6hsx09r7v f1-xs 69.61.2.213 alpine311 Aug 20 2020 leckie.cyberia.club (ansible bastion + build submitter) capsul-f6crtfzx5c f1-xs 69.61.2.218 alpine313 Mar 01 2021 comet.cyberia.club (owncast server) capsul-e1tfrw0637 f1-xs 69.61.2.201 alpine313 Mar 13 2021 kindred.cyberia.club (mastodon server) Legacy Capsuls: capsul-yi9ffqbjly f1-x 69.61.2.188 debian10 Apr 15 2020 btcpay.cyberia.club (btcpay) cvm-lqj2x9nxic f1-l 69.61.2.190 debian10 Mar 07 2020 matrix.cyberia.club (cyberia matrix) cvm-m1tjv0lljd f1-xs 69.61.2.178 debian10 Mar 10 2020 elliot.cyberia.club (websites, this wiki, nullhex.com) capsul-sbsmrkpgx7 f1-xs 69.61.38.199 debian10 Aug 01 2021 paimon.cyberia.club (git.cyberia.club) ``` The Ansible Managed servers should have a user account for each user. The Legacy servers & baikal only have one user named cyberian, with everyone's keys authorized for that server. #### Host Key Fingerprints **NOTE:** you can control what kind of host key your ssh client will use like this: `ssh -o HostKeyAlgorithms=ssh-ed25519 example.cyberia.club` ``` baikal.cyberia.club ECDSA SHA256:85GTFfUpDDefcNcIROtFpuTiHC1j3iNU74aaKFO03+0 ED25519 SHA256:v9MEa97wnmA75CyzQC5lW8nOI56LJ4jTmD2f68udK80 magnataur.cyberia.club NOTE: magnataur uses port 3216 for ssh. connect with ssh -p 3216 magnataur.cyberia.club ECDSA SHA256:kPOBn03CH176zrTlFDVmjFJpWi1OGHhkNCiK6stNn/0 ED25519 SHA256:7M8ppVJ534Axz1ZXt6NheBxYkqY9UJ3AAmb9BmY9bYk dredd.cyberia.club NOTE: dredd uses port 3217 for ssh. connect with ssh -p 3217 dredd.cyberia.club ECDSA SHA256:5157aYG7PT8Y0I4sTzlpQ5i/E3bq4aPF9T1P+xj+l9Q ED25519 SHA256:w6F0NXBoLCXG60yXoI3QhYGiLlPCr6YrK/OUSSDcmAw mothership.cyberia.club ECDSA SHA256:3XJG2fyaPDJWjnEOW3q2KiWg5qLV6hmEPczvp8GqhE0 ED25519 SHA256:njIT2k1t6hHuOO0VjBNmHW1QSGN4GEqQQMj/BGpnBa0 domechild.cyberia.club ECDSA SHA256:IQqTPv14u3dG62hS0q2Mr6pef6KwpjPKM2uVP+SK+qA ED25519 SHA256:3z5BI2ZEZjzDEh0B7a2GxgMa4faqA3Y6bQdGcQp4G88 rosewater.cyberia.club ECDSA SHA256:dAbABreDUpV9AG7kChcx9S6+6f+fmnhqwwInqYoxcwU ED25519 SHA256:nT+ISIGV95MBKkIpcHTKo30lx4qRQ0Cpu1iM3w6+Sh0 legion.cyberia.club ECDSA SHA256:EW9ydcgLg/pwoA0GPsI0VVeIBpnSi7aIHhvXOQBa+Xg ED25519 SHA256:cWLBFESOHrmVFrLRLjxrY4tcPmVRerJe1SB/+6tXSxAv leckie.cyberia.club ECDSA SHA256:KbzxzEKP21B0S3A/SKqqGmjiymnkk7byvoc6W4SxEwM ED25519 SHA256:M1QPflfIrsbhVlMaomvGQsr5AZS5YRkBHv+pnyI7bg4 raaz.cyberia.club ECDSA SHA256:AJb0bZN2PTTm83zf5zI1IOEIVfeXUxQl/vTode/88jA ED25519 SHA256:zJv6E6lG4dAsqNmDHTO/qFVlTESKYq/KD29e8Nt/6j4 matrix.cyberia.club ECDSA SHA256:VlRPAqLGxY4JUVhYirOVlfuDFtgTbaiw3x29xYizEeU ED25519 SHA256:BExhsVPNTp49jyJ6ezRf+Nn4TxPj8D9VZMhnjMABq6g elliot.cyberia.club ECDSA SHA256:/tsASDZ+MX519DC/Y7mHbV2CYCPnyMAbX1e0GHBOin0 ED25519 SHA256:B9QNCnz57agsI40tMVU8UwyvZqMbz/p1ZNH5E1gL3io comet.cyberia.club ECDSA SHA256:UcDUCFd/U3F8ECG/RKxLbJRAAiMBSRKVKqDM0hmjwJ8 ED25519 SHA256:SoOuSzKmpUd4x8Y8G32EAfQTY15agz1z7zJJCWdI8Tw kindred.cyberia.club ECDSA SHA256:M2oWKPgOqynag2nXrxnideac+r4Vb2tAsEz5ddEh/EM ED25519 SHA256:wCyMJYgoPAwlFKTXw41v/q8kypuand4fmhY4zsWdGlc paimon.cyberia.club ECDSA SHA256:IILubNkDwqzi1/6l5UCm24MvoxyxB6Y/m0BGWSmePZ0 ED25519 SHA256:UsYrMq7nqxAND81Qzpgpzqz5ZxC/or6T0KIikM0tY9Q ``` ## Automation (Ansible) The [Ops Handbook](https://git.cyberia.club/services/ops-handbook/) is still on the old git server, it is the main repo with the ansible inventory & playbooks. Ansible bastion host/automation is on leckie.cyberia.club ## Service Inventory table { ! User-oriented Name ! URL ! Developer-oriented Name ! Host ! Deployment Code ! Application Code ! |cyberia's matrix server|https://matrix.cyberia.club/| synapse | matrix.cyberia.club | [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/synapse | ansible/roles/synapse ]] | [[ https://github.com/matrix-org/synapse | matrix-org/synapse ]] | |cyberia's matrix server|https://riot.cyberia.club/| element (used to be called riot) | matrix.cyberia.club | [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/riot | ansible/roles/riot ]] | [[ https://github.com/vector-im/element-web | vector-im/element-web ]] | |cyberia's matrix server| N/A | postgres | matrix.cyberia.club | [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/postgresql | ansible/roles/postgresql ]] | [[ https://git.postgresql.org/git/postgresql.git | git.postgresql.org ]] | |cyberia's matrix server| N/A | irc bridge to freenode | matrix.cyberia.club | TBD | [[ https://github.com/matrix-org/matrix-appservice-irc | matrix-org/matrix-appservice-irc ]] | |cyberia's matrix server| https://matrix.cyberia.club/_synapse/metrics | matrix prometheus exporter | matrix.cyberia.club | TBD | [[ https://github.com/matrix-org/synapse/tree/develop/synapse/metrics | matrix-org/synapse/metrics ]] | |nullhex email | https://nullhex.com/ | alps | elliot.cyberia.club | TBD | [[ https://sr.ht/~emersion/alps/ | ~emersion/alps/ ]] | |nullhex email | nullhex.com ports 25 & 587 (STARTTLS) | opensmtpd | domechild.cyberia.club | [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/opensmtpd | ansible/roles/opensmtpd ]] | [[ https://github.com/OpenSMTPD/OpenSMTPD/ | OpenSMTPD/OpenSMTPD ]] | |nullhex email | nullhex.com:993 (imap) | dovecot | domechild.cyberia.club | [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/dovecot | ansible/roles/dovecot ]] | [[ https://github.com/dovecot/core | dovecot/core ]] | |nullhex email | N/A | rspamd | domechild.cyberia.club | TBD | [[ https://github.com/rspamd/rspamd | rspamd/rspamd ]] | | capsul | https://capsul.org | capsul | baikal.cyberia.club | TBD | [[ https://giit.cyberia.club/~forest/capsul-flask/ | ~forest/capsul-flask/ ]] | | cyberia's git server | https://git.cyberia.club/ | gitea ( | paimon.cyberia.club | TBD | [[ https://gitea.com/ | gitea ]] | | cyberias mailing lists | https://lists.cyberia.club/ | postgres | legion.cyberia.club | TBD | [[ https://git.postgresql.org/git/postgresql.git | git.postgresql.org ]] | | concourse (the new build server) | https://concourse.cyberia.club/ | concourse | rosewater.cyberia.club | TBD | [[ https://github.com/concourse/concourse | concourse/concourse ]] | | vault (build secrets manager) | N/A | vault | rosewater.cyberia.club | [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/concourse-vault/files | ansible/roles/concourse-vault ]] | [[ https://github.com/hashicorp/vault | hashicorp/vault ]] | | cyberia's website | https://cyberia.club/ | nginx static site | elliot.cyberia.club | TBD | [[ https://git.cyberia.club/services/website/ | services/website ]] | | prometheus | https://prometheus.cyberia.club/ | prometheus | mothership.cyberia.club | [[ https://git.cyberia.club/services/ops-handbook/tree/rules | rules ]] & [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/prometheus | ansible/roles/prometheus ]] | [[ https://github.com/prometheus/prometheus | prometheus/prometheus ]] | | alertmanager | N/A | alertmanager | mothership.cyberia.club | same as prometheus | [[ https://github.com/prometheus/alertmanager | prometheus/alertmanager ]] | | grafana | https://grafana.cyberia.club/ | grafana | mothership.cyberia.club | [[ https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/grafana | ansible/roles/grafana ]] | [[ https://github.com/grafana/grafana | grafana/grafana ]] | | Jackal | https://bot.j3s.sh | go-neb (matrix bot) | mothership.cyberia.club | TBD | [[ https://giit.cyberia.club/~forest/go-neb/log/forest-feature-rebase-2 | matrix-org/go-neb (forest's fork) ]] | | Stream | https://stream.cyberia.club | owncast | comet.cyberia.club | TBD | [[ https://github.com/owncast/owncast | owncast/owncast ]] | | Mastodon | https://social.cyberia.club | hometown | kindred.cyberia.club | TBD | [[https://github.com/hometown-fork/hometown | hometown-fork/hometown]] | | wiki / cyberdex | https://wiki.cyberia.club | Mycorrhiza | elliot.cyberia.club | TBD | [[ https://github.com/bouncepaw/mycorrhiza | bouncepaw/mycorrhiza]] | } ## Metrics Inventory We use Prometheus and Grafana to capture, store, and display metrics related to our services. There are four main dashboards in grafana: * [[ Node Exporter | https://grafana.cyberia.club/d/rYdddlPWk/node-exporter-full?orgId=1 ]] ** The prometheus node exporter runs on all of our servers and is [[ https://git.cyberia.club/cyberia/ops-handbook/src/branch/master/ansible/roles/common/tasks/main.yml#L66 | managed by ansible ]]. * [[ Postgres Overview | https://grafana.cyberia.club/d/wGgaPlciz/postgres-overview?orgId=1 ]] ** These metrics are reported by a postgres exporter which was installed manually on matrix.cyberia.club and legion.cyberia.club * [[ Cyberia Synapse | https://grafana.cyberia.club/d/000000026/cyberia-synapse?orgId=1 ]] ** These metrics are reported by synapse itself. If they are not working, it may be an issue with the reverse proxy in front of synapse. * [[ Capsul Stats | https://grafana.cyberia.club/d/jMw9xSRMz/capsul-stats?orgId=1 ]] ** these metrics are reported by https://git.cyberia.club/cyberia/libvirt_exporter (running on baikal). They are used to generate the graphs in the capsul.org web interface ## LetsEncrypt Certificate Inventory For information on certificates which are managed by uacme automatically, see https://git.cyberia.club/services/ops-handbook/tree/ansible/hosts and the `tls_certs` variable in https://git.cyberia.club/services/ops-handbook/tree/ansible/group_vars Certificates which are exceptions to the rule: ``` btcpay.cyberia.club - btcpay.cyberia.club certificate is automatically managed by btcpayserver-docker elliot.cyberia.club The following are managed by a script called acme.sh located at `/root/.acme.sh/` - capsul.org - www.capsul.org - nullhex.com - cyberia.club - git.cyberia.club matrix.cyberia.club The following are managed by a script called acme.sh located at `/root/.acme.sh/` - matrix.cyberia.club - riot.cyberia.club magnataur.cyberia.club The following are managed by Caddy on the router which sits in front of magnataur. Ask j3s or fack about this. - cafe.cyberia.club - mumble.cyberia.club paimon.cyberia.club The following are managed by something that sanine set up. Ask sanine about this. - git.cyberia.club ``` ### How to use acme.sh: ``` systemctl stop nginx ; acme.sh --renew --domain git.cyberia.club; systemctl start nginx ; ``` If you get an error like ``` Please specify at least one validation method: '--webroot', '--standalone', '--apache', '--nginx' or '--dns' etc. ``` Then you must edit the config file, for example ``` nano root@elliot:~/.acme.sh# nano cyberia.club/cyberia.club.conf ``` and change `Le_Webroot=''` to `Le_Webroot='no'` inside the `/` file. [see: github issue](https://github.com/acmesh-official/acme.sh/issues/1172) ### certificate expiry alerts The certificate expiry alerts are defined here: https://git.cyberia.club/services/ops-handbook/tree/rules/alerts.yml#n112 The `probe_ssl_earliest_cert_expiry` metric is written by the blackbox exporter, configured here: https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/prometheus/templates/prometheus.yml.j2#n82 ## Notes https://poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/