Diff of Infrastructure And Operations/Inventory at 19a8e23

diff --git a/infrastructure_and_operations/inventory.myco b/infrastructure_and_operations/inventory.myco
new file mode 100644
index 0000000..53323c3
--- /dev/null
+++ b/infrastructure_and_operations/inventory.myco
@@ -0,0 +1,233 @@
+---
+title: Inventory
+---
+
+Last Updated: March 2021
+
+## Overview
+
+![](inventory-overview.png)
+
+This diagram was created with https://app.diagrams.net/.
+To edit it, download the <a download href="inventory-overview.drawio">diagram file</a> and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish.
+
+## Physical Hosts
+
+ - baikal
+   - 69.61.110.118
+   - cyberia's first rack server, installed in  CyberWurx datacenter in Atlanta Georgia
+   - j3s is the only authorized support contact right now and the only one who can log into the CyberWurx portal
+
+ - dredd
+   - dynamic(ish) ip address
+   - NOTE: dredd uses port 3217 for ssh. connect with `ssh -p 3217 dredd.cyberia.club`
+   - olde desktop located in j3s's house
+
+ - magnataur
+   - dynamic(ish) ip address
+   - NOTE: magnataur uses port 3216 for ssh. connect with `ssh -p 3216 magnataur.cyberia.club`
+   - olde desktop located in j3s's house
+
+## Cloud Service Accounts
+
+  - namecheap
+    - fack's namecheap account is currently being used to manage DNS entires for:
+      - cyberia.club
+      - nullhex.com
+      - capsul.org
+    - all DNS updates are being done manually by j3s.
+    - conventions:
+      - A records are named after hostnames & point to VMs / physical hosts
+      - CNAMEs are named after the service & point to the A record of the host the service runs on
+
+  - CyberWurx portal
+    - Allows us to add reverse DNS entries for Capsuls
+    - View metrics, get datacenter information, support tickets, etc
+    - Right now j3s is the only one who can log in / be authorized for support. Can add others though!
+
+## Capsul
+
+Most of cyberia's services run on [Capsul](https://capsul.org), our Virtual Machine Management tool & service.
+
+```
+Ansible Managed Capsuls:
+capsul-ay3yh10q2q  f1-xs  69.61.2.230  alpine311  Jun 20 2020 domechild.cyberia.club  (email server)
+capsul-c04bbf593b  f1-s   69.61.2.246  alpine311  Jun 01 2020 raaz.cyberia.club       (NSHC / North Star Health Collective) 
+capsul-pfgy2tthx9  f1-xs  69.61.2.167  alpine311  May 10 2020 legion.cyberia.club     (postgres for forge & others in the future)
+capsul-id502edkg0  f1-xs  69.61.2.170  alpine311  Apr 01 2020 rosewater.cyberia.club  (cyberia forge)
+capsul-t6tfb2dh5p  f1-m   69.61.2.183  alpine311  May 10 2020 mothership.cyberia.club (prometheus & grafana & future logg agg)
+capsul-w6hsx09r7v  f1-xs  69.61.2.213  alpine311  Aug 20 2020 leckie.cyberia.club     (ansible bastion + build submitter)
+capsul-f6crtfzx5c  f1-xs  69.61.2.218  alpine313  Mar 01 2021 comet.cyberia.club      (owncast server)
+capsul-e1tfrw0637  f1-xs  69.61.2.201  alpine313  Mar 13 2021 kindred.cyberia.club    (mastodon server)
+
+Legacy Capsuls:
+capsul-yi9ffqbjly  f1-x   69.61.2.188  debian10   Apr 15 2020 btcpay.cyberia.club     (btcpay) 
+cvm-lqj2x9nxic	   f1-l   69.61.2.190  debian10   Mar 07 2020 matrix.cyberia.club     (cyberia matrix) 
+cvm-m1tjv0lljd	   f1-xs  69.61.2.178  debian10   Mar 10 2020 elliot.cyberia.club     (websites & git.cyberia.club, nullhex.com)
+```
+
+The Ansible Managed servers should have a user account for each user. The Legacy servers & baikal only have one user named cyberian, with everyone's keys authorized for that server.
+
+Contact j3s, forest, or vvesley for more information on cyberia's capsul account.
+
+#### Host Key Fingerprints
+
+**NOTE:** you can control what kind of host key your ssh client will use like this: 
+
+`ssh -o HostKeyAlgorithms=ssh-ed25519 example.cyberia.club`
+
+```
+baikal.cyberia.club
+  ECDSA    SHA256:85GTFfUpDDefcNcIROtFpuTiHC1j3iNU74aaKFO03+0
+  ED25519  SHA256:v9MEa97wnmA75CyzQC5lW8nOI56LJ4jTmD2f68udK80
+
+magnataur.cyberia.club
+  NOTE:    magnataur uses port 3216 for ssh. connect with ssh -p 3216 magnataur.cyberia.club
+  ECDSA    SHA256:kPOBn03CH176zrTlFDVmjFJpWi1OGHhkNCiK6stNn/0
+  ED25519  SHA256:7M8ppVJ534Axz1ZXt6NheBxYkqY9UJ3AAmb9BmY9bYk
+
+dredd.cyberia.club
+  NOTE:    dredd uses port 3217 for ssh. connect with ssh -p 3217 dredd.cyberia.club
+  ECDSA    SHA256:5157aYG7PT8Y0I4sTzlpQ5i/E3bq4aPF9T1P+xj+l9Q
+  ED25519  SHA256:w6F0NXBoLCXG60yXoI3QhYGiLlPCr6YrK/OUSSDcmAw
+
+mothership.cyberia.club
+  ECDSA    SHA256:3XJG2fyaPDJWjnEOW3q2KiWg5qLV6hmEPczvp8GqhE0
+  ED25519  SHA256:njIT2k1t6hHuOO0VjBNmHW1QSGN4GEqQQMj/BGpnBa0
+
+domechild.cyberia.club
+  ECDSA    SHA256:IQqTPv14u3dG62hS0q2Mr6pef6KwpjPKM2uVP+SK+qA
+  ED25519  SHA256:3z5BI2ZEZjzDEh0B7a2GxgMa4faqA3Y6bQdGcQp4G88
+
+rosewater.cyberia.club
+  ECDSA    SHA256:dAbABreDUpV9AG7kChcx9S6+6f+fmnhqwwInqYoxcwU
+  ED25519  SHA256:nT+ISIGV95MBKkIpcHTKo30lx4qRQ0Cpu1iM3w6+Sh0
+
+legion.cyberia.club
+  ECDSA    SHA256:EW9ydcgLg/pwoA0GPsI0VVeIBpnSi7aIHhvXOQBa+Xg
+  ED25519  SHA256:cWLBFESOHrmVFrLRLjxrY4tcPmVRerJe1SB/+6tXSxAv
+
+leckie.cyberia.club
+  ECDSA    SHA256:KbzxzEKP21B0S3A/SKqqGmjiymnkk7byvoc6W4SxEwM
+  ED25519  SHA256:M1QPflfIrsbhVlMaomvGQsr5AZS5YRkBHv+pnyI7bg4
+
+raaz.cyberia.club
+  ECDSA    SHA256:AJb0bZN2PTTm83zf5zI1IOEIVfeXUxQl/vTode/88jA
+  ED25519  SHA256:zJv6E6lG4dAsqNmDHTO/qFVlTESKYq/KD29e8Nt/6j4
+
+matrix.cyberia.club
+  ECDSA    SHA256:VlRPAqLGxY4JUVhYirOVlfuDFtgTbaiw3x29xYizEeU
+  ED25519  SHA256:BExhsVPNTp49jyJ6ezRf+Nn4TxPj8D9VZMhnjMABq6g
+
+elliot.cyberia.club
+  ECDSA    SHA256:/tsASDZ+MX519DC/Y7mHbV2CYCPnyMAbX1e0GHBOin0
+  ED25519  SHA256:B9QNCnz57agsI40tMVU8UwyvZqMbz/p1ZNH5E1gL3io
+
+btcpay.cyberia.club
+  ECDSA    SHA256:CdqdUvG0Obfdq9kkeQSETVhSJO2oCAdEAjDCydQWcDI
+  ED25519  SHA256:WcjrJtvev3+rAu98TFGJoxx/CytLCg+GfEXBMVOl5Hw
+
+comet.cyberia.club
+  ECDSA    SHA256:UcDUCFd/U3F8ECG/RKxLbJRAAiMBSRKVKqDM0hmjwJ8
+  ED25519  SHA256:SoOuSzKmpUd4x8Y8G32EAfQTY15agz1z7zJJCWdI8Tw
+
+kindred.cyberia.club
+  ECDSA    SHA256:M2oWKPgOqynag2nXrxnideac+r4Vb2tAsEz5ddEh/EM
+  ED25519  SHA256:wCyMJYgoPAwlFKTXw41v/q8kypuand4fmhY4zsWdGlc
+```
+
+
+## Automation (Ansible)
+
+The [Ops Handbook](https://git.cyberia.club/services/ops-handbook/) is still on the old git server, it is the main repo with the ansible inventory & playbooks.
+
+Ansible bastion host/automation is on leckie.cyberia.club
+
+## Service Inventory
+
+|  User-oriented Name   |         URL             | Developer-oriented Name |        Host           |    Deployment Code |  Application Code |
+|-----------------------|-------------------------|-------------------------|-----------------------|--------------------|-------------------|
+|cyberia's matrix server|https://matrix.cyberia.club/|         synapse         |  matrix.cyberia.club  | [ansible/roles/synapse](https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/synapse) | [matrix-org/synapse](https://github.com/matrix-org/synapse) |
+|cyberia's matrix server|https://riot.cyberia.club/| element (used to be called riot) |  matrix.cyberia.club  | [ansible/roles/riot](https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/riot) | [vector-im/element-web](https://github.com/vector-im/element-web) |
+|cyberia's matrix server| N/A                       |  postgres |  matrix.cyberia.club  | [ansible/roles/postgresql](https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/postgresql) |  [git.postgresql.org](https://git.postgresql.org/git/postgresql.git) |
+|cyberia's matrix server| N/A                       |  irc bridge to freenode    |  matrix.cyberia.club  |     TBD    | [matrix-org/matrix-appservice-irc](https://github.com/matrix-org/matrix-appservice-irc) |
+|cyberia's matrix server| https://matrix.cyberia.club/_synapse/metrics | matrix prometheus exporter |  matrix.cyberia.club  |     TBD    | [matrix-org/synapse/metrics](https://github.com/matrix-org/synapse/tree/develop/synapse/metrics) |
+|nullhex email          | https://nullhex.com/      | alps |  elliot.cyberia.club  |     TBD    | [~emersion/alps/](https://sr.ht/~emersion/alps/) |
+|nullhex email          | nullhex.com ports 25 & 587 (STARTTLS) | opensmtpd |  domechild.cyberia.club  |     [ansible/roles/opensmtpd](https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/opensmtpd)    | [OpenSMTPD/OpenSMTPD](https://github.com/OpenSMTPD/OpenSMTPD/) |
+|nullhex email          | nullhex.com:993 (imap)   | dovecot |  domechild.cyberia.club  |     [ansible/roles/dovecot](https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/dovecot)    | [dovecot/core](https://github.com/dovecot/core) |
+|nullhex email          | N/A                       | rspamd  |  domechild.cyberia.club  |     TBD    | [rspamd/rspamd](https://github.com/rspamd/rspamd) |
+| capsul                | https://capsul.org       | capsul | baikal.cyberia.club | TBD       | [~forest/capsul-flask/](https://giit.cyberia.club/~forest/capsul-flask/) |
+| forge (cyberia's git server) |https://forge.cyberia.club/ | sourcehut | rosewater.cyberia.club |      [see the ops-handbook](https://git.cyberia.club/services/ops-handbook/tree/docs/forge.md)       | [~sircmpwn/sourcehut](https://sr.ht/~sircmpwn/sourcehut/) |
+| forge (cyberia's git server) |N/A                | postgres | legion.cyberia.club |     TBD    | [git.postgresql.org](https://git.postgresql.org/git/postgresql.git) |
+| concourse (the new build server) | https://concourse.cyberia.club/ | concourse | rosewater.cyberia.club | TBD | [concourse/concourse](https://github.com/concourse/concourse) |
+| vault (build secrets manager) | N/A | vault | rosewater.cyberia.club | [ansible/roles/concourse-vault](https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/concourse-vault/files) | [hashicorp/vault](https://github.com/hashicorp/vault) |
+| cyberia's website     | https://cyberia.club/    | nginx static site   | elliot.cyberia.club |     TBD    | [services/website](https://git.cyberia.club/services/website/) |
+| the old git server    | https://git.cyberia.club/ | cgit   | elliot.cyberia.club |     TBD    | [git.zx2c4.com/cgit](https://git.zx2c4.com/cgit/) |
+| prometheus   | https://prometheus.cyberia.club/ | prometheus   | mothership.cyberia.club | [rules](https://git.cyberia.club/services/ops-handbook/tree/rules) & [ansible/roles/prometheus](https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/prometheus) | [prometheus/prometheus](https://github.com/prometheus/prometheus) |
+| alertmanager   | N/A                            | alertmanager   | mothership.cyberia.club | same as prometheus | [prometheus/alertmanager](https://github.com/prometheus/alertmanager) |
+| grafana          | https://grafana.cyberia.club/ | grafana   | mothership.cyberia.club | [ansible/roles/grafana](https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/grafana) | [grafana/grafana](https://github.com/grafana/grafana) |
+| Jackal           | https://bot.j3s.sh            |   go-neb (matrix bot)  | mothership.cyberia.club | TBD | [matrix-org/go-neb (forest's fork)](https://giit.cyberia.club/~forest/go-neb/log/forest-feature-rebase-2) |
+| Stream           | https://stream.cyberia.club            | owncast   | comet.cyberia.club | TBD | [owncast/owncast](https://github.com/owncast/owncast) |
+| Mastodon           | https://social.cyberia.club            | hometown   | kindred.cyberia.club | TBD | [hometown-fork/hometown](https://github.com/hometown-fork/hometown) |
+
+
+## LetsEncrypt Certificate Inventory
+
+For information on certificates which are managed by uacme automatically, see 
+https://git.cyberia.club/services/ops-handbook/tree/ansible/hosts
+and the `tls_certs` variable in https://git.cyberia.club/services/ops-handbook/tree/ansible/group_vars
+
+Certificates which are exceptions to the rule: 
+
+```
+btcpay.cyberia.club
+ - btcpay.cyberia.club certificate is automatically managed by btcpayserver-docker
+
+elliot.cyberia.club
+The following are managed by a script called acme.sh located at `/root/.acme.sh/`
+ - capsul.org
+ - www.capsul.org
+ - nullhex.com
+ - cyberia.club
+ - git.cyberia.club
+
+matrix.cyberia.club
+The following are managed by a script called acme.sh located at `/root/.acme.sh/`
+  - matrix.cyberia.club
+  - riot.cyberia.club
+
+magnataur.cyberia.club
+The following are managed by Caddy on the router which sits in front of magnataur. Ask j3s or fack about this.
+  - cafe.cyberia.club
+  - mumble.cyberia.club
+```
+
+### How to use acme.sh:
+
+```
+systemctl stop nginx ; acme.sh --renew --domain git.cyberia.club; systemctl start nginx ;
+```
+
+If you get an error like 
+```
+Please specify at least one validation method: '--webroot', '--standalone', '--apache', '--nginx' or '--dns' etc.
+```
+
+Then you must edit the config file, for example 
+
+```
+nano root@elliot:~/.acme.sh# nano cyberia.club/cyberia.club.conf
+```
+
+and change `Le_Webroot=''` to `Le_Webroot='no'`  inside the `<domain-name>/<domain-name.conf>` file.  [see: github issue](https://github.com/acmesh-official/acme.sh/issues/1172)
+
+
+### certificate expiry alerts
+
+The certificate expiry alerts are defined here: https://git.cyberia.club/services/ops-handbook/tree/rules/alerts.yml#n112
+
+The `probe_ssl_earliest_cert_expiry` metric is written by the blackbox exporter, configured here: https://git.cyberia.club/services/ops-handbook/tree/ansible/roles/prometheus/templates/prometheus.yml.j2#n82
+
+## Notes
+
+https://poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/
\ No newline at end of file