Diff of Services/Forge/Troubleshooting at 0c3aabb

diff --git a/services/forge/troubleshooting.myco b/services/forge/troubleshooting.myco
new file mode 100644
index 0000000..b8d0129
--- /dev/null
+++ b/services/forge/troubleshooting.myco
@@ -0,0 +1,81 @@
+
+## **Users attempting to login via the web client receive nothing but a big "500" **
+
+__History__
+
+To put it in terms of the log retrieved via a CLI, the root cause of this issue was found on 2023-12-23 to be:
+
+```
+root@paimon:/etc/docker-compose/gitea# docker logs -n 1000 -f gitea | grep -v router
+2023/12/23 16:40:18 ...ers/web/repo/repo.go:434:Download() [E] archiver.Await: archiver.StartArchive: write /data/gitea/queues/common/000445.log: no space left on device
+2023/12/23 16:43:32 ...ces/mirror/mirror.go:100:Update() [E] MirrorsIterate: write /data/gitea/queues/common/000445.log: no space left on device
+2023/12/23 16:43:32 ...rvices/cron/tasks.go:129:func2() [E] CreateNotice: database or disk is full
+2023/12/23 16:43:46 ...ers/web/repo/repo.go:434:Download() [E] archiver.Await: archiver.StartArchive: write /data/gitea/queues/common/000445.log: no space left on device
+2023/12/23 16:44:22 ...ers/web/repo/repo.go:434:Download() [E] archiver.Await: archiver.StartArchive: write /data/gitea/queues/common/000445.log: no space left on device
+```
+
+It has been found that a cause of this issue is the way the Forge (git server) responds to indexing by search engines. As part of the indexing process, a copy of <repository>/archive/master.zip (as one example) is created to facilitate downloading via the repository's "download as zip" button in the Forge web interface. 
+
+forest cleaned this up. From the #services:cyberia.club room on Matrix:
+
+*. I used du --max-depth=1 -h to narrow down where the disk usage is. I found out its in the famous repo-archive folder:
+
+
+```
+root@paimon:/etc/docker-compose/gitea# du --max-depth=1 -h /var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/
+3.5M	/var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/213
+2.6M	/var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/97
+12K	/var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/7
+...
+9.1G	/var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/
+
+```
+This happens because search engine crawlers navigate to the "get this repository as a zip file" link, and by default gitea caches those forever or something??
+
+https://forum.gitea.com/t/how-to-configure-cron-task-for-delete-all-repositories-archives-zip-tar-gz-etc/4848
+
+It looks like the "approved" way to clean up this crap would be by clicking a button in the admin panel. But I can't log in right now to even get to that admin panel.
+
+
+*. First I checked the CLI, the corresponding action is not present in the CLI:
+```
+
+root@paimon:/etc/docker-compose/gitea# docker exec -it gitea /bin/sh
+/ # forgejo --help
+...
+COMMANDS:
+   web              Start the Forgejo web server
+   serv             This command should only be called by SSH shell
+   hook             Delegate commands to corresponding Git hooks
+   dump             Dump Forgejo files and database
+   admin            Command line interface to perform common administrative operations
+...
+
+```
+
+*. Finally I realized I have to be able to log in 1st, in order to do that I truncated the journald logs to free up some space:
+```
+root@paimon:/etc/docker-compose/gitea# journalctl --disk-usage
+Archived and active journals take up 2.4G in the file system.
+```
+```
+root@paimon:/etc/docker-compose/gitea# journalctl --vacuum-size=1G
+...
+Vacuuming done, freed 1.4G of archived journals from /var/log/journal/5fa105a4f90a40a799e9cad174f058c0.
+Vacuuming done, freed 0B of archived journals from /var/log/journal.
+Vacuuming done, freed 0B of archived journals from /run/log/journal.
+```
+
+
+*. I logged in as my forest-admin account that has the admin=true on it and clicked this in the admin panel:
+[ git.cyberia.club/admin -V Dashboard -> Maintenance Operations -V "Delete all repositories' archives (ZIP, TAR.GZ, etc...)" -> {Run Button} ]
+
+Here's the disk usage for the gitea docker volume before and after:
+```
+root@paimon:/etc/docker-compose/gitea# du --max-depth=1 -h /var/lib/docker/volumes/gitea_gitea/
+14G	/var/lib/docker/volumes/gitea_gitea/_data
+14G	/var/lib/docker/volumes/gitea_gitea/
+root@paimon:/etc/docker-compose/gitea# du --max-depth=1 -h /var/lib/docker/volumes/gitea_gitea/
+4.1G	/var/lib/docker/volumes/gitea_gitea/_data
+4.1G	/var/lib/docker/volumes/gitea_gitea/
+```