Please note that viewing attachments of hyphae is not supported in history for now. Get Mycomarkup source of this revision

🌻ServicesForgeTroubleshooting

Users attempting to login via the web client receive nothing but a big "500"

History

To put it in terms of the log retrieved via a CLI, the root cause of this issue was found on 2023-12-23 to be:

root@paimon:/etc/docker-compose/gitea# docker logs -n 1000 -f gitea | grep -v router
2023/12/23 16:40:18 ...ers/web/repo/repo.go:434:Download() [E] archiver.Await: archiver.StartArchive: write /data/gitea/queues/common/000445.log: no space left on device
2023/12/23 16:43:32 ...ces/mirror/mirror.go:100:Update() [E] MirrorsIterate: write /data/gitea/queues/common/000445.log: no space left on device
2023/12/23 16:43:32 ...rvices/cron/tasks.go:129:func2() [E] CreateNotice: database or disk is full
2023/12/23 16:43:46 ...ers/web/repo/repo.go:434:Download() [E] archiver.Await: archiver.StartArchive: write /data/gitea/queues/common/000445.log: no space left on device
2023/12/23 16:44:22 ...ers/web/repo/repo.go:434:Download() [E] archiver.Await: archiver.StartArchive: write /data/gitea/queues/common/000445.log: no space left on device

It has been found that a cause of this issue is the way the Forge (git server) responds to indexing by search engines. As part of the indexing process, a copy of <repository>/archive/master.zip (as one example) is created to facilitate downloading via the repository's "download as zip" button in the Forge web interface.

forest cleaned this up. From the #services:cyberia.club room on Matrix:

  1. I used du --max-depth=1 -h to narrow down where the disk usage is. I found out its in the famous repo-archive folder:

root@paimon:/etc/docker-compose/gitea# du --max-depth=1 -h /var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/
3.5M	/var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/213
2.6M	/var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/97
12K	/var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/7
...
9.1G	/var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/

This happens because search engine crawlers navigate to the "get this repository as a zip file" link, and by default gitea caches those forever or something??

forum.gitea.com/t/how-to-configure-cron-task-for-delete-all-repositories-archives-zip-tar-gz-etc/4848

It looks like the "approved" way to clean up this crap would be by clicking a button in the admin panel. But I can't log in right now to even get to that admin panel.

  1. First I checked the CLI, the corresponding action is not present in the CLI:

root@paimon:/etc/docker-compose/gitea# docker exec -it gitea /bin/sh
/ # forgejo --help
...
COMMANDS:
   web              Start the Forgejo web server
   serv             This command should only be called by SSH shell
   hook             Delegate commands to corresponding Git hooks
   dump             Dump Forgejo files and database
   admin            Command line interface to perform common administrative operations
...

  1. Finally I realized I have to be able to log in 1st, in order to do that I truncated the journald logs to free up some space:

root@paimon:/etc/docker-compose/gitea# journalctl --disk-usage
Archived and active journals take up 2.4G in the file system.
root@paimon:/etc/docker-compose/gitea# journalctl --vacuum-size=1G
...
Vacuuming done, freed 1.4G of archived journals from /var/log/journal/5fa105a4f90a40a799e9cad174f058c0.
Vacuuming done, freed 0B of archived journals from /var/log/journal.
Vacuuming done, freed 0B of archived journals from /run/log/journal.

  1. I logged in as my forest-admin account that has the admin=true on it and clicked this in the admin panel:

[ git.cyberia.club/admin -V Dashboard -> Maintenance Operations -V "Delete all repositories' archives (ZIP, TAR.GZ, etc...)" -> {Run Button} ]

Here's the disk usage for the gitea docker volume before and after:

root@paimon:/etc/docker-compose/gitea# du --max-depth=1 -h /var/lib/docker/volumes/gitea_gitea/
14G	/var/lib/docker/volumes/gitea_gitea/_data
14G	/var/lib/docker/volumes/gitea_gitea/
root@paimon:/etc/docker-compose/gitea# du --max-depth=1 -h /var/lib/docker/volumes/gitea_gitea/
4.1G	/var/lib/docker/volumes/gitea_gitea/_data
4.1G	/var/lib/docker/volumes/gitea_gitea/

Subhyphae