↓

🌻ServicesForgeTroubleshooting

Problem 0: Users attempting to login via the web client receive nothing but a big "500"

History

The root cause:

To put it in terms of the log retrieved via a CLI, the root cause of this issue was found on 2023-12-23 to be:

root@paimon:/etc/docker-compose/gitea# docker logs -n 1000 -f gitea | grep -v router
2023/12/23 16:40:18 ...ers/web/repo/repo.go:434:Download() [E] archiver.Await: archiver.StartArchive: write /data/gitea/queues/common/000445.log: no space left on device
2023/12/23 16:43:32 ...ces/mirror/mirror.go:100:Update() [E] MirrorsIterate: write /data/gitea/queues/common/000445.log: no space left on device
2023/12/23 16:43:32 ...rvices/cron/tasks.go:129:func2() [E] CreateNotice: database or disk is full
2023/12/23 16:43:46 ...ers/web/repo/repo.go:434:Download() [E] archiver.Await: archiver.StartArchive: write /data/gitea/queues/common/000445.log: no space left on device
2023/12/23 16:44:22 ...ers/web/repo/repo.go:434:Download() [E] archiver.Await: archiver.StartArchive: write /data/gitea/queues/common/000445.log: no space left on device

The fix:

forest cleaned this up. From the #services:cyberia.club room on Matrix:

<forest:>

1. I used du --max-depth=1 -h to narrow down where the disk usage is. I found out its in the famous repo-archive folder:

root@paimon:/etc/docker-compose/gitea# du --max-depth=1 -h /var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/
3.5M	/var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/213
2.6M	/var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/97
12K	/var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/7
...
9.1G	/var/lib/docker/volumes/gitea_gitea/_data/gitea/repo-archive/

This happens because search engine crawlers navigate to the "get this repository as a zip file" link, and by default gitea caches those forever or something??

forum.gitea.com/t/how-to-configure-cron-task-for-delete-all-repositories-archives-zip-tar-gz-etc/4848

It looks like the "approved" way to clean up this crap would be by clicking a button in the admin panel. But I can't log in right now to even get to that admin panel.

2. First I checked the CLI, the corresponding action is not present in the CLI:

root@paimon:/etc/docker-compose/gitea# docker exec -it gitea /bin/sh
/ # forgejo --help
...
COMMANDS:
   web              Start the Forgejo web server
   serv             This command should only be called by SSH shell
   hook             Delegate commands to corresponding Git hooks
   dump             Dump Forgejo files and database
   admin            Command line interface to perform common administrative operations
...

3. Finally I realized I have to be able to log in 1st, in order to do that I truncated the journald logs to free up some space:

root@paimon:/etc/docker-compose/gitea# journalctl --disk-usage
Archived and active journals take up 2.4G in the file system.
root@paimon:/etc/docker-compose/gitea# journalctl --vacuum-size=1G
...
Vacuuming done, freed 1.4G of archived journals from /var/log/journal/5fa105a4f90a40a799e9cad174f058c0.
Vacuuming done, freed 0B of archived journals from /var/log/journal.
Vacuuming done, freed 0B of archived journals from /run/log/journal.

4. I logged in as my forest-admin account that has the admin=true on it and clicked this in the admin panel:

git.cyberia.club/admin -> Admin Panel -> Maintenance Operations -> "Delete all repositories' archives (ZIP, TAR.GZ, etc...)"

Here's the disk usage for the gitea docker volume before and after:

root@paimon:/etc/docker-compose/gitea# du --max-depth=1 -h /var/lib/docker/volumes/gitea_gitea/
14G	/var/lib/docker/volumes/gitea_gitea/_data
14G	/var/lib/docker/volumes/gitea_gitea/
root@paimon:/etc/docker-compose/gitea# du --max-depth=1 -h /var/lib/docker/volumes/gitea_gitea/
4.1G	/var/lib/docker/volumes/gitea_gitea/_data
4.1G	/var/lib/docker/volumes/gitea_gitea/

Later:
<forest:>
the eample in the link is outdated
see forgejo.org/docs/latest/admin/config-cheat-sheet/#basic-cron-tasks---enabled-by-default
allegedly its supposed to be enabled by default but idk if it actually is :/
ill try adding
root@paimon:/etc/docker-compose/gitea# nano /var/lib/docker/volumes/gitea_gitea/_data/gitea/conf/app.ini

[cron.archive_cleanup]
ENABLED = true
RUN_AT_START = true
SCHEDULE = @midnight;
OLDER_THAN: 24h

root@paimon:/etc/docker-compose/gitea# systemctl restart docker-compose@gitea.service

2024-01-06 3:34 PM:
symys found the server had run out of storage again and followed the above process manually. As this was the third time the server ran out of storage recently, symys changed the gitea configuration as follows:
root@paimon:/etc/docker-compose/gitea# vim /var/lib/docker/volumes/gitea_gitea/_data/gitea/conf/app.ini

[cron.archive_cleanup]
ENABLED = true
RUN_AT_START = true
SCHEDULE = @every 15m
OLDER_THAN = 15m

root@paimon:/etc/docker-compose/gitea# systemctl restart docker-compose@gitea.service
I think this more aggressive configuration should be okay, as a repo doesn't really need to be cached for more than 15 minutes except under rare circumstances.

Subhyphae

↑