Create a way to limit disk usage

[coryalder]

I installed bitmagnet
It was working great, until It ran out of disk space
It stopped serving search requests, the web ui broke, and there appears to be no way to limit its endless expansion?

For me, a mere mortal with only ~100gb of space to dedicate to torrent indexing, this software is not usable. If I am not the intended audience, fair enough, close the issue and move on with your day - otherwise, it would be amazing if there was a way to limit the disk usage. It would even be useful to me if it could be capped at a size, and then I wipe it once a month and have it re-generate - but ideally some way where new entries are indexed, while old ones are tossed out.

Thanks!

[abitofevrything]

Disable the DHT crawler and you'll stop finding new torrents (and your database will then stop growing). You explicitly state which components you want to run when you start bitmagnet.

[coryalder]

Ah, I see! I installed using the proxmox helper script, so I didn't manually run bitmagnet worker run --keys=dht_crawler, or set a docker command.

So the suggested workflow for limiting disk usage is:

• setup bitmagnet, enabling the DHT crawler
• let it run until it's reached near the disk limit
• disable the DHT crawler

And then when I want new torrents to show up in search:

• wipe postgres database (or figure out a query that erases only older entries)
• enable DHT crawler again
• wait until it's near filling up its allotted disk
• disable the DHT crawler

Thanks!

[kde99]

Your biggest on disk table is the files one. You can decrease the amount of file data saved with the dht_crawler.save_files_threshold option, which by default is 100. Other tables can get quite large as well, and you cannot do much about that, that's just data after all. If you can change your filesystem, zfs can compress the db quite well (about 3x smaller).

Some numbers (though I have 1000 set as file_threshold) 22m in db:

          relation           | total_size
-----------------------------+------------
 torrent_files               | 159 GB
 torrent_contents            | 36 GB
 torrents                    | 8407 MB
 torrents_torrent_sources    | 6871 MB
 queue_jobs                  | 397 MB
 content                     | 198 MB
 content_attributes          | 115 MB
 content_collections_content | 65 MB
 bloom_filters               | 9312 kB
 content_collections         | 1640 kB
 torrent_sources             | 64 kB
 metadata_sources            | 64 kB
 goose_db_version            | 56 kB
 torrent_tags                | 32 kB
 key_values                  | 16 kB
 torrent_hints               | 16 kB
 torrent_pieces              | 16 kB
 goose_db_version_id_seq     | 8192 bytes

22219231 in db with 490015089 files or about 22.05 file per.

[lukyjay]

If there are certain torrents you are not going to need (e.g. comics, XXX, anime, software, etc.), you could update your classifier to delete these when they are found. This will have a large impact on the size of your database and will only keep torrents you're likely to need.

My Bitmagnet database is currently approaching 3 TB of used disk space because I haven't deleted anything yet.