Since strangely enough lack of decent site downloaders I've written one myself.

It's battle tested by downloading WHOLE devrant and a big part of molodetz. Both big sites. It makes the downloaded sites portable by making absolute urls relative.

It downloads with a high concurrency.

Reason I've made this, is because I want to have all this data is so I have a lot of spam examples to train a model on.

Project page and features here: https://retoor.molodetz.nl/retoor/.... Source code at bottom as always.

I hope someone will give it a try :)

And yes, the docs costed almost the same time as the code. Code doesn't contain unit tests, it's production tested instead. I applied many optimizations mentioned by my review tool. When i was done I was too tired for unit tests.