devRant - A fun community for developers to connect over code, tech & life as a programmer

Search - "site downloader"

43

b2plane

6414

4y

Based on a true story that happened right now.

Dad: "how do i download youtube videos?"

Me: "just google youtube downloader and download them from some site, thats how i do it"

Dad: "WHAT!!??? You want me to fucking google it? I dont know how to fucking google for those things, you're the IT guy and you should know how to do this, if I wanted to google it i wouldnt ask you for help. You know what, get the fuck out of my face i dont need ur help, get out"

rant

28
12

donuts

23614

6y

Dilbert no longer lets me leech their comics from their site... It was a good... 10 years?

Seems I need to rebuild my Downloader... using Selenium.

Now...where is that project where i used Selenium...

random

1
3

donuts

23614

6y

Haven't been reading Dilbert for other a month but finally got around to it today. I read it through an app I made that downloads all the comics from the site.

Well apparently the downloader downloaded too much/too fast. It seems my IP is now blocked....

Wonder if it's temporary. O well... I got VPNs...

rant
3

retoor

4507

8d

Since strangely enough lack of decent site downloaders I've written one myself.

It's battle tested by downloading WHOLE devrant and a big part of molodetz. Both big sites. It makes the downloaded sites portable by making absolute urls relative.

It downloads with a high concurrency.

Reason I've made this, is because I want to have all this data is so I have a lot of spam examples to train a model on.

Project page and features here: https://retoor.molodetz.nl/retoor/.... Source code at bottom as always.

I hope someone will give it a try :)

And yes, the docs costed almost the same time as the code. Code doesn't contain unit tests, it's production tested instead. I applied many optimizations mentioned by my review tool. When i was done I was too tired for unit tests.

random concurrent https absolute portable molodetz downie relative site downloader devrant crawler battle tested

5

Top Tags

rant linux code windows fuck i java c programming android dev the is javascript js a life joke python

Weekly Rant

Most unrealistic deadline you've had?

devRant © 2021 Hexical Labs LLC
Privacy Policy | Terms of Service