scraping - What is the best way to scrape websites? - devRant

Ranter

Comments

5

Snob

2105

5y

Python + BeautifulSoup
1

Stebner55

1858

5y

I've enjoyed PhantomJS/CasperJS but the project is essentially dead now, I know a ton of people use Selenium, and headless chrome is becoming a popular solution.
1

beegC0de

1313

5y

An implementation of curl and an HTML parser
0

tkdmatze

443

5y

Greasemonkey and nodejs
1

Kimmax

11247

5y

Don't mind me, I'm just waiting to shit on the person who's going to recommend regex 👀
0

vocuzi

8283

5y

Sites not using JS : Requests + BeautifulSoup

if site using JS add :
- Chromedriver + Selenium
- Scrapy Splash (one beautiful piece of work)

Or use scrapy, a crappy lib but okay
0

Stebner55

1858

5y

@justin-tamblyn

When there are counter measures, you often need to run your script through a proxy to change up your IP address or you'll just get blocked.

Related Rants

devRant © 2021 Hexical Labs LLC
Privacy Policy | Terms of Service