what do you think about web scraping?
how often do you use it?

Dr. Web Dev:
Hi @amirbig thank you for your question! What you're expiriencing is completely normal and you don't have to worry about your parrents finding out. Webscraping is a lot of fun and there is no reason to be embarrassed of doing it. In fact most web devs in your age expirience the joy of getting data of the internet 2 to 3 times a day. But remember to use protection. A good firewall like pfsense is advised. Enjoy life and dear greetings Dr. Web Dev. 😄

Web scraping is something sacred. You should only do it if both parties want it.

I wouldn't lose that yummy data breach a couple of weeks ago only if I had some serious experience with scraping.

Practice now, and thank yourself later. ;-)

Wen scraping is a good skill to have and a good tool to use.
The worst thing about it is maintenance across time, especially if you are working on multiple pipelines.

I made a scraper yesterday. The site in question has tables, you can scroll the table, javascript runs and rewrites the table content. The table is found via several generated iframe sources.

Selenium saved the day. Won't work headless though for some reason.

I'm writing a scraper right now.
The markup is madness. Four fucking nested tables! Elements have 30-character names containing multiple $'s! They look generated, but fortunately they don't seem to change between page loads. All content is fetched and inserted via ajax.

Selenium ftw.

A good to have skill

I work for a data analysis startup, and here we use scraping quite frequently.

And it's super fun to manage all that scraping, gets better with the number of sites you scrape