More news from the verge, as the internet itself becomes smarter, big tech companies rethink internet policy.

https://theverge.com/24067997/...

https://en.wikipedia.org/wiki/...

#news #links #theverge #wikipedia #interesting #changing #bigtech #ai

did you really use hashtags here?? Go to Instagram for that shit.

that's not how robots.txt works. It was never meant to stop people scraping your site for content they had an interest in, it would be a very obviously unfit tool for that. The purpose has always been to indicate what is *worth* scraping for various purposes. Pages that search engine users would likely find useful, temporary and password protected URLs that are pointless to load unless you're someone specific.

Pointless to load unless youre a robot.