1
nike
153d

More news from the verge, as the internet itself becomes smarter, big tech companies rethink internet policy.

https://theverge.com/24067997/...

https://en.wikipedia.org/wiki/...

#news #links #theverge #wikipedia #interesting #changing #bigtech #ai

Comments
  • 3
    did you really use hashtags here?? Go to Instagram for that shit.
  • 3
    that's not how robots.txt works. It was never meant to stop people scraping your site for content they had an interest in, it would be a very obviously unfit tool for that. The purpose has always been to indicate what is *worth* scraping for various purposes. Pages that search engine users would likely find useful, temporary and password protected URLs that are pointless to load unless you're someone specific.
  • 0
    Pointless to load unless youre a robot.
Add Comment