The rise and fall of robots.txt
www.theverge.com
As unscrupulous AI companies crawl for more and more data, the basic social contract of the web is falling apart.
  • Cosmic Cleric@lemmy.worldEnglish
    61·
    10 months ago

    When sites regularly have a robots.txt that says “only Google may crawl”

    Is that actually true?

    If so, why would they do that?