Right now, robots.txt on lemmy.ca is configured this way

User-Agent: *
  Disallow: /login
  Disallow: /login_reset
  Disallow: /settings
  Disallow: /create_community
  Disallow: /create_post
  Disallow: /create_private_message
  Disallow: /inbox
  Disallow: /setup
  Disallow: /admin
  Disallow: /password_change
  Disallow: /search/
  Disallow: /modlog

Would it be a good idea privacy-wise to deny GPTBot from scrapping content from the server?

User-agent: GPTBot
Disallow: /

Thanks!

  • sndmn
    link
    fedilink
    arrow-up
    8
    ·
    11 months ago

    Is this even possible without all federated instances also prohibiting them?

    • m-p{3}OPA
      link
      fedilink
      English
      arrow-up
      14
      ·
      11 months ago

      You take action where you can ;)