Right now, robots.txt on lemmy.ca is configured this way

User-Agent: *
  Disallow: /login
  Disallow: /login_reset
  Disallow: /settings
  Disallow: /create_community
  Disallow: /create_post
  Disallow: /create_private_message
  Disallow: /inbox
  Disallow: /setup
  Disallow: /admin
  Disallow: /password_change
  Disallow: /search/
  Disallow: /modlog

Would it be a good idea privacy-wise to deny GPTBot from scrapping content from the server?

User-agent: GPTBot
Disallow: /

Thanks!

  • ono
    link
    fedilink
    English
    arrow-up
    20
    ·
    10 months ago

    Yes, please.

    We can’t stop LLM developers from scraping our conversations if they’re determined to do so, but we can at least make our wishes clear. If they respect our wishes, then great. If they don’t, then they’ll be unable to plead ignorance, and our signpost in the road (along with those from other instances) might influence legislation as it’s drafted in the coming years.

  • ShadowMA
    link
    fedilink
    arrow-up
    19
    ·
    10 months ago

    I’m on board for this, but I feel obliged to point out that it’s basically symbolic and won’t mean anything. Since all the data is federated out, they have a plethora of places to harvest it from - or more likely just run their own activitypub harvester.

    I’ve thrown a block into nginx so I don’t need to muck with robots.txt inside the lemmy-ui container.

    # curl -H 'User-agent: GPTBot' https://lemmy.ca/ -i
    HTTP/2 403
    
    • skankhunt42
      link
      fedilink
      arrow-up
      3
      ·
      10 months ago

      I imagine they rate limit their requests too so I doubt you’ll notice any difference in resource usage. OVH is Unmetered* so bandwidth isn’t really a concern either.

      I don’t think it will hurt anything but adding it is kind of pointless for the reasons you said.

  • nbailey
    link
    fedilink
    English
    arrow-up
    18
    ·
    10 months ago

    Yes. Ban them.

    if ($http_user_agent = "GPTBot") {
      return 403;
    }
    
  • sndmn
    link
    fedilink
    arrow-up
    8
    ·
    10 months ago

    Is this even possible without all federated instances also prohibiting them?

    • m-p{3}OPA
      link
      fedilink
      English
      arrow-up
      14
      ·
      10 months ago

      You take action where you can ;)

  • narF
    link
    fedilink
    arrow-up
    5
    ·
    10 months ago

    Are they even respecting those files?

    But yeah, sure, it’s worth trying!

  • Elise@beehaw.org
    link
    fedilink
    English
    arrow-up
    2
    ·
    10 months ago

    Just out of curiosity, why is everyone so up in arms about this? I mean sure it’s just another corp but any other reasons?

    • corsicanguppy
      link
      fedilink
      arrow-up
      6
      ·
      10 months ago

      Server load spent on a bot scraping our contributions to be used to make money.

      There’s so much there that it’s gonna offend someone.

      • Elise@beehaw.org
        link
        fedilink
        arrow-up
        1
        ·
        10 months ago

        Wouldn’t it just be scraped once (per company)? That doesn’t sound like such a problem.

  • EhForumUser
    link
    fedilink
    arrow-up
    1
    arrow-down
    3
    ·
    edit-2
    10 months ago

    No, definitely not. Our work posted in the open is done so because we want it to be open!

    It is understandable that not all work wants to be open, but access would already be appropriately locked down for all robots (and humans!) who are not a member of the secret club in those cases. There is no need for special treatment here.