Apologies to all for today’s performance issues and downtime.

Logging for our lemmy instance filled our main partition and brought everything to a halt. Our lemmy instance shares hardware with blahaj.zone, and the significant growth in lemmy over the last 10 days has caused the logs to grow in size exponentially! As a result of the logs filling up the main partition, both lemmy.blahaj.zone and blahaj.zone itself went down.

We’ve freed up some space, upgraded the hardware and spun everything up again, and it looks everything is back to normal.

Apologies for the inconvenience, and welcome back to everyone!

  • jherazob@beehaw.org
    link
    fedilink
    English
    arrow-up
    12
    ·
    1 year ago

    Servers going down because logs fill a disk is one common Sysadmin experience, achievement unlocked! 😄

    Thanks for doing the work 👍

    • Ada@lemmy.blahaj.zoneOPM
      link
      fedilink
      English
      arrow-up
      8
      ·
      1 year ago

      I mean, if we expected lemmy to grow in to an instance 3 times the size of blahaj.zone, we’d have done things differently from the start, but suffice to say, we didn’t expect Blahaj Lemmy to be anything but a bit of a sandbox for me to play with, so we made some initial choices that didn’t scale well with sudden unexpected growth :)

      • jherazob@beehaw.org
        link
        fedilink
        English
        arrow-up
        2
        ·
        1 year ago

        And that too is an achievement unlocked! 😄

        Been there, done that, got some premature white hairs :P