Sorry for the downtime! Unfortunately our secondary firewall took over for some reason, and haproxy failed to properly come up.

I’ll be scheduling a maintenance window in the next few days to do some further digging, so I can make sure this is fully resolved.

  • BCsvenEnglish
    34·
    14 hours ago

    As Canadian as it gets :) Apologizing for the free service you offer.

    Don’t sweat it, everyone here appreciates the effort put in to run this, even if they don’t reach out and express it.

      • DarkSirrushEnglish
        6·
        12 hours ago

        During your next maintenance put in a second, secret uptime counter and once it hits a reasonable amount of time swap them, nobody will notice!

      • corsicanguppyEnglish
        2·
        12 hours ago

        The high work ethic is plain to see. Nice debug and recovery.

      • Grappling7155English
        2·
        12 hours ago

        Even hyperscaler cloud service providers don’t aim for 100%, don’t sweat it

  • slothropEnglish
    19·
    15 hours ago

    I had the shakes, deetees, drools and bends for a while.
    This is the only instance I’m not banned on.
    .
    .
    .
    .

    /s

    • ShadowOPMAEnglish
      28·
      14 hours ago

      Oh hold on, let me go fix that…

  • whoisearthEnglish
    18·
    15 hours ago

    Wearing my Ops hat (what I do for a living can’t help it) have a few questions I’m more than happy to assist with depending on the answers.

    1. What TZ are you in?
    2. Are you open to volunteer support staff in other TZ?
    3. What monitoring solutions are you using? Nagios, Zabbix, etc.
    4. PagerDuty has a free tier do you have alerting and escalation setup?
    5. Do you have run books on the associated issues that come up?
    6. Could access be limited for a volunteer support staff?

    I woke up just before 7am EDT and noticed the status page makes reference to PDT so I can only assume you’re based out of BC. Given Canada has so many TZ it may be worthwhile investigating a more mature support model and give you some breathing room?

    • ShadowOPMAEnglish
      13·
      14 hours ago
      1. Myself, Otter and the server are all in Vancouver. Smorks is out east and mp3 is in the middle of all of us. Between us we actually have pretty good coverage most days.
      2. Yes, if someone has professional SRE experience and wants to help out I’m open to it. There’s very little on-going maintenance as things just run smoothly 99.999% of the time, but some additional eyes can’t hurt and if someone wants to build new stuff there’s things I’d like to do =)
      3. Betterstack at the moment, combined with some custom healthscript scripts which write to our discord and an improperly configured alert config on my phone that didn’t wake me up.
      4. Oh neat, I didn’t realize pagerduty had a free tier, I’ll check it out.
      5. Nope. There haven’t been issues that come up, execpt for this one stupid opnsense issue everything has been amazingly stable.
      6. To some degree yes.
    • SturgistEnglish
      9·
      15 hours ago

      Not admin team, but I think I remember them saying the servers are in Vancouver.

    • skankhunt42English
      6·
      15 hours ago

      I’m personally not willing to be on call but I was up around the same time and willing to help if they need it. I’ve been a Linux Admin for 13+ years and have my own k8s cluster in the basement.

      I also don’t mind the downtime. Thank you guys for everything you’re doing!

  • AlbumEnglish
    6·
    12 hours ago

    Thank you for keeping the lights on!

  • SneakyWeaselEnglish
    7·
    15 hours ago

    All good. Made me fall asleep easier last night as i didn’t doom scroll. Haha

  • GameGodEnglish
    4·
    15 hours ago

    You need that SRE team you said you don’t have. :)