Sorry for the downtime! Unfortunately our secondary firewall took over for some reason, and haproxy failed to properly come up.

I’ll be scheduling a maintenance window in the next few days to do some further digging, so I can make sure this is fully resolved.

  • whoisearthEnglish
    18·
    22 hours ago

    Wearing my Ops hat (what I do for a living can’t help it) have a few questions I’m more than happy to assist with depending on the answers.

    1. What TZ are you in?
    2. Are you open to volunteer support staff in other TZ?
    3. What monitoring solutions are you using? Nagios, Zabbix, etc.
    4. PagerDuty has a free tier do you have alerting and escalation setup?
    5. Do you have run books on the associated issues that come up?
    6. Could access be limited for a volunteer support staff?

    I woke up just before 7am EDT and noticed the status page makes reference to PDT so I can only assume you’re based out of BC. Given Canada has so many TZ it may be worthwhile investigating a more mature support model and give you some breathing room?

    • ShadowOPMAEnglish
      14·
      21 hours ago
      1. Myself, Otter and the server are all in Vancouver. Smorks is out east and mp3 is in the middle of all of us. Between us we actually have pretty good coverage most days.
      2. Yes, if someone has professional SRE experience and wants to help out I’m open to it. There’s very little on-going maintenance as things just run smoothly 99.999% of the time, but some additional eyes can’t hurt and if someone wants to build new stuff there’s things I’d like to do =)
      3. Betterstack at the moment, combined with some custom healthscript scripts which write to our discord and an improperly configured alert config on my phone that didn’t wake me up.
      4. Oh neat, I didn’t realize pagerduty had a free tier, I’ll check it out.
      5. Nope. There haven’t been issues that come up, execpt for this one stupid opnsense issue everything has been amazingly stable.
      6. To some degree yes.
    • SturgistEnglish
      9·
      22 hours ago

      Not admin team, but I think I remember them saying the servers are in Vancouver.

    • skankhunt42English
      6·
      21 hours ago

      I’m personally not willing to be on call but I was up around the same time and willing to help if they need it. I’ve been a Linux Admin for 13+ years and have my own k8s cluster in the basement.

      I also don’t mind the downtime. Thank you guys for everything you’re doing!