Sorry for the downtime! Unfortunately our secondary firewall took over for some reason, and haproxy failed to properly come up.

I’ll be scheduling a maintenance window in the next few days to do some further digging, so I can make sure this is fully resolved.

  • ShadowOPMAEnglish
    14·
    22 hours ago
    1. Myself, Otter and the server are all in Vancouver. Smorks is out east and mp3 is in the middle of all of us. Between us we actually have pretty good coverage most days.
    2. Yes, if someone has professional SRE experience and wants to help out I’m open to it. There’s very little on-going maintenance as things just run smoothly 99.999% of the time, but some additional eyes can’t hurt and if someone wants to build new stuff there’s things I’d like to do =)
    3. Betterstack at the moment, combined with some custom healthscript scripts which write to our discord and an improperly configured alert config on my phone that didn’t wake me up.
    4. Oh neat, I didn’t realize pagerduty had a free tier, I’ll check it out.
    5. Nope. There haven’t been issues that come up, execpt for this one stupid opnsense issue everything has been amazingly stable.
    6. To some degree yes.