You're going to have outages in production. They're inevitable. The question is how to best minimize outages, both their frequency and duration. Common wi...

Sharing some lessons I learned from 10 years/millions of users in production. I’ll be in the comments if anyone has any questions!

  • MajorHavoc@programming.dev
    171·
    13 days ago

    You make some great points, but I’m concerned that your preferred solutions may ignore the needs of working with peers. When I’ve worked with similar solutions before, we had a lot of on call, and it all went to the same person, regardless of who actually answered the phone.

    There’s nuance to be had in the middle ground:

    • The CI/CD pipeline should deliver releases, but shouldn’t be the only way to put them into production. People often get this wrong and have to invoke CI/CD to even do a roll back. Putting something into production without CI/CD should be possible, but it should be loud, to avoid nasty surprises, later.
    • Infrastructure as code is great. It just happens to be backwards, today. We could all go back to point and click changes, if the infrastructure had full journaling (including who made the change) of changes and full rollback capability. Ironically, I expect k8s (or a fork of it) to finally deliver this.
    • the only nice thing I have to say about K8s today is that it’s not locking myself into a proprietary cloud, or worse, VMWare. Today, K8s is towering mere centimeters above even worse solutions, all of them objectively awful. I’m not mad at anyone using any of these. We all know we want the freedom of K8s, but no one wants the bullshit interfaces it currently comes with. It will get better.

    Anyway, interesting read. Thank you. The only way the current awful state of hosting is going to improve is by having this conversation.

    I keep hearing “most people aren’t ready for K8S”. But there’s no such thing. There’s just whether K8S (or whatever replaces it) is ready for most people’s use cases.

    • something_random_tho@lemmy.worldOP
      11·
      13 days ago

      I’m concerned that your preferred solutions may ignore the needs of working with peers. When I’ve worked with similar solutions before, we had a lot of on call, and it all went to the same person, regardless of who actually answered the phone.

      Totally hear you and have the same experience myself. The approach I’m advocating for is simply running a binary on a server with rsync to deploy, and architecting your product around that limitation. Teaching a team the basics of Linux sysadmin will be incredibly useful for their careers, and it’s something that the whole team can easily learn. Then you don’t need to hire a k8s team – any engineer can do some basic debugging when things go sideways.

      • MajorHavoc@programming.dev
        7·
        13 days ago

        The approach I’m advocating for is simply running a binary on a server with rsync to deploy, and architecting your product around that limitation.

        Intriguing!

        I’m looking forward to your blog series on this.