you might have noticed we lost a bunch of posts…

in short: we tried to fix a Lemmy issue that started on 9/13 and is causing or has caused problems with apps for the software. we’re aware of basically all of them besides Sync exhibiting some sort of problematic behavior since this issue arose. however: attempting to fix this issue on our side of things did not go well.

i am not qualified to make a formal write-up of the exact mechanism of what went wrong here–and in this case the mechanism isn’t really important for your informational purposes, although @[email protected] can elaborate as needed–but basically we made a fix that seemed fine in testing and was not fine when actually applied. when it became obvious something was wrong, we tried restoring literally everything but the database. that, unfortunately, did not work, so eventually we just pulled the trigger on restoring from a backup. this has lost at least some posts, but probably no more than 8 hours of them by our estimation.

hopefully this will not happen again, either upstream or on our side of things.

  • rozwud@beehaw.org
    link
    fedilink
    arrow-up
    24
    ·
    1 year ago

    I just want to say that the downtime made me extra sure that I will follow beehaw wherever it goes. I’m definitely more lurky than chatty, but after browsing via other instances, I missed beehaw.

    • dawt@beehaw.org
      link
      fedilink
      arrow-up
      10
      ·
      1 year ago

      Yeah… Other lemmy instances just don’t cut it for me. Beehaw definitely has a quality difference.

  • JCPhoenix@beehaw.org
    link
    fedilink
    English
    arrow-up
    19
    arrow-down
    1
    ·
    1 year ago

    When we moving to another platform again? 🙃

    The more I look at Lemmy…the more I think this platform is just not ready for primetime. I suppose that’s obvious since its version is still sub 1.0. Idk.

    • Chris Remington@beehaw.org
      link
      fedilink
      arrow-up
      19
      ·
      1 year ago

      When we moving to another platform again?

      Several months from now. We need time to test out about 4-5 different platforms and get feedback from the community.

      • Scary le Poo@beehaw.org
        link
        fedilink
        arrow-up
        1
        ·
        1 year ago

        I’m hoping that Postmill makes the cut. It looks very easy to work on, but I haven’t had the time to set up an instance and poke around. Maybe I’ll do that today.

      • MJBrune@beehaw.org
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        it seems pretty clear that we aren’t staying on Lemmy or on federated services at this point due to beehaw’s policies and lack of fediverse moderation.

          • MJBrune@beehaw.org
            link
            fedilink
            English
            arrow-up
            2
            ·
            1 year ago

            fediverse is an API in which these services can share information. That’s how Mastodon users and Kbin users can comment on Lemmy posts. These services all federate together through federation lists. Making a new API for this would be pointless.

            The fediverse is supposed to allow for more open and transformable platforms but in reality, what it’s doing is making groups of people between these services that fit into their own echo chambers. I predict in the future people are going to start sharing allow/deny lists of groups and join mini federations that will consolidate more and more power in the fediverse. New instances will have to be adopted into a shared list of allowed instances. Like bitcoin, the fediverse promises to fix something but will likely make it worse.

  • the w@beehaw.org
    link
    fedilink
    English
    arrow-up
    10
    ·
    1 year ago

    I once again very much appreciate the transparency. Sounds like a not fun time. Thanks for doing what you do.

  • Gamma@beehaw.org
    link
    fedilink
    English
    arrow-up
    6
    ·
    edit-2
    1 year ago

    Thanks for the update, I hope you all can rest for a bit after that!

  • realChem@beehaw.org
    link
    fedilink
    arrow-up
    5
    ·
    1 year ago

    Did the fix end up working in the end, or did restoring everything from backups mean that the fix didn’t work out either? (I don’t use Jerboa or any other lemmy apps, so mainly just curious.)

    • PenguinCoder@beehaw.orgM
      link
      fedilink
      English
      arrow-up
      6
      ·
      1 year ago

      This was the initially issue I was trying to fix; apps not loading. Tracked it down to the timezone handling of lemmy. The ‘fix’ (upgrading to upstream lemmy code) didn’t work, and broke a lot else.

  • jarfil@beehaw.org
    link
    fedilink
    arrow-up
    4
    ·
    1 year ago

    this has lost at least some posts, but probably no more than 8 hours of them by our estimation.

    Based on comment history, it would seem more like 2 days… or maybe that’s just for comments?