Hi all!

I will soon acquire a pretty beefy unit compared to my current setup (3 node server with each 16C, 512G RAM and 32T Storage).

Currently I run TrueNAS and Proxmox on bare metal and most of my storage is made available to apps via SSHFS or NFS.

I recently started looking for “modern” distributed filesystems and found some interesting S3-like/compatible projects.

To name a few:

  • MinIO
  • SeaweedFS
  • Garage
  • GlusterFS

I like the idea of abstracting the filesystem to allow me to move data around, play with redundancy and balancing, etc.

My most important services are:

  • Plex (Media management/sharing)
  • Stash (Like Plex 🙃)
  • Nextcloud
  • Caddy with Adguard Home and Unbound DNS
  • Most of the Arr suite
  • Git, Wiki, File/Link sharing services

As you can see, a lot of download/streaming/torrenting of files accross services. Smaller services are on a Docker VM on Proxmox.

Currently the setup is messy due to the organic evolution of my setup, but since I will upgrade on brand new metal, I was looking for suggestions on the pillars.

So far, I am considering installing a Proxmox cluster with the 3 nodes and host VMs for the heavy stuff and a Docker VM.

How do you see the file storage portion? Should I try a full/partial plunge info S3-compatible object storage? What architecture/tech would be interesting to experiment with?

Or should I stick with tried-and-true, boring solutions like NFS Shares?

Thank you for your suggestions!

  • non_burglar@lemmy.world
    link
    fedilink
    English
    arrow-up
    11
    arrow-down
    1
    ·
    2 days ago

    Your workload just won’t see much difference with any of them, so take your pick.

    NFS is old, but if you add security constraints, it works really well. If you want to tune for bandwidth, try iSCSI , bonus points if you get zfs-over-iSCSI working with tuned block size. This last one is blazing fast if you have zfs at each and you do Zfs snapshots.

    Beyond that, you’re getting into very tuned SAN things, which people build their careers on, its a real rabbit hole.

    • Possibly linux@lemmy.zip
      link
      fedilink
      English
      arrow-up
      6
      arrow-down
      2
      ·
      1 day ago

      NFS with security does harm performance. For raw throughput it is best to use no encryption. Instead, use physical security.

      • non_burglar@lemmy.world
        link
        fedilink
        English
        arrow-up
        6
        ·
        1 day ago

        I don’t know what you’re on about, I’m talking about segregating with vlans and firewall.

        If you’re encrypting your San connection, your architecture is wrong.

          • non_burglar@lemmy.world
            link
            fedilink
            English
            arrow-up
            2
            ·
            1 day ago

            Oh, OK. I should have elaborated.

            Yes, agreed. It’s so difficult to secure NFS that it’s best to treat it like a local connection and just lock it right down, physically and logically.

            When i can, I use iscsi, but tuned NFS is almost as fast. I have a much higher workload than op, and i still am unable to bottleneck.

            • Possibly linux@lemmy.zip
              link
              fedilink
              English
              arrow-up
              2
              arrow-down
              1
              ·
              1 day ago

              Have you ever used NFS in a larger production environment? Many companies coming from VMware have expensive SAN systems and Proxmox doesn’t have great support for iscsi

                • Possibly linux@lemmy.zip
                  link
                  fedilink
                  English
                  arrow-up
                  2
                  arrow-down
                  1
                  ·
                  1 day ago

                  It really isn’t.

                  You can’t automatically create new disks with the create new VM wizard.

                  Also I hope you aren’t using the same security principals as 2005. The landscape has evolved immensity.

                  • non_burglar@lemmy.world
                    link
                    fedilink
                    English
                    arrow-up
                    1
                    ·
                    23 hours ago

                    Are you having trouble reading context?

                    No, I’m not applying 2005 security, I’m saying NFS hasn’t evolved much since 2005, so throw it in a dedicated link by itself with no other traffic and call it a day.

                    Yes, iscsi allows the use of mounted luns as datastores like any other, you just need to use the user space iscsi driver and tools so that iscsi-ls is available. Do not use the kernel driver and args. This is documented in many places.

                    If you’re gonna make claims to strangers on the internet, make sure you know what you’re talking about first.