I know that Lemmy is open source and it can only get better from here on out, but I do wonder if any experts can weigh in whether the foundation is well written? Or are we building on top of 4 years worth of tech debt?
I know that Lemmy is open source and it can only get better from here on out, but I do wonder if any experts can weigh in whether the foundation is well written? Or are we building on top of 4 years worth of tech debt?
Lemmy’s backend is native code, not run in a virtual machine or interpreted from text like most backends (Java and PHP are still so, so popular, as is Python). You’re not going to pay much extra for an extra megabyte or 10 of RAM being used per instance for the extra code sitting idle. It certainly shouldn’t use much processing power when not in use.
I’d be less concerned with memory (of which Lemmy seems to use very little), and much more concerned with CPU core count. I touched on it in my other comment, but I don’t understand how a few cores is supposed to handle the ridiculous number of federation workers people are setting their instances to.
Does code that exists but isn’t being executed often actually impact CPU usage that much?
Linux kernel has like 30 million lines of code and you probably compile a fifth of it into your actual kernel binary by default. That’s still several million lines, but it doesn’t use much CPU at all. Rather, eliminating all the excess code keeps your size down so you can load it from disk in 0.0001 seconds instead of 0.0002.
Now if code that doesn’t need to run often, runs more often because we scaled the monolith horizontally - that IS a problem, but it’s not a problem inherent to the monolith design pattern, but rather a specific instance of bad design.
Again, my knowledge of the Lemmy codebase is very small, and we could possibly host the monolith in microservices style. The point I am making is this (when it comes to scaling monolith vs microservices):
If the federation logic were split out, we could configure it to run on super tiny docker instances on Google Cloud or AWS. Any time we needed it to, it would autoscale to handle the traffic. The configuration for these dockers could be super minimal memory, no storage, and multiple weak CPU cores. This would be super affordable while still being able to handle as much traffic from federation as we ask it to. One of the cool things with Google Cloud Run is that it handles load balancing between docker containers for you (just point the federation traffic at the necessary URL)
IF Lemmy has things like background services, scheduled tasks, etc, this would significantly muddy the water (we would need each service to be able to handle being run on a multitude of instances, or we would need to be able to disable each one instance by instance). And if we just scaled by spinning up more instances of Lemmy, we would also need to ensure that only federation traffic is heading to the weaker instances that we spun up for such purpose, or we would need to ensure that each spun up instance has enough resources to handle federation traffic along with the main application.
I feel like I need to state once more: I don’t necessarily think Lemmy needs to move to microservices. Only that scaling monolith vs microservices is not necessarily the same.
I suppose that’s completely fair - async workers tend to fare well as standalone services and are often split off even in monoliths. But I guess what I’m saying is that splitting it might not actually win you THAT much compared to just scaling the whole thing. Not until we’re talking like 100 runners to 1 API instance or something. It gives you a bit of additional flexibility, but won’t necessarily be a huge difference in total resource cost, is what I’m saying. But it is still a good idea because it results in cleaner code and, as outlined before, tinier docker images.
Also the thing about Google Cloud Run is that it’s probably not a good idea for many instance owners. Autoscaling can lead to unexpected costs if set up by an amateur. But that’s an unrelated can of worms.