I thought about the same thing for insects and came to the conclusion, you would need some sort of barrier that only lets the prey through, occasionally. a bait, like a light, to attract the prey could also help to make feeding more regularly.

Cultivating live food for your fish friends is a great idea! Research, trial, and error are going to be really important in the process, so it’s better to do it in a separate tank to keep your fish safe. We don’t have the ability to bring an entire complex ecosystem into our homes and put it in a glass box; the best we can do is mimic one as best as possible. This means that an experimental tank won’t have as many of the same “redundancies” nature does when nutrient, light, or oxygen/dissolved gases get out of balance, and population crashes can happen a lot more easily. (Even if it’s just the food creatures that die off, the decaying matter can cause problems like ammonia burns for your fish.) Rather than trying to establish a sustainable colony of food animals in your fish tank, starting a tank with food animals and slowly adding fish to it will give you a much clearer picture of how large a population they can sustain.

Some tips to help you build as robust a system as you can:

• Use a 10 gallon (about 40 liters) or larger tank. Greater water volume means increased stability because the water dilutes concentrations of things like ammonia and gives you a nice buffer for environmental changes; a 5 gallon (about 20 liters) tank will have much narrower safety margins and will make it harder to figure out where the problems lie.
• Add a lot of plants, both in number and variety, to improve oxygenation and reduce carbon dioxide in the water. They’ll also provide food and grazing surfaces for a lot of smaller food animals.
• Use a canister filter or AquaClear-style hang-on-back (HOB) filter that gives you a lot more filter medium for housing the microbes that process nitrogenous wastes like ammonia, nitrite, and nitrate than a single, fiber-covered plastic cartridge. Don’t run charcoal or activated carbon in it unless you’re finishing up a round of disease treatment–it’ll pull out a lot of the stuff the tank’s plants and animals can use. Canisters and HOB filters will also give you better gas exchange than an undergravel or sponge filter will.
• Tank barriers are easy to make out of plastic mesh used for crafts like needlepoint. I think the holes would be big enough for most food animals to swim through, but not allow fish in their area, but I’m not sure.

This sounds like a really cool project. If you decide to do it, I hope you’ll post updates!

I found a discussion with a lot of info if someone else was looking for this as well. Apparently called a refugium https://www.plantedtank.net/threads/sustainable-live-food-in-tank.1330307/?post_id=11491601&nested_view=1&sortby=oldest#post-11491601