I have had this plant for several years. Within the past couple months I noticed a few leaves yellowing, but lately it has accelerated greatly to where about half have fallen. I water regularly and keep the soil moist. The pot is large and does have drainage holes. I stopped watering about 2 weeks ago to let the soil dry out but that hasnt seemed to help. This year I have noticed earthworms and centipedes in the soil if I dig around.
Interesting. I have not repotted or fertilized. The pot is fairly large so repotting would be hard but I will definitely try some fertilizer
You definitely need to fertilize. It might not be the root problem, but any potted plant needs fertilizer because nitrogen is water soluble and washes out when you water it. The leaves look chlorotic, so lack of fertilizer tracks. I recommend a balanced water-soluble fertilizer. Use it in a dilute application any time the plant shows growth or is flowering.
Once that is going, you can further diagnose issues.