The first part “silly things” refers to the phrase “silly goose”, which pluralized makes “geese”. The second part “capital savers” is harder to understand. According to an analysis of the original Guardian puzzle solutions, it alludes to an ancient tale of how geese (supposedly) saved the roman capitol from invading gauls. That explanation, however, does not explain why the word is spelled "capital. Perhaps that difference is just a nested pun on the letter-switching (oo to ee) used to make the plural in the first part, or “capital savers” may instead (or as well) refer to the fairy tale of the goose that laid golden eggs (gold being capital, and the goose being something that should be saved and not cooked/killed).