Preheat the oven to 350 degrees. Butter a 10 by 15-inch jelly roll pan and line the bottom with a sheet of parchment or waxed paper.
In a large mixer bowl, beat the butter and sugar until soft and light. Add the eggs one at a time, beating well between each addition. Beat in the vanilla. Sift together the flour and baking powder and stir into the batter in three additions, alternating with the yolks and milk, beginning and ending with the flour. Spread the batter evenly in the prepared pan. Scatter the cherries evenly over the batter, pressing them in gently.
Scatter the Crumb Topping over the berries as evenly as possible. Bake the cake about 40 minutes until batter is firm and crumbs are well colored. Cool the cake in the pan on a rack. Cut the cooled cake into 24 (3-inch) squares. Remove the squares from the pan to a platter.
Mix the flour, sugars, cinnamon, and nutmeg in a bowl. Melt the butter and stir it in evenly. Rub the mixture to coarse crumbs by hand.