Preheat the oven to 350 F. Line one 12-cup standard muffin tin with paper cupcake liners. Gently spray the liners with baking spray and set aside.
Sift the flour, baking powder and salt over a large piece of paper. In a large bowl, cream the butter and granulated sugar with a hand mixer on medium speed until light and fluffy. Beat in the eggs, egg yolk and vanilla. Reduce the speed to low and scrape down the sides of the bowl. Pour in the milk and continue to mix until smooth (the batter may look curdled at first but will smooth out). Pick up the paper with the dry ingredients and gradually pour into the wet ingredients. Continue to mix just until blended.
Spoon the batter evenly into the prepared tin, filling the cups about three-quarters full. Bake until the tops of the cupcakes spring back to the touch and are not too golden, about 20 minutes. Cool in the tin for 15 minutes, and then allow to cool completely on a wire rack before filling, frosting or decorating.
Fill a squeeze bottle with the grape jelly and screw on the cap. Carefully insert the tip of the squeeze bottle as far as it will go into the tops of the cupcakes. Gently squeeze about 1 tablespoon jelly inside of each cupcake. Ice the tops of the cupcakes with Peanut Butter Frosting to cover. Decorate with your favorite candy.
Beat the peanut butter, butter and cream cheese with a hand or stand mixer on medium speed until light and fluffy. Slowly add the confectioners' sugar and continue to mix until the frosting is smooth. Mix in the milk and continue to mix until it reaches a good spreading consistency.