Preheat the oven to 350 degrees F. Line two 12-cup muffin pans with paper liners.
Make the cupcakes: Whisk the flour, baking soda and salt in a medium bowl; set aside. Beat the butter and granulated sugar in a large bowl with a mixer on medium speed until creamy, 2 inutes. Add the mashed bananas, eggs and vanilla; beat until combined. Reduce the mixer speed to low and beat in the flour mixture in three batches, alternating with the sour cream, mixing well after each addition.
Fill the muffin cups about halfway with batter; add 1 teaspoon Nutella to the center of each, then top with more batter so the cups are about three-quarters full. Bake until a toothpick inserted into the centers comes out with no raw batter, 20 to 25 minutes. Let cool 3 minutes in the pans, then carefully remove the cupcakes to a rack to cool completely.
Make the frosting: Beat the butter in a large bowl with a mixer on medium-high speed until creamy, about 5 minutes. Reduce the mixer speed to low and gradually beat in the confectioners' sugar until combined. Add the peanut butter and vanilla and beat until combined. Transfer the frosting to a pastry bag and pipe on the cupcakes.