For the banana bread: Preheat the oven to 350 degrees F. Spray two 9-by-5-inch loaf pans with nonstick cooking spray.
Combine the cake mix, oil, mashed bananas, eggs and 1/2 cup water in a large bowl and mix with a hand mixer or whisk until well blended, about 2 minutes. Fold in the peanut butter chips. Evenly divide the batter between the 2 prepared loaf pans.
Bake until a toothpick inserted in the center comes out clean, 40 to 50 minutes. Let cool in the pans for 10 minutes, then remove and let cool on a cooling rack.
For the peanut butter glaze: Combine the confectioners' sugar, peanut butter and 3 tablespoons of the warm milk in a bowl and whisk until smooth. It should be about as thick as pancake batter; if it's too thick, just add a little more milk.
Pour some of the glaze over each loaf and let set for about 10 minutes. Slice and serve.