Sift dry ingredients into a medium bowl. Combine egg and buttermilk, and stir into dry ingredients, mixing until well combined. Place bananas carefully in batter. (They can rest in the batter, covered until you are ready to fry them.)
Heat the oil to 350 to 390 degrees F or until the oil gives off a slight haze. Make sure bananas are well coated with batter before setting them in oil. Using a fork, place coated bananas in oil and fry for 2 to 4 minutes, or until golden brown. Remove from oil with tongs and drain on several layers of paper towels. Dredge drained fritters in sugar and serve hot with ice cream and caramel or chocolate sauce.