Heat a griddle pan over medium heat. Cut 4 thick slices of cinnamon raisin bread. Butter both sides of the bread slices with softened butter. Grill bread until brown and crispy on both sides, 2 minutes on each side.
While bread grills, in a small saucepot, heat sweetened condensed milk or reduced fat sweetened condensed milk over medium low heat for 4 minutes. Remove from heat and stir in brandy.
Cut each grilled slice of bread into quarters and pile into 4 dessert cups. Top bread with a few spoonfuls of hard sauce of sweetened condensed milk and brandy and a generous swirl of whipped cream. Garnish cream with a sprinkle of nutmeg and serve.