Mix together the sugar, butter, vanilla and egg in a saucepan set over medium heat. Cook, stirring, until the sugar melts. Add the brandy and stir for 1 minute. Add the nutmeg and orange zest, and serve warm.
Hard sauce is traditionally served with figgy pudding, but can also be used with other cakes and desserts.