In a saucepan, combine the sugar with 1/4 cup of water and bring to a boil. Remove from the heat and allow let cool. Whisk in the buttermilk and passion fruit puree. Refrigerate until thoroughly chilled, about 3 hours.

Process in an ice cream maker, according to the manufacturer's instructions. Place in a plastic container and freeze until firm, about 2 hours or overnight.

To serve, scoop the sherbet into tall, frosted martini glasses and garnish with mango slices.