Preheat the oven to 450 degrees F.
To make the compote: In a medium heavy-bottom pot, bring the water and sugar to a boil. Continue cooking until the mixture is reduced by half, about 5 minutes. Turn the heat to medium-low and add the remaining ingredients except the cheese and bread. Cook, stirring occasionally, until the kumquats are soft, 5 to 6 minutes. Let cool to room temperature. (Makes about 1/2 cup.)
Cut the bread slices into 3 pieces each. Place on a baking sheet and toast in the oven until light brown. Turn the slices and toast. Let cool completely.
To serve, place a little compote on each slice of bread. Serve the cheese alongside.