Using a pastry brush, brush the bread on both sides with the oil. Spread 1 side of each slice of bread with the raspberry jam. Sprinkle the rosemary over the jam. Cut the mozzarella into 8 slices. Place 2 slices of cheese on 4 of the bread slices. Season the cheese with a pinch of salt, if using. Place the remaining slices of bread on top, jam-side down. Sprinkle the tops with the brown sugar. Press in the panini press for 3 to 5 minutes until the cheese has melted and the bread is golden and crispy.