Combine the lime chunks and sugar in a large pitcher. Using a muddler or large wooden spoon, mash the limes to release the juices. Add the peach chunks and mint and lightly muddle to bruise the mint and break up the peaches; stir in the rum. (The mixture can be refrigerated up to 6 hours.)
Just before serving, pour the peach-rum mixture into ice-filled glasses and top with seltzer. Garnish with peach wedges.