Grill the bread over direct medium heat until toasted and grill marks appear, about 2 minutes, turning once halfway through grilling time. Cut bread into cubes and set aside.
In a large bowl, beat the eggs until frothy, about 3 minutes. Add the sugar, vanilla, nutmeg, cinnamon, and melted butter. Beat until well blended. Add milk, then stir in blackberries and lemon zest. Add bread cubes and toss until mixed well.
Let rest for about 45 minutes, patting the bread down into the liquid occasionally. Transfer bread and egg mixture to a seasoned cast-iron skillet. Cover bottom of skillet with foil to protect it from soot. Place pan in center of cooking grate and grill over indirect medium heat for 50 to 60 minutes, or until top is well-browned and puffy. Slice and serve warm, with ice cream or whipped cream, if desired.