In a small saucepan, combine the allspice, thyme, Scotch bonnet, grated garlic, ginger, sugar, vinegar, 1 teaspoon salt and 1/4 cup water. Bring to a simmer and cook until the sugar and salt are dissolved and the spices are fragrant, about 2 minutes. Transfer to a serving bowl, along with the carrots, red onions, red peppers and yellow peppers; toss to coat. Cover and refrigerate while you make the fish.
Stir together the mayonnaise, sweet chile sauce, lime juice and scallions in a bowl. Set aside while you grill the fish.
Preheat a grill pan over medium-high heat. Rub the snapper fillets all over with the crushed garlic clove; discard the garlic. Sprinkle the fillets with salt and a generous grinding of black pepper, then brush on all sides with oil. Brush the grill pan with oil. Grill the fish, turning once, until just cooked through, about 2 minutes per side. Remove to a cutting board and cut or break into pieces to fit the rolls.
Spread the mayo mixture on the rolls. Remove and discard the Scotch bonnet and the thyme sprigs from the pickled pepper relish. Top the rolls with the fish, a pinch of the pickled vegetables and a sprinkle of the pickling liquid. Serve the sliders with the remaining mayo and pepper relish on the side.