Preheat the oven to 500 degrees F.
For the dipping sauce: In a small bowl, whisk together the buttermilk, yogurt, capers, agave, lemon zest, cayenne and salt. Set aside to let the flavors marry.
For the shrimp: In a medium bowl, mix together the shrimp, olive oil, black pepper, pink pepper and salt. Toss well to coat. Spread the rosemary sprigs on a rimmed baking sheet and pour the shrimp on top. Bake until the shrimp are pink and opaque throughout, 8 minutes.
Serve the shrimp alongside the dipping sauce, with lemon wedges and a separate bowl for the shells.