In a large bowl, combine the cornmeal, flour, garlic powder, and cayenne together. Pour the buttermilk into a separate bowl and season with salt and pepper. Dip the tomatoes in the buttermilk and then dredge them in the cornmeal mixture, coating both sides well.

Place a large cast iron skillet over medium heat and coat with the oil. When the oil is hot, pan-fry the tomatoes (in batches if necessary) until golden brown and crispy on both sides, about 3 to 4 minutes on each side. Carefully remove the tomatoes and drain on paper towels. Serve with hot pepper sauce and lemon.