Bring a large pot of salted water to a boil. Add the cauliflower and cook until just fork tender, 3 to 5 minutes. Drain thoroughly and transfer the cauliflower to a cutting board. Chop into very small pieces (about the size of peas or smaller) with a sharp knife.
Measure 4 cups of the chopped cauliflower and place in a large bowl. (Reserve the rest for another use, such as cauliflower tots or mashed cauliflower.) Add the flour, eggs, garlic, 3 tablespoons scallions, 1/2 teaspoon salt and 1/4 teaspoon pepper; stir until well combined.
Line a plate with paper towels. Heat a large nonstick skillet over medium-low heat and lightly coat it with olive oil. Working in batches, scoop 2- to 3-tablespoon mounds of the cauliflower mixture into the pan, spacing them at least 1 inch apart. Flatten each mound slightly with a spatula and cook for 2 to 3 minutes. Flip the fritters once and cook until golden brown and cooked through, 1 to 2 minutes. Transfer the fritters to the prepared plate. Serve warm, topped with yogurt or sour cream. Garnish with scallions.