Preheat oven 375 degrees F. Put some ice and cold water in a bowl. Bring a pot of salted water up to a boil. Add the broccoli, cook it for 2 minutes, and then transfer it to the ice water to stop cooking. Drain and set aside.
Put the bacon in a medium saucepan over medium heat and cook until crispy, 7 to 8 minutes. Remove the bacon, set aside, and reserve the fat in saucepan.
Turn the heat to medium-low heat, add the bacon and cook for 1 minute. Add the flour, and cook, whisking, until it starts to smell nutty and is a bit darker, 5 to 7 minutes; this is a roux. Slowly whisk the milk in to create a sauce with gravy-like consistency. Stir in the salt, pepper and nutmeg. Turn the heat to low and stir the Cheddar. Once the cheese has melted, take the saucepan off the heat.
Combine the panko and olive oil in a medium bowl. Coat a large baking dish or individual ramekins with non-stick cooking spray. Toss the broccoli with the cheese sauce and bacon. Put the mixture in the baking dish or ramekins, top it with the panko mixture and bake until the top is golden and the cheese is bubbly, 20 to 25 minutes. Serve hot.