The brioche should be slightly stale so it can absorb the custard without becoming too soggy and ultimately falling apart. If you only have fresh brioche, you can also toast the bread to dry it out. Place the bread cubes on an unlined baking sheet, then bake at 350 degrees F, tossing halfway through, until toasted and light golden at the edges, 10 to 15 minutes.