Preheat the oven to 350 degrees F. Generously butter a 13 by 9-inch baking dish.

In a medium saucepan, combine the milk, sugars, sweet potato puree, vanilla, cinnamon, and nutmeg. Cook over medium-high heat, just until the mixture begins to boil. Remove from the heat and set aside.

In a medium bowl, whisk the egg yolks until smooth and combined. Using a ladle, pour 1/2 cup of the hot milk mixture in a slow, steady stream into the yolks, whisking constantly. Add another 1/2 cup of the hot milk mixture and whisk to combine. Pour the egg mixture into the milk mixture. Cook over medium-high heat, stirring constantly, until it's thick enough to coat the back of a wooden spoon, about 3 to 5 minutes. Strain the custard and reserve.

Cover the bottom of the baking dish with 1 layer of bread slices. Pour about 1/4 of the custard mixture over the bread. Repeat with remaining bread and custard, ending with custard. Cover and refrigerate for 8 hours or overnight.

Sprinkle the bread pudding with the pecan crumble. Bake until the crumble mixture is golden and the custard is bubbly, about 45 minutes. Serve warm.

Pecan Crumble:

In a medium bowl, combine all the ingredients, and mix with a hand mixer on low speed, until well combined.