Make dipping mixture: Whisk 2 eggs, milk, vanilla, cinnamon, and nutmeg in wide bowl. Refrigerate until ready to use.
Peel potatoes and cover with cold water in pot. Season with salt. Bring to a boil; lower heat and simmer for 5 minutes; turn off burner and let sit 15-20 minutes. Drain; rinse potatoes with cold water. When cool, grate with the large holes of box grater. Heat heavy skillet, preferably cast iron, over high heat. Add canola oil; heat. Spread shredded potatoes evenly over skillet bottom. Season with salt and pepper and cook for 5 minutes or until brown on bottom. Do not stir or disturb potatoes while cooking. Using a wide spatula, flip potatoes over and brown other side without stirring.
Fry bacon in skillet until brown and crispy. Remove; drain on paper towels. Return skillet to stove over medium heat and fry remaining 4 eggs. Remove eggs to a plate. Keep warm.
Heat griddle or skillet over medium heat and add butter. Remove dipping mixture from refrigerator. Rewhisk to blend well. Dip the bread slices, one at a time, in mixture. Grill one side. Flip. Brown second side. Top 4 slices each with 1 ounce Brie, a portion of the potatoes, 1 egg, two slices of bacon, and another ounce of Brie. Top with another prepared bread slice. Continue to grill sandwiches briefly to heat the Brie, being careful not to overbrown bread. Serve with maple syrup on the side, if desired.