In a heavy saucepot, over medium-low heat, cook the flour and butter, stirring constantly until roux achieves a light caramel color. Add the broth and brown sugar, bring to a boil, then lower to a simmer. Stir in the sweet potatoes and spices, bring to a simmer again, and cook for 5 minutes more. In a blender, puree the soup in batches and return to saucepot. Add the milk, and reheat soup. Season with salt and pepper, ladle into warm soup bowls and serve.