Preheat the oven to 325 degrees F. Butter a 9-inch baking dish.
Arrange the bread cubes tightly in the prepared baking dish. Beat together the butter, milk, vanilla and eggs in a large bowl. Add the granulated and brown sugars and mix until dissolved. Pour the mixture over the bread.
Put the baking dish on a baking sheet. Bake until the crust is golden brown, 55 to 70 minutes. Let rest for 20 minutes before serving.