Mix the ice cream, ginger, cinnamon and nutmeg in a large bowl until combined. Cover and freeze until ready to use.
Combine 1/2 cup heavy cream with the brandy in a small bowl; set aside. Line the bottom of a 9-inch-round springform pan with parchment paper. Arrange about one-third of the panettone in the pan in a single layer, cutting the pieces as needed to completely cover. Brush the panettone lightly with some of the brandy cream. Spread half of the spiced ice cream over the panettone layer. Repeat with another layer of panettone, brush with more cream and top with the remaining ice cream. Cover with the remaining panettone and brush with the remaining cream. Cover the pan with foil and freeze until firm, at least 4 hours or overnight.
Remove the cake from the freezer about 20 minutes before serving. Beat the remaining 2 cups heavy cream, the confectioners' sugar and vanilla in a large bowl with a mixer on medium-high speed until soft peaks form, about 2 minutes. Remove the springform ring. Cover the cake with the whipped cream and sprinkle with nutmeg.