For the cake: Preheat oven to 350 degrees F and grease a 9-by-4 1/2-inch loaf pan with cooking spray.
Toast the fennel seeds in a dry pan over medium-low heat until fragrant, then grind up in a spice grinder.
Beat the butter and granulated sugar together with an electric mixer until smooth and white.
Slowly add the eggs and vanilla to combine well, beating well after each egg is added.
Add the flour, salt and fennel seed and mix until just combined. (Do not overmix!)
Pour batter into greased pan and bake until a toothpick inserted in the center comes out clean, 40 to 50 minutes. Allow to cool in pan for about 10 minutes. Remove from pan and transfer to a wire rack to cool to room temperature, about 45 minutes.
For the creme: Place the creme fraiche, granulated sugar and vanilla in the bowl of a stand mixer and whip until soft peaks form, about 5 minutes. Set aside.
For the berries: Slice the blackberries and strawberries in half and mix with the blueberries and raspberries. Toss in granulated sugar, vanilla and salt, and let sit for at least 10 minutes at room temperature or refrigerated up to overnight.
Tear the cake into large chunks. Place 1 or 2 large chunks on each plate, and top with a spoonful of the berries, making sure to get some of the liquid. Place a dollop of the creme over that. Dust with powdered sugar and garnish with chervil.