Preheat oven to 350°F. Spread all of the almonds (1½ cups) onto a cookie sheet and bake for 4 to 5 minutes or until the nuts just begin to brown and become aromatic. Remove from oven and let cool. Line the bottoms of two 9-inch cake pans with parchment or waxed paper and spray with nonstick cooking spray.
Prepare the cake batter: Combine the flour, baking powder and salt in a medium bowl; set aside. Beat the butter and sugar in the bowl of an electric mixer fitted with the paddle attachment for 5 to 8 minutes or until smooth. Add the eggs, one at a time, scraping between additions. Stir in the flour mix, almond extract and ½ cup of the toasted almonds until just blended. Do not over mix.
Bake the cake: Fill each of the prepared pans with half of the batter. Place the pans in the center of the oven. Bake until the center is firm or when the tip of a small knife emerges clean when it pierces the center of each cake, 30 to 35 minutes. Remove from the oven and cool.
Make the lemon curd: While the cake is baking, whisk the sugar, eggs, lemon juice and zest together in a microwave-safe bowl. Add the butter. Microwave for 4 to 6 minutes, stirring every minute until the mixture thickens. Cover the bowl with plastic wrap and poke a few holes into the wrap with the tip of a knife to allow steam to escape. Refrigerate until completely cooled.
Prepare the cakes, fruit and almonds: Slice the rounded tops of the cakes to flatten them. Carefully slice the cakes in half, horizontally, through the middle to create 4 layers. Stir the raspberry jam with 1 tablespoon of water to loosen the texture and toss gently with the raspberries. Whip the cream to medium peaks.
Assemble the trifle: Place one cake round in the bottom of the trifle bowl, trimming the edges if necessary to fit. Spread about 1/3 of the lemon curd over the cake. Cover with about ½ cup of whipped cream and sprinkle with ¼ cup of the toasted almonds, ¼ of the raspberry mixture and 1 cup of blueberries. Repeat the same process for two more layers. For the last layer, spread the cake with the remaining whipped cream, berries and nuts.
If you want to make it easier, use a cake mix or two store-bought 9-inch cakes.