Preheat the oven to 350 degrees, and spray a standard loaf pan with gluten-free cooking spray.
Line the pan with a layer of parchment that goes up the sides and drapes over a bit. Spray again with cooking spray and set aside.
Beat the butter and sugar together in a stand mixer fitted with the paddle attachment on high speed for 5 minutes, until it is pale, light and fluffy. Add the eggs, one by one, scraping down the sides of the bowl as necessary.
Whisk the flour, baking powder, cardamom, xanthan, and salt together in a medium bowl. Add to the butter mixture (while beating on low), alternately with the creme fraiche and milk.
Scrape down the sides of the bowl to make sure everything is evenly incorporated and pour into prepared pan, smoothing the top. Sprinkle the slivered almonds evenly onto the cake.
Bake for 60-65 minutes, or until a toothpick inserted into the center of the cake comes out clean. Cool in the pan for 10 minutes, then remove to finish cooling on a rack.
To serve, beat the cream until soft peaks form. Add the sugar, creme fraiche and vanilla and beat until combined. Serve the cake with sliced fruit and the creme fraiche whipped cream.