Combine milk, ground almonds and sugar in a medium saucepan and bring to the boil. Remove from heat and let steep for 10 minutes.
Immerse gelatine leaves in a small bowl of cold water and leave till softened.
Remove gelatine sheets from water and drain. Add gelatine and orange blossom water to the hot milk mixture and stir till the gelatine is fully dissolved. Strain into a bowl or serving dish and stir in crème fraîche till incorporated. Cover with cling film and refrigerate for at least 2 hours.
Preheat oven to 200 C / Gas 6 and line baking tray with greaseproof paper.
In a bowl mix 125g almonds with softened butter and icing sugar. Spread mixture on baking tray and bake in preheated oven until caramelised, about 10 minutes. Remove from oven and let cool.
Break up almonds and serve blancmange topped with almond brittle.