Preheat the oven to 190 C / gas 5. Grease and flour a 23x33cm baking tin. Sift together the flour, semolina and baking powder. Set aside.
Cream the butter and sugar until light and fluffy. Add the egg yolks one at a time, beating well after each addition, until incorporated. Stir in the orange juice, orange zest, vanilla extract and brandy.
Beat the egg whites in a clean, dry bowl until medium peaks form. Fold 1/3 of the beaten egg whites into the butter-sugar-egg yolk mixture until combined. Gently stir in 1/3 of the flour mixture. Continue adding the remaining egg whites alternately with the flour, folding in until just combined.
Stir in the coarsely chopped and finely chopped almonds. Pour cake mixture into the prepared baking tin and sprinkle with flaked almonds.
Bake in the preheated oven for 30 minutes. Reduce the heat to 180 C / gas 4 and bake until a skewer inserted into the centre of the cake comes out clean, about 15 minutes. Remove from oven and sprinkle with the tablespoon of brandy, if desired. Allow to cool completely before serving.