Heat the milk and lavender in a saucepan over low heat until warmed through. Remove from heat, and allow lavender to infuse for about 20 minutes. Remove flowers.
Beat the egg yolks and sugar together in large bowl. Whisk the lavender-infused milk into the egg mixture, then pour it back into the saucepan. Heat the mixture over low heat, whisking constantly, until the mixture thickens and can coat the back of a spoon, 7 to 10 minutes. Remove from heat and cool slightly, about 5 minutes. Stir in the double cream. Transfer the mixture to a bowl, and chill in fridge for at least 4 hours.
Pour the chilled mixture into an ice cream maker and freeze according to manufacturer's directions until it reaches "soft-serve" consistency. Transfer ice cream to a lidded container; cover surface with cling film and seal. For best results, ice cream should be place in the freezer at least 2 hours or overnight.