Preheat the oven to 400F/200C. Halve the nectarines and discard the stones. Cut the nectarines into chunks and place in a roasting dish.
Sprinkle the brown sugar over the nectarine pieces then roast for approximately 15-20 minutes until the nectarines are soft and starting to caramelize on top a little. Remove from oven and allow to cool (can be prepared day ahead).
Once cooled, remove the skins from the nectarines and place the peeled chunks in a food processor or blender. Blend until relatively smooth (smaller chunks are fine).
Separately, whip the cream until it forms medium-firm peaks.
Carefully fold the nectarine puree and sweetened condensed milk into the cream so that it is combined (no white streaks) but without loosing too much air.
Pour the mixture into a plastic container or loaf tin, cover and transfer to the freezer.
Leave to freeze for 4 hours, minimum, for soft-serve or ideally overnight.