I wait until the cherry gets near the band. I think the heat helps loosen the glue.

I had a Sam Leccia White where I tried to take it off early and the band was damaged (the wrapper cracked a little, but the damage was just cosmetic.