Rather than olive oil, I use mineral oil (not mineral spirits) to remove sticker residue. Just as Alex explained when using olive oil, mineral oil also takes time to soak in, as well as a bit of patience and persistence keep rubbing until all the sticker residue is liquified by the oil. For me, mineral oil has always worked in about 5 minutes or less, and has never harmed any surface I cleaned.
As the final step, I prefer to use Windex, rather than dish soap, to remove the oil I put on the surface.
My complete set of Apple product cleaning "tricks" consists of Windex, mineral oil, and