I tackled this job on my last IS250. The original motive was to replace the puddle light bulbs with leds, but the passenger mirror wasn't fully performing.
When I got inside, I saw the gears of the mechanism all caked in crud and gunge !
I cleaned the parts with white spirit using an old toothbrush, dried it all, then sprayed with a silicon lubricant. Changed the bulbs too.
Yes, the initial dislodging of the mirror glass is nerve wrenching, but once over that everything went swimmingly.
It's all about a bit of confidence. All worked fine afterwards.