Dear me - that sounds like a poor experience of VIP.
As for your summary of the ES, you are wrong. I went into my ES after 5 years in a GS450h and in most respects the ES is a better car. I have also owned several Lexus models and over my years of Lexus ownership driven almost all the range, sometimes for weeks at a time as courtesy cars, if not ones I owned for years. The ES is every bit a Lexus and is not full of “cheap hard plastics” - my GS and IS’s had far more, and my Avensis has plenty.
The Camry, which I have also test driven and spent time in, is nowhere near the quality, spec or car of the ES. It is much blander inside, lower spec and cheaper plastics.