This is simple to correct. The padded side panels are easily removed with a pry tool. You then remove the silver side strakes by unscrewing a few simple Philips-head screws (no stupid Torx screws on Lexi). If you want to then remove the main centre console, you pull up the Remote Touch and cupholder assembly (it just clips in). You then undo a hidden Philips screw on the drivers side underneath the transmission selector and unclip the console with a pry tool.
You will notice some felt padding from the factory. To improve it, all you need to do is buy some felt tape from Amazon then cover every clip and all areas where parts touch each other. When you reassemble, you should see a marked improvement.
The great thing with the Lexus interior is that the fundamental build is nice and tight, simplistic and solid. So when you do this kind of work, it pays off because the basic quality is good. I once tried to do this on a Jaguar XF and it was nigh on impossible to get it solid because the design was overcomplex and the parts used where flimsy.