clay barring does make all the difference,i clay my car once maybe twice a year thats all,i would clay bar the fuel flap and polish and wax it,then have a feel of the paint around the fuel flap next have a feel of the fuel flap,the difference will be apparent.i work near an oil refinery so my car gets covered with contaminants, you can feel the paint becomming rough from the impurities in the air landing on the paint.
there is one golden rule when claying if you drop the clay bar throw it out do not use it anymore you will of picked up some dirt / grit that will damage the paint, for this reason its a good idea to cut the bar in half.