It's concerning to hear of the fine metallic particles being present in the fluid, the sump is normally fitted with several magnets to pull these out of the fluid in the sump.
The drain refill process is a pain with around only 2 litres being drained on each occasion, it can be flushed via the oil cooler but there are reports both good and bad regarding this method as previously trapped contaminants can be released into the flow of fluid.
As Colin mentioned the Aisin transmissions are fitted with a fluid temperature sensor which prevents upshifts to top gear until the transmission reaches a certain temperature in order to protect it.
The main thing is to ensure the specified Toyota fluid only is used that being Type IV.
If you do replace the filter block be aware that the sump requires a special sealant for refitting and sump bolt torque settings are crital to prevent future leaks.