You need to replace the complete throttle assembly since the idle speed controller is not sold separately. I had problems with the ISC and removed the throttle body and cleaned it with a throttle body cleaner spray, it worked fine for some months but then started failing again. Since I had experience with removing the throttle body, I just bought a new one from ebay, it cost me $100 plus shipping. I replaced the throttle body miself, re-used the hoses and gaskets and solved the problem. A new part will cost more then $300 plus labor, for sure they will want to replace gaskets and hoses, so plan for $800 if you take it to the dealer or maybe $500 at a local shop. Oh, I also obtained a copy of the Haynes repair manual just to make sure that I was putting everything in its place. The manual is available on the internet but it's hard to find.
The 2004 Toyota Avalon has a 4-speed automatic.
The 2001 Toyota Avalon has a 4-speed automatic.
The 2002 Toyota Avalon has a 4-speed automatic.
The 2003 Toyota Avalon has a 4-speed automatic.
The 2014 Toyota Avalon has a 6-speed shiftable automatic.
The 2005 Toyota Avalon has a 5-speed shiftable automatic.
The 2012 Toyota Avalon has a 6-speed shiftable automatic.
The 2013 Toyota Avalon has a 6-speed shiftable automatic.
The 2011 Toyota Avalon has a 6-speed shiftable automatic.
The 2009 Toyota Avalon has a 6-speed shiftable automatic.
The 2008 Toyota Avalon has a 6-speed shiftable automatic.
The 2010 Toyota Avalon has a 6-speed shiftable automatic.