Based on a consumer-available manual and my experience, there are 6-8 screws/fasteners that will need to be removed. (6 if you have the mudflaps).
2 pop-out fasteners under the cover - if you look along the bottom edge of the cover, you'll see 2 tabs that connect the cover to the underbody. Use a flat-head screwdriver to pop out the connectors.
2-4 screws - where the cover connects to the bottom of the rear wheel well, there are 4mm hex-head screws that will need to be removed.
2 interior screws - lift up the tailgate, and you'll see a screw in the left and right corners. Remove these screws and pop out the fastener.
4 Bolts hold on to the bumper cover - lift up the tailgate and take off the cover where the tailgate latch is. There are 4 black flat head fastners which keep the tailgate latch cover. After you take that off, you'll see the 4 Bolts that need to be taken off. (Highlander 2001)
After removing those, carefully start on the left or right sides, lifting up and pulling straight out to release the bumper from the clips.
The center part is tricky. I know there are 'slide' tabs. I broke mine off since I had no idea how they slid (stupid manual didn't mention it!). So take your time, and figure out which way to slide the cover off of the connecting tabs. I think you had to pull to the left on the left half, and right on the right side, but I'm not 100% sure. (THIS PARAGRAPH MIGHT BE FOR THE 2004 -- See the paragraph with 4 Bolts, this was what this original writer was writing about mine was for 2001, which should not make a difference.)
For my 2006 Toyota highlander the bumper came out easily so I didn't have to go through the process in the above paragraph 7.
November/09 addition: I just took the plastic rear bumper cover off my 2004 Highlander following the instructions above. It took 25 minutes. There are 4 rectangular tabs ( 1''x1/4") holding each side of the cover at the top behind the wheel wells. You can pry the cover off by using a flat head screw driver inserted between the cover and the metal side panel. Start with the tab closest to the wheel, then work toward the back.