Will this plug directly into my factory wiring harness?

Yes. This is an OEM part with the original connector pinout. Connect it to the existing mirror harness under your door panel and it will work without modification or splicing.

Do I need to replace both mirrors, or just the left?

You only need the left mirror if that's the side you're replacing. The right mirror can stay in place. Both sides are independent circuits, so there's no matching requirement.

What if the mirror glass is damaged but the motor and housing are fine?

Only the glass can be replaced separately through a dealership. This listing is the complete mirror assembly, so if you only need glass, contact a Toyota parts counter for a standalone pane replacement.