While Ian Holloway's Pilgrims are proving a surprise package, Palace slumped to their fourth defeat in five home outings and Taylor was booed off at the final whistle.
Plymouth showed their worth with a well-organised performance, but they were hardly bothered by Palace side clearly low on confidence.
Chadwick got the decisive goal six minutes before the break, although they had a couple of good attempts before that.
David Norris forced Eagles goalkeeper Gabor Kiraly into a fine reflex save after 15 minutes and, ten minutes later, Sylvan Ebanks-Blake was left unmarked ten yards out and Kiraly was called into action again to prevent a goal.
A Plymouth goal seemed inevitable, but when it came it proved a gift as the Pilgrims capitalised on some sloppy defending by Darren Ward.
The Eagles had trouble dealing with a cross from the left, but after the panic Ward had time to clear, but instead got caught in possession.
Chadwick reacted quickly and lashed the ball into the roof of the net from 12 yards out.
Taylor made changes at half-time and there were signs of improvement.
Dougie Freedman forced a great save from Luke McCormick on 50 minutes - his first of the match - while substitute Shefki Kuqi missed his kick in front of goal.
But Plymouth also had their chances and skipper Paul Wotton fired just over.
Although the visitors hardly bothered Palace again, they were nonetheless always in control and Palace eventually ran out of ideas.
The frustration built up and at the final whistle Taylor left the dugout with his head down.
After four defeats in the last five home games the Palace fans are losing their patience. But for Plymouth's 1800 travelling supporters it was a long journey back to Devon, but it was a happy one.