The Eagles boss would have been happy enough at half time, having seen his side take a deserved lead through Dougie Freedman. However, a string of missed chances enabled Rovers to get back into the game and hang on to finish the job on penalties.
Troubled striker Shefki Kuqi was the main culprit, missing two gilt-edged headed opportunities - as well as blowing a penalty during the shootout.
"We had good chances and didn't take them," said Taylor. "If we had, we wouldn't have had to worry about a penalty shoot-out.
"But Bristol Rovers were better with the penalties than we were. You have to score your own and save a few and we didn't do either."