Palace boss Neil Warnock made five changes to the side that drew 0-0 with Watford at the weekend.
Darryl Flahavan made his debut in goal, while there was also a first start in Palace colours for former Reading and Everton winger Oster following his arrival at the beginning of the week on a free transfer.
Hereford were looking to bounce back from Saturday's season-opening 2-1 defeat to Leyton Orient and the newly-promoted League One side carved out the first opportunity of the match when Matt Done tested Flahavan with a long-range strike on 17 minutes.
Robbie Threlfall curled a 20-yard shot over the bar as the visitors continued to offer more of a threat in a low-key opening and it was little surprise when they took the lead four minutes before the interval.
Kris Taylor crossed from the left and on-loan Watford striker Moses Ashikodi slipped in between Jose Fonte and Paddy McCarthy to head home off the crossbar to register his first goal for the Bulls.
However, the goal seemed to spur the home side into life as Carle drew his side level just moments after pulling an initial effort from Oster's cut-back wildly over the bar.
This time the summer signing from Bristol City found the bottom corner from just outside the penalty to restore parity with virtually the last kick of the first half.
Palace then struck the game's decisive moment with the second period barely minutes old.
Oster flighted a cross from the right over the head of Hereford goalkeeper Darren Randolph and into the top left-hand corner in the 49th minute to illuminate his already impressive debut.
Young midfielder Kieron Djilali came on midway through the second half for his first-team debut and looked lively with some darting runs.
But it was Oster who continued to terrorise the United defence as he picked out Carle in the 75th minute with a sublime long-range pass, but the Australian midfielder could only fire his shot into the side-netting.
However, the miss was little more than academic as the home side successfully played out time to book their place in the second round with relative ease.