(CNN) -- The man who led Mitt Romney's outreach to Hispanic voters said Sunday the candidate "made some mistakes" during his campaign that ultimately led to a precipitous drop in Latino support.

Carlos Gutierrez, the former secretary of commerce, blamed the Republican primary process, which he said forced Romney to harden his immigration stance in an appeal to the far-right wing of the Republican Party.

"Mitt Romney made some mistakes," Gutierrez told CNN senior congressional correspondent Dana Bash on "State of the Union." "I think he is an extraordinary man, and I think he made an extraordinary candidate. I think Mitt Romney's comments are a symptom. I think the disease is the fact that the far right of the party controls the primary process."

On immigration, Romney often sought to balance his positions in ways that appealed both to Hispanic voters and the base of the Republican Party.

In December, Romney vowed to veto the DREAM Act if he became president, saying instead he would support a path to residency -- not citizenship -- for undocumented immigrants who served in the military, but not other DREAM Act proposals.

Later, Romney gave a more detailed version of his stance, telling supporters at a fund-raiser in Florida that Republicans needed to offer their own version of the DREAM Act.

At a Republican presidential debate in January, Romney said he favored a system of "self-deportation," a policy that involves making economic conditions so difficult for undocumented workers that they choose to leave the country to find better opportunities. That stance was derided both by Democrats and his Republican rivals.

Speaking Sunday, Gutierrez said Latino voters were scared of a Republican Party they regarded as anti-immigrant and downright xenophobic.

"They were scared of the anti-immigration talk. They were scared of xenophobes. It's almost as if we're living in the past," Gutierrez added.

The proof, he said, was in the way Latinos voted: 27 percent cast ballots for Romney, compared to 31 percent who voted for Sen. John McCain in 2008 and 44 percent who supported George W. Bush in 2004.

"I would lay the blame squarely on the far-right wing of the Republican Party," Gutierrez said.

Evangelical leader Gary Bauer said Sunday that the opposite was true, pinning Romney's loss on a party too focused on appealing to moderate voters.

"I think it's due, at least in part, to folks in our party that seem intent on attacking the fact that we're the conservative party in the United States," Bauer said, pointing specifically to the issue of abortion, which he said should have played a larger role in the 2012 campaign.

"There's no yearning by the American people for a second pro-abortion party. I mean, we've already got one of those," he said.

™ & © 2012 Cable News Network, Inc., a Time Warner Company. All rights reserved.