NEW YORK (CNNMoney) -- An Obama administration program is under fire, with federal investigators finding that community banks used the government's funds to pay back recession-era bailouts -- instead of lending the money to small businesses as originally intended.

The watchdog report released Tuesday found that $2.1 billion of the administration's $4 billion Small Business Lending Fund went to repaying bailouts. Many community banks were bailed out by the government in 2008 and 2009 under the Troubled Asset Relief Fund, or TARP, as it was commonly known.

CNNMoney has previously noted that the small business fund had failed to take off -- disbursing only $4 billion of the $30 billion it had originally carved out. The fund was established in 2011 to funnel cash to small firms, who were facing a borrowing crunch.

A significant number of banks used the small business lending program "to exit TARP using government funds... with little resulting benefit for small businesses," according to the report's author, Christy Romero, special inspector general for the Troubled Asset Relief Program.

The program's funds were cheaper -- with annual dividends of 1% for the first four years -- compared to the 5% to 9% for the TARP funds. To get the low rate, all a bank had to do was show a sufficient increase in its small business lending.

Romero found that the bailed out banks used 80% of the money to pay back TARP.

That's no surprise to Cliff McCauley, a banker in Texas who steered clear of both government programs.

"Everyone went in thinking it was one of the ways to pay back TARP," McCauley said. "It was disguised as promoting to encourage business lending."

McCauley, a senior executive vice president of Frost Bank in San Antonio, said the inspector's finding was inevitable. He said banks that had been bailed out were desperately looking for ways to pay back the expensive government bailout funds, especially at a time when the economy had just emerged from a recession and barely limping along.

About 332 banks and community lending groups took part in the small business lending program. Of those, 137 banks used those funds to pay back bailout money.

The Treasury Department said there's nothing wrong with the way the banks used the funds. In a memo last month, Deputy Assistant Secretary Don Graves pointed out that Congress intended that when it set up the fund.

Treasury also took issue with the report's findings that the program was ineffective, noting that 84% of TARP banks increased their small business lending. And their median lending increase was 18%.

However, Romero points out that the TARP banks received $2.7 billion in funds, and they increased small business lending by $3 billion -- or $1.13 for every dollar they received. What's more, 24 bailed-out banks in the program did not increase their small business lending at all.

Meanwhile, the non-TARP banks received $1.2 billion in funds and increased small business lending by $4.2 billion.