Attorney General Eric Holder said Wednesday that his decision to remove himself from the Justice Department investigation into a leak that led it to surreptitiously collect telephone records from the Associated Press leaves him unable to respond to questions about it.

"I don't know what happened there with the intersection between the AP and the Justice Department," Holder told the House Judiciary Committee. "I was recused from the case."

The news agency revealed Monday that federal agents had secretly collected two months of telephone records for some of its reporters and editors.

The AP said agents were apparently investigating the source of a story revealing that the CIA had thwarted an al Qaeda plot to blow up a U.S.-bound jetliner in May 2012, around the anniversary of the killing of Osama bin Laden.

Sources later told CNN that the operative who was supposed to have carried the bomb had been inserted into al Qaeda's Yemeni affiliate by Saudi intelligence, and that the device had been handed over to U.S. analysts.

Holder told the hearing that he had recused himself to avoid any potential conflict of interest in the case and had left the decision to subpoena the phone records to Deputy Attorney General James M. Cole, who authorized the subpoena.

"I recused myself because I thought it would be inappropriate and have a bad appearance to be a person who was a fact witness in the case to actually lead the investigation given the fact, unlike Mr. Cole, that I have a greater interaction with members of the press than he does," Holder said.

Asked what made him a fact witness, he said, "I am a possessor of the information that was ultimately leaked. And the question then is who of those people who possessed that information, which was a relatively limited number of people within the Justice Department, who of those people, who of those possessors actually spoke in an inappropriate way to members of the Associated Press."

Asked who else had access to the information, Holder cited the ongoing nature of the investigation. "I would not want to reveal what I know and I don't know if there are other people who have been developed as possible recipients or possessors of that information during the course of this investigation," he said. "I don't know."

Answers like those prompted some sharp criticism of Holder and his deputy from Wisconsin Republican Rep. James Sensenbrenner, a former Judiciary chairman.

"There doesn't seem to be any acceptance of responsibility in the Justice Department for things that have gone wrong," Sensenbrenner said. "Now may I suggest that you and Mr. Cole and maybe a few other people go to the Truman Library and take a picture of the thing that he had on his desk that said 'The Buck Stops Here?' Because we don't know where the buck stops, and I think to do adequate oversight, we better find out and we better find out how this mess happened."

And a Democrat, New York's Hakeem Jeffries, said the subpoenas appeared to be "overly broad in scope."

"Hopefully that is something that the investigation that takes place will examine with close scrutiny," he said. "And second, that I think, as many of my colleagues have expressed, I'm also troubled by the fact that the negotiation or consultation with the AP did not occur in advance of the decision to issue the subpoenas."

Holder said he recused himself because he had been questioned by FBI agents as part of the leak probe. He Tuesday that the leak, which he did not describe, had put Americans at risk and demanded "very aggressive action."

But AP Executive Editor Kathleen Carroll said Tuesday that the records collected by the investigation cast a "very broad net" and involved AP operations "that have, as far as I know, no particular connection to the story that they (federal authorities) seem to be investigating."

"We've never seen anything along the size and scope of this particular investigation," she told CNN's Erin Burnett.

In all, the AP says, federal agents collected records involving more than 20 phone lines -- including personal lines -- used by about 100 journalists in New York; Hartford, Connecticut; and Washington.

The Justice Department on Tuesday defended its decision to subpoena the records, saying the requests were limited and necessary.

"We are required to negotiate with the media organization in advance of issuing the subpoenas unless doing so would pose a substantial threat to the integrity of the investigation," Cole wrote in a letter to the AP. "We take this policy, and the interests that it is intended to protect, very seriously and followed it in this matter."

White House spokesman Jay Carney said White House officials were not involved in the investigation and knew nothing about the AP inquiry.

The Obama administration has launched several high-profile leak probes, leading to the prosecution of two government employees accused of revealing classified information.

Thomas Drake, a former National Security Agency official, was sentenced to one year of probation and 240 hours of community service in 2011, while former CIA officer John Kiriakou was sentenced to 30 months in prison after admitting to identifying a covert intelligence officer.

In 2002, Iowa Sen. Charles Grassley criticized the Justice Department for its subpoenas of John Solomon, an AP writer who had written about an investigation into then-Sen. Robert Torricelli, D-New Jersey. The subpoena also spurred a protest from the journalism association Investigative Reporters and Editors.

Documentary filmmaker Robert Greenwald, whose movies have criticized the war in Iraq, Fox News and Walmart, called the administration's pursuit of leakers "an effort to silence and scare whistleblowers, and to get the press to be quiet and do what it wants them to do."

"This is a systemic, continuing problem," said Greenwald, whose latest film, "War on Whistleblowers," focuses on the issue. "It's not a one-off, and it's not an accident, sadly."

CNN legal analyst Jeffrey Toobin said Monday that the subpoenas were likely legal, but they go further than previous administrations in pursuing private information of journalists.

"I have never heard of a subpoena this broad," Toobin said.

CNN's Tom Cohen reported from Washington. Michael Pearson reported from Atlanta. CNN's Matt Smith, Jessica Yellin, Carol Cratty, Kevin Bohn, Greg Botelho, Joe Sterling and Josh Levs contributed to this report.