WASHINGTON (CNN) -- A war opponent's chance encounter with then-Vice President Dick Cheney -- which triggered an arrest and a lawsuit -- saw his appeal rejected by the Supreme Court Monday.

The case tested the balance between free speech and security concerns for top government officials.

The justices, by a unanimous vote, ruled in favor of two U.S. Secret Service agents, who remain shielded from a lawsuit filed by Steven Howards of Golden, Colorado. The man was arrested after confronting Cheney in a public area in 2006 -- making physical contact with the vice president -- and announcing his disagreement over the Iraqi war.

At issue was whether the agents deserved immunity as government employees. Believing there was probable cause, they detained the 59-year-old environmental consultant.

"This court has never recognized a First Amendment right to be free from a retaliatory arrest that is supported by probable cause," wrote Justice Clarence Thomas. "Nor was such a right otherwise clearly established at the time of Howards' arrest."

Howards claims his detention was in retaliation for his political views.

The incident occurred at Beaver Creek Mall in the Colorado mountain resort town of the same name. Howards was taking his 8-year-old son to a piano recital when he noticed Cheney coming out of a grocery store, accompanied by his security detail.

Howards used his cell phone to note the vice president was shaking hands with passers-by, and stated -- according to court records -- "I'm going to ask him (Cheney) how many kids he's killed today," an apparent reference to casualties in the Iraq conflict.

That remark was overheard by one of the agents. Howards let his son continue walking to the recital while he waited to speak with the vice president. The protester eventually told Cheney the administration's "policies in Iraq are disgusting," and then placed his open hand on Cheney's shoulder.

There is much dispute over whether that contact represented a "pat" as Howards later claimed, or a "shove" as some agents interpreted it. The touching alone did not lead to the man's immediate arrest, but he was later taken aside and questioned.

Howards at first refused to talk, then strongly denied touching Cheney. He also repeated his views on the war. "If you don't want other people sharing their opinions, you should have him (Cheney) avoid public places," he said, according to court records.

Agent Virgil "Gus" Reichle, who had been dispatched to do the questioning, became "visibly angry" at those remarks, according to the lower court ruling. He admitted later not overhearing the cell phone conversation -- nor witnessing the shoulder contact -- but said he had been briefed by fellow agents. Reichle was the detail's intelligence coordinator, and was dressed in plain clothes.

Howards was then arrested for assaulting the vice president, but he was never prosecuted for that or for a separate charge of harassment. He sued Reichle and another agent for alleged civil rights violations, and a federal appeals court in Denver tentatively allowed the case to proceed.

That three-judge panel said Howard's initial denial of the touching was sufficient reason -- or "probable cause" -- for agents to arrest him, but also concluded, on balance, the man's First Amendment rights were violated in the process.

The Justice Department urged the high court to reverse, saying protective details must often make lightning-fast judgments of life and death for top government officials. Those agents, said the Obama administration, should not err on the side of caution when handling potential threats for fear of being sued later.

The high court agreed. "An officer might bear animus toward the content of a suspect's speech," Thomas said. "But the officer may decide to arrest the suspect because the speech provides evidence of a crime or suggests a potential threat."

Justices Ruth Bader Ginsburg and Stephen Breyer agreed these Secret Service officials were not liable, but said other law enforcement agents in some cases may be held accountable for retaliatory arrests.

The justices five years ago ruled in a separate appeal that an individual claiming to have been prosecuted in retaliation for exercising his rights must show that government officials lacked probable cause when bringing criminal charges. The issue here was whether that rule applies to retaliatory arrests.

Justice Elena Kagan did not participate in this case, since she apparently worked on the government's legal strategy while serving as the Justice Department's solicitor general, before being nominated to the court in May 2010.

The case is Reichle v. Howards (11-262).