WASHINGTON (CNN) — Sandra Day O'Connor noticed a disturbing development as her last day on the Supreme Court neared. Over her final years on the bench, more people were talking about "activist judges," an issue she said that appeared to be "erupting all over the country."

"That seemed to be a mantra of some kind," she told CNN.

Members of the Supreme Court are notoriously press-shy, but O'Connor, who retired in January, and Justice Stephen Breyer, a 13-year veteran, recently spoke candidly to CNN about their concerns about judicial independence and political retribution against judges.

"I saw increasing indicators of unhappiness with judges," O'Connor told CNN's Jeffrey Toobin as part of the network's "Broken Government" series. "We heard all kinds of statements by members of Congress, by state legislators. We saw legislation introduced to somehow restrict or affect judges at both the state and federal levels."

O'Connor's bluntness was tempered somewhat by Breyer's thoughtfulness, not surprising given he still sits on the bench.

"The fact is, once I'm appointed, I'm not a judge for one group or another group," he said. "And when I write a dissent or when I write a majority, the people that disagree with me the most — I'm their justice too."

Some feel the judiciary needs to be free to keep the other branches of government in check.

"We're in a very perilous time when the independence of the courts is vital to making sure the president doesn't go too far, even in an act of well-meaning zeal, to keep the fabric of our Constitution together," said Kathleen Sullivan, former dean of Stanford Law School.

In 2005, two court cases that caught the public's attention and the president's choices for new Supreme Court justices made the judiciary a political fireball, prompting an intense debate on how judges should operate.

The Terri Schiavo case turned on who, if anyone, had the legal right to allow Schiavo, who was ruled by lower courts to be in a "persistent vegetative state," to die. Despite attempts from Republican lawmakers and President Bush to intervene by passing a law pertaining to the case, the Supreme Court ultimately refused to hear the matter, and she died days after her feeding tube was removed.

Many members of Congress were, livid and their criticism troubled O'Connor.

"It was asking for review of one specific case," she said. "That's so unusual. Usually laws are passed that govern a category of matters for courts to decide, not case by case. ...

"It then came to this court and that resulted in a great deal of unspoken sentiment — unhappy with the courts for how they had dealt with it."

A second case that raised public hackles involved the right of local governments to seize private land for private development. In general, under a practice known as "eminent domain," a person's property may be condemned and the land converted for a greater "public use," like highways, schools, or revitalizing blighted areas.

In this case, however, a well-kept residential neighborhood was being forced to make way for a private shopping development.

The Supreme Court narrowly concluded that local governments could use eminent domain to seize private land to use for private economic development if the seizure was for a greater public purpose.

Critics said judges were injecting their beliefs into decisions best left to the legislative and executive branch.

They called it "judicial activism" and called for a check on the fundamental business of the courts.

"Quite frankly, the judiciary could use an inspector general," Sen. Charles Grassley, an Iowa Republican and member of the Judiciary Committee, told CNN.

O'Connor disagrees, saying a new regulatory system is an odd concept to her.

"I just think it might lead to some unfortunate relations between the branches of government," she said.

The third dynamic came when the president chose two judges, Chief Justice John Roberts to replace the late William Rehnquist and Justice Samuel Alito to replace O'Connor.

But a seven-month debate filled with endless issue ads, rallies, and Web postings from conservatives and liberals only fueled the partisan tone of the nominations, polarizing an already divided electorate.

"The system has gotten more political," said Thomas Goldstein, an attorney and Supreme Court legal analyst. "But we fortunately haven't gotten to the point where the process has completely broken down."

The system, while shaky, still works, he said.

Some conservatives believe many judges have become too sensitive to criticism.

In a recent op-ed in the Wall Street Journal, federal appeals court Judge William Pryor called current complaints against the courts "relatively mild" compared to the past. His solution?

"Judges must do more than respond to criticisms; we must exercise restraint," he wrote. "Judges have a unique responsibility to safeguard our independence. It is not too much for us to look in the mirror and ask whether some criticisms are fair."

O'Connor believes it is extremely important to try to maintain a stellar perception of the American legal system around the world.

"We espouse the notion in this country, and around the world, that our best hope for peace is having all nations abide by the rule of law," she said. "And our federal courts have been a rather good example for other nations around the world.

"And to see our courts now, under such serious attack, is a concern to me."

Breyer argued it was the intent of the nation's founders that the judiciary — an independent group insulated from politics — should be the final arbiter on matters of law.

"The best guarantee that minorities will not be oppressed, that the Constitution will be lived up to, is to give that very last word — under narrow circumstances — to a group of judges," he said.

"Someone has to have the last word."