Washington (CNN) -- The White House defended the Secret Service and its director Tuesday amid an embarrassing investigation into whether several agents brought prostitutes back to their hotel in Colombia ahead of a presidential visit.
Eleven Secret Service members have been implicated in the investigation, which began Thursday after one of the women complained that she hadn't been paid. In addition, as many as 10 U.S. military personnel from all branches of the armed forces are being questioned about potential involvement in any misconduct, two military officials told CNN.
The Americans were in Cartagena to prepare for President Barack Obama's weekend visit to the Summit of the Americas, and Obama has said he expects a "rigorous" investigation.
The investigation is being led by Secret Service Director Mark Sullivan, who has been briefing members of Congress. A leading senator said Tuesday she had been told as many as 21 women had been involved, and questioned whether the incident could have endangered the president.
"Who were these women? Could they have been members of groups hostile to the United States? Could they have planted bugs, disabled weapons, or in any other (ways) jeopardized security of the president or our country?" asked Maine's Susan Collins, the ranking Republican on the Senate Homeland Security and Governmental Affairs Committee.
At the White House, presidential spokesman Jay Carney said Obama "has confidence" in Sullivan, who he said "acted quickly in response to this incident," and in the agents around him.
"The work the Secret Service does, the men and women who protect him and his family and those who work with him, is exemplary as a rule," Carney said. "They put their lives on the line, and it is a very difficult job, and he acknowledges that and he appreciates it."
Collins said she believed Sullivan "will fully investigate" the allegations and take "appropriate action" if the allegations bear out. But she questioned whether there was any similar misconduct on previous missions, and whether the issue is a sign of a deeper problem within the agency.
The Secret Service agents and officers involved range in experience from relative newcomers to nearly 20-year veterans, and all have been interviewed at least once, two government officials with knowledge of the investigation told CNN on Monday. Their security clearances have been pulled while the investigation is under way and could be reinstated if they are cleared, the officials said.
The agents were offered an opportunity to take a polygraph test, according to a U.S. official.
Some of the agents and military personnel maintain they didn't know the women were prostitutes, the official told CNN.
"Even if they weren't (prostitutes), it was totally wrong to take a foreign national back to a hotel when the president is about to arrive," Rep. Peter King, R-New York and the chairman of the House Homeland Security Committee, told CNN's "Piers Morgan Tonight."
House Oversight Committee Chairman Rep. Darrell Issa, R-California, said he thinks the agents should take the polygraph tests, if they haven't already done so.
"For these individuals, if they want to have any career at all, they have to decide on telling the entire truth and seeing whether they have something going forward."
Issa said his level of confidence in Sullivan is "high."
The Homeland Security Committee's chairman, Connecticut independent Joe Lieberman, said his staff is looking into the accusations and he may call hearings on the matter.
"History, unfortunately, is full of cases where people in positions of great responsibility, including security, have been compromised by, well, enemies or spies," Lieberman said. "I'm not saying that happened here, but once you conduct yourself in this way, you open that risk."
California Democrat Sen. Dianne Feinstein, who leads the chamber's intelligence committee, said she was "profoundly disappointed" by the allegations.
"I've always respected the Secret Service as kind of numero uno of our law enforcement community," she said.
U.S. government sources have said there was a dispute between at least one Secret Service member and a woman demanding payment. At least one of the women brought to the hotel talked with police, and complaints were filed with the U.S. Embassy, the sources said.
While soliciting prostitution is legal in certain areas of Colombia, it is considered a breach of the agency's conduct code, the government sources said. Military law also bars service members from patronizing prostitutes, displaying conduct unbecoming an officer or, for enlisted personnel, conduct "prejudicial to good order and discipline."
The military personnel involved were sent to Colombia to support the Secret Service. A military official who spoke on condition of anonymity due to the ongoing investigation told CNN that two of those being questioned are Marines who handle military working dogs. Air Force and Navy personnel, some of whom are believed to be explosive disposal experts, also are being questioned, the official said.
The alleged misconduct occurred before Obama arrived in Cartagena, and the Secret Service said the personnel involved were relieved of duty and sent home before the president landed. But the news broke while he was there -- and Gen. Martin Dempsey, chairman of the Joint Chiefs of Staff, told reporters at the Pentagon on Monday that the incident distracted attention "from what was a very important regional engagement for our president."
"So we let the boss down, because nobody's talking about what went on in Colombia other than this incident," Dempsey said.
Senate Armed Services Committee Chairman Carl Levin, D-Michigan, said he would consider holding hearings on the conduct of the members of the military involved in the scandal, but wants to learn more first. The committee's top Republican, Arizona Sen. John McCain, said he is sure "the guilty will be punished," but lamented that a few members of the military and Secret Service "have tarnished the reputations of many."
CNN's Barbara Starr, Jessica Yellin, Deirdre Walsh and Ted Barrett contributed to this report.