Washington (CNN) -- More Secret Service resignations are expected this week in the wake of an alleged prostitution scandal in Colombia, according to a congressional staff member who participated in briefings with the agency.
Three Secret Service members already are leaving the agency, but the fallout continued Thursday with congressional demands for more details about what happened in Cartagena before last week's Summit of the Americas.
A total of 11 members of the special security agency that protects the president and other top officials have been linked by the agency to the controversy, including the three who are leaving.
One of those is a supervisory employee who is being allowed to retire, and another employee has resigned, the agency said.
A third agent, another supervisory employee, is being pushed out, with the agency proposing he be removed. A U.S. official said on condition of not being identified that the agent plans to fight his ouster.
Another eight members allegedly involved in the scandal are on administrative leave with their security clearances suspended, according to the agency.
The employees are accused of bringing prostitutes to a hotel in Colombia ahead of last week's visit by U.S. President Barack Obama, who was there to attend the pan-American summit.
Rep. Elijah Cummings, D-Georgia, told CNN that Secret Service Director Mark Sullivan told him Wednesday that a culture of pride at the agency would likely cause implicated agents to resign.
"I said, 'Is it possible that these men would resign?' He said he had no doubt that they would, that they probably would," Cummings said of his conversation with Sullivan that occurred before the first three departures were announced. "Why? Because of the culture. They have this pride, they don't want any bad apples and so it probably would be so uncomfortable to them that they would leave. So, yesterday's actions with regard to folks leaving and being fired did not surprise me one bit."
According to sources, the alleged prostitutes, the youngest of whom were in their early 20s, had all signed in at Cartagena's Hotel El Caribe, where the Secret Service members apparently stayed, flashing their local ID cards.
But one of the women was involved in a dispute about how much she was allegedly to be paid for the night.
That dispute brought the incident to light and sparked controversy in both countries.
A review board is expected to be created to determine whether the alleged scandal is an isolated incident or emblematic of a broader agency culture.
In addition, the House Oversight Committee leaders sent a letter to the Secret Service requesting specifics about what happened before, during and after the incident.
As many as 10 U.S. military personnel from all branches of the armed forces also are being questioned about potential misconduct, including five members of America's elite Army Special Forces.
The military members being investigated are not likely to deploy until the matter is resolved, military officials said Thursday. While no formal order bars their deployment, it's unlikely while investigators seek answers about what happened in Cartagena, the officials said.
The five Army Special Forces soldiers being questioned are from the 7th Special Forces Group based at Eglin Air Force Base in Florida, the officials said. The group operates mostly in South America, the officials said.
Its mission includes aiding foreign internal defense, special reconnaissance, counter-terrorism, counter-proliferation and unconventional warfare, according to information on the 7th Special Forces' website.
Obama has said he expects a "rigorous" investigation.
"The only way they will prevent this from happening again in the future is to find out if this is one particular case or if it's a pattern," said U.S. Rep. Peter King, R-New York, the chairman of the House Homeland Security Committee.
"We're working and doing our own investigation and whatever we need from the Secret Service we've been getting," King said Wednesday. "We want a minute-by-minute, hour-by-hour account of what happened, how it happened, what went on, who knew what was happening. And I have no doubt the Secret Service will give us that."
At least one congressman, U.S. Rep. Randy Forbes, R-Virginia, has called for Sullivan to be replaced.
"There's only so many strikes you get, in baseball it's three," said Forbes, a senior member of the House Armed Serves Committee, referencing a 2009 security breach in which a Virginia couple crashed Obama's first White House state dinner, as well as apparent agency overspending in that same year.
"I think he's had three," Forbes added. "I think it's time to put somebody else in there to make sure we're getting a different culture in the Secret Service."
However, King and others came to the defense of Sullivan, who has directed the Secret Service since May 2006 and been with the agency since 1983.
Senate Judiciary Committee Chairman Patrick Leahy, D-Vermont, said he was in close touch with Sullivan and believes the agency director is taking "serious action" to investigate the incident, while House Oversight Committee Chairman Rep. Darrell Issa, R-California, said he has a high level of confidence in Sullivan.
Sullivan has told subordinates to use "all tools available" to conduct the investigation, one source said.
The scandal is sure to come up when Homeland Security Secretary Janet Napolitano testifies next week at a previously scheduled Senate Judiciary Committee hearing on oversight of the Department of Homeland Security. The hearing is scheduled for 10 a.m. ET Wednesday.
The Secret Service agents and officers being investigated range in experience from relative newcomers to nearly 20-year veterans, two government officials with knowledge of the investigation said Monday.
Each agent was offered an opportunity to take a polygraph test, according to a U.S. official. Some of the agents and military personnel maintain they didn't know the women were prostitutes, the official said.
Even so, King said, "it was totally wrong to take a foreign national back to a hotel when the president is about to arrive."
While soliciting prostitution is in most cases legal for adults in Colombia, it is considered a breach of the Secret Service's conduct code, government sources said. Military law also bars service members from patronizing prostitutes, engaging in conduct unbecoming an officer or, for enlisted personnel, conduct "prejudicial to good order and discipline."
The military personnel allegedly involved were sent to Colombia to support the Secret Service. A military official who spoke on condition of anonymity because of the ongoing investigation told CNN that two of those being questioned are Marines and that Air Force and Navy personnel also are being questioned.