(CNN) -- The White House wanted to have 10,000 active duty troops on the streets of Washington and other cities earlier this week to quell protesters, but Secretary of Defense Mark Esper and Chairman of the Joint of Chiefs of Staff General Mark Milley pushed back at the use of any active duty troops, according to a senior defense official.
Esper did move approximately 1,600 active duty troops to be in the Washington, DC, region to respond if needed but the approximately 5,000 National Guard troops already there never needed assistance and the active troops began to leave Thursday night.
Recommended Videos
A second defense official said Milley strongly felt the threshold -- informally described as dire circumstances -- for calling in active duty troops could not be met, opening the door to whether such a potential presidential order would be legal.
CBS first reported that the White House wanted 10,000 troops.
CNN has reached out to the White House for comment.
After violence in DC on Monday, along with a militarized law enforcement presence, Milley held phone calls over two days with President Donald Trump's arch political enemies, the Democratic leaders of Congress.
A congressional source confirmed a Milley call with Senate Democrat leader Senator Chuck Schumer on Monday. A defense official also confirmed a call with House Speaker Nancy Pelosi on Tuesday. The official said several other calls were made by Milley to Congress.
While the exact discussions with the Democratic congressional leaders have not been disclosed, by Monday night Milley had already had tense words with Trump that active duty military troops against protesters in DC was absolutely not needed despite the President's threats, several officials confirmed.
The second defense official said despite a Saturday White House statement of support for Esper, both men remain aware they are at risk of Trump's ire and there is always the possibility their jobs are at risk.
The outreach to Congress is part of a broader effort by both men to explain their actions and try to move forward in the face of massive personal criticism from former colleagues and members of Congress. So far, the Pentagon has not agreed to a request for them to testify before the House Armed Services Committee.
Both men have also told colleagues they are distressed with the criticism about the walk to St. John's Church because it became a photo-op with political overtones. When asked if they regret being on the walk, one administration official close to both men answered, "Of course they do."
Esper this week told reporters he tries to stay out of political events but knows he doesn’t always succeed. Milley has not yet spoken publicly about the walk in which he was in a battlefield uniform.