Rep. Joe Sestak's allegation that the White House offered him a job to drop out of the Pennsylvania Senate primary race against Arlen Specter is a crime that could lead to the impeachment of President Obama, Rep. Darrell Issa said.

But the decision by the Pennsylvania congressman not to elaborate on a so-called deal also could become a political problem as Sestak tries for the U.S. Senate seat.

The White House reportedly is going to formally address the allegation in the next few days. In the meantime, Issa, R-Calif., is one of many inside and outside Washington who want the Democratic Senate primary candidate to explain in detail what offer the White House made.

"It's very clear that allegation is one that everyone from Arlen Spector to Dick Morris has said is in fact a crime, and could be impeachable," said Issa, who is threatening to file an ethics compliant if Sestak doesn't provide more details about the alleged job offer.

Sestak, a former vice admiral in the Navy, first alleged in February that the White House offered him a high-ranking position in the administration last summer if he would sit out the primary against Specter, who won the backing of the White House and state Democratic leaders for switching parties. 

The allegation is considered one of the factors that helped him defeat Specter, who was viewed as unscrupulous in doing whatever he could to keep his seat, including changing his party to win White House support for an uphill re-election battle.

But now, Sestak has to go into the general election, where his opponent, former Republican Rep. Pat Toomey, is willing to use the topic as referendum on both Sestak's and Obama's credibility. 

"Congressman Sestak should tell the public everything he knows about the job he was offered, and who offered it," Republican Senate candidate Pat Toomey said in a written statement. "To do otherwise will only continue to raise questions and continue to be a needless distraction in this campaign."

And Democratic Illinois Sen. Dick Durbin, a close ally of the president, said Sestak can't continue to lay out the charge without backing it up.

"At some point, Congressman Sestak needs to make it clear what happened," he said.

Issa said this allegation is bigger than the Senate race.

"For Joe Sestak, he can dance around it and he may or may not be a senator," he said. "But for the White House, this problem's not going away. Adm. Sestak is in fact a very reliable source."

Ann Marie McAvoy, a former federal prosecutor, said the White House could have a problem on its hands depending on what the facts show.

"If they were simply offering him a job because they thought he was a qualified person for it and there was no request made that he in essence drop out of the race, it would be different," she told Fox News. "This is why there really needs to be an in depth investigation. There needs to be witness interviews and so on to figure out what happened, who said what, what were the other circumstances surrounding it."

Issa has called on the Justice Department to appoint a special prosecutor but Attorney General Eric Holder has said that won't be necessary.

"We assure you that the Department of Justice takes very seriously allegations of criminal conduct by public officials," Assistant Attorney General Ronald Weich told Issa in a letter. "All such matters are reviewed carefully by career prosecutors and law enforcement agents, and appropriate action, if warranted, is taken."

Weich said a special prosecutor won't be needed because the Justice Department "has a long history of handling investigations of high level officials professionally and independently, without the need to appoint a special counsel."

But McAvoy said any allegation of wrongdoing shouldn't be left to the Justice Department to decide.

"Someone in the administration had this conversation which means that would be the person mostly likely who committed the crime if there is a crime," she said. "So you have the people who are representing the people who potentially committed the crime are making the determinations as to whether anything wrong happened. That's not the way it's supposed to happen."

Issa compared any potential cover-up to the Watergate scandal of the Nixon era.the White House Tuesday of a cover up similar to the Watergate scandal.

"It's not about what was done wrong. It's about the cover up," Issa told Fox News. "And right now, there's a cover up going on at the White House 10 weeks after the allegation."