According to the BBC, if Vaz, who was born to Goan parents in Yemen's Aden, would not stand aside, committee members would give him 24 hours to "reflect on his position" before he faces a possible no confidence vote.
The Leicester East MP has yet to comment in detail on the allegations.
A senior member of the committee told the BBC that Vaz would be given the opportunity to "justify himself", adding that he was "pretty sure" the MP would at least stand aside on a temporary basis.
According to the source, the position of Vaz, the parliament's longest-serving British Asian MP, was "not tenable".
MPs were unclear over what powers they had to force Vaz out if he refused to quit. Senior figures on the committee were consulting Commons clerks.
The BBC reported that there was still some support in the committee for Vaz from members who held him in "high regard" for his chairmanship.
Prime Minister Theresa May on Monday said the public must have confidence in its politicians, while Labour leader Jeremy Corbyn said it was a "private matter".
According to Labour MP Kate Hoey, it would be in the interests of Vaz and parliament for him to leave the committee.
She said there should be an election of a new committee chair involving all MPs.
At the weekend, the Sunday Mirror published pictures showing Vaz with male sex workers in a flat in north London that he owns. Illegal drugs were mentioned during a secretly recorded conversation.
Conservative MP Andrew Bridgen said he would refer the matter to the Commons Standards commissioner and may also report Vaz to police.
Married father-of-two Vaz said he was referring the paper's allegations to his solicitor.
Vaz, 59, also described it as a "privilege" to be the chairman of the select committee, which monitors crime and drugs policy, for the past nine years.
"I will, of course, inform committee members first of my plans when we meet on Tuesday. My decision has been based entirely on what is in the best interests of the committee," he said.
This website uses cookies.