Mr Johnson suffered a double blow as voters rejected the Conservatives in both Wakefield and Tiverton and Honiton.
The defeats pile more pressure on the embattled PM, fresh from a bruising confidence vote filed by his own MPs just weeks ago.
And ex-Tory MP David Mellor issued a warning for Mr Johnson, appearing on GB News' Breakfast with Stephen and Anne.
Host Stephen Dixon asked: "In terms of those Cabinet ministers, who are all loyal to Boris, they will also be thinking that if 'I want to get into another Cabinet with another leader' perhaps now is the time to act?"
Former Tory MP David Mellor
GB News
GB News presenter Stephen Dixon
GB News
Mr Mellor replied: "Yeah, you know, there are plenty of rats and they're now aboard a sinking ship."
He went on to praise ex-party chairman Oliver Dowden, who resigned soon after news of the two defeats were confirmed.
Mr Mellor dubbed Mr Dowden "brave" as he noted that the points he made in his resignation letter, used to justify his decision, were made "with dignity".
But Mr Mellow told GB News: "What he didn't say was so much more important. There was not a single word in tribute to Boris."
Mr Dowden said in a letter to Mr Johnson that “someone must take responsibility” for the state the Tories find themselves in.
He wrote: “Our supporters are distressed and disappointed by recent events, and I share their feelings.
“We cannot carry on with business as usual. Somebody must take responsibility and I have concluded that, in these circumstances, it would not be right for me to remain in office.”
Labour leader Sir Keir Starmer said: “This result is a clear judgment on a Conservative Party that has run out of energy and ideas. Britain deserves better.”
He said the Wakefield win shows Labour “is back on the side of working people, winning seats where we lost before, and ready for government”.
Mr Johnson, who is at a Commonwealth leaders’ summit in Rwanda, suggested it would be “crazy” for him to quit if the party lost the two seats and said mid-term by-elections were “never necessarily easy for any government”.