Lineker, 61, said he received racial abuse during his playing career because of his "darker skin".
The outspoken ex-footballer said he "would have been bullied" if he had not been good at sport and claimed he "pretty much had racist abuse" but "wouldn't ever name names".
His comments have sparked backlash, though.
Appearing on GB News' Dan Wootton Tonight, political commentator Calvin Robinson did not hold back on the issue.
Gary Lineker has enraged Calvin Robinson with his claims
PA/GB News
Gary Lineker has worked as a football pundit ever since he finished playing
Ian Walton
Opening the debate, host for the evening Patrick Christys said: "Gary Lineker is many things, but he's not black!"
An enraged Calvin replied: "I'm not the person to use 'the race card' but I find that offensive. I find that outrageous.
"That's undermining genuine racism that's happening in this country.
"This is cultural appropriation, isn't it? Outright."
Fellow panellist Rebecca Reid also took issue with Lineker's claims.
She added: "He says 'I'm as English as they come' – which doesn't mean you couldn't be black!"
It comes after Lineker told the High Performance podcast: "Without being good at sport, life would have been very different for me because I was, I think I would have been bullied at school.
"I was kind of marginally that way anyway because I was this tiny, geeky kid, with darkish skin and I had pretty much racist abuse although I’m not, I’m as English as they come. All the time, all the time.
"Even in professional football I had that a couple of times, I wouldn’t ever name any names.
"So I got that kind of nonsense, which was a bit weird.
"Whether that was part of something that made me, I don’t know, but other people might not be able to handle that.”