These days, he has traded in his gloves for checkers.
"He writes his own rules," opponent Dan Malley laughs.
Malley, a boxing fan and Cleveland's own Chocolate Czar, learned about Bivins from his father at a Golden Gloves Fight. Malley later introduced himself to Jimmy and his family.
Today, he is a frequent visitor to McGregor Nursing home where Bivins lives.
"He is kind of like a forgotten celebrity of past, he brought a lot of fame to the city of Cleveland when he was fighting," Malley said.
For 13 years, Bivins was a ranked fighter. During his time, he fought Joe Lewis, Ezzard Charles, Archie Moore and fellow Clevelander Joey Maxim.
"He fought a lot of champions and he beat some and he lost to some," but nephew Frank Nelson says what is important is that his uncle is, "just a great person, a great human being."
Bivins' professional career began in the early 1940's. In 1943, he was given the title "Duration Heavyweight Champ" by Joe Lewis while he went away to war.
Bivins is the only person to ever hold the title. He also held the duration title for the light-heavyweight division as well.
Jerry Fitch is a local boxing historian who helped open doors for Jimmy when it came to the Hall of Fame memberships.
"He raised a lot of questions about how come Jimmy wasn't in any of the Hall of Fames," said Gene Glen.
Glen who works with and is responsible for several boxing events in Cleveland has all kinds of articles, books and information about Bivins' career -- including a book written by Fitch called "Cleveland's Greatest Fighters of All Time." The cover is a muscular and much-younger Bivins.
The next book due out by Fitch is solely about Bivins. It's called "James Louis Bivins: The Man Who Would Be Champ." Fitch says it will include photos not seen in more than 60 years.
A release date has not been set.
According to Fitch, Bivins was a good student in high school. He ran track and got advice from Jessie Owens to focus on boxing because unlike track, it could provide an income.
Fitch was there when Bivins accepted the Integrity Award from the Rochester New York Boxing Hall of Fame.
Fitch says that when Bivins was asked why he didn't get a shot at a title, the athlete responded: "I was asked to play ball with the powers that be. I told them I am a fighter, not a ballplayer."
This fighter fought through difficult times himself.
In 1998 he faced an unthinkable personal battle. Police found him, emaciated, locked in the attic of his daughter and son-in-law's home.
Glen believes it's the mental and emotional toughness he gained through boxing that helped him get through the ordeal.
Some days Jimmy is up for talking; others he is a little more quiet.
Malley says in the summer he wheels his friend into the courtyard and they sit in the sunshine and talk.
Jimmy smiles when he eats his favorite Malley's treat -- chocolate covered marshmallows. Bivins was even featured in last years Malley's Easter catalouge wearing bunny ears and a smile.
Bivins was a fighter and a worker. For years he worked for the Laub Banking Company and then on to Dan Dee Potato Chips.
The public knew who the deliveryman was.
"People would see him and know him and say, 'Hey champ!' and he would wave and I think he liked that," Nelson said.
Bivins boxing career ended just as Nelson was old enough to understand boxing. He smiles when he thinks of going to Cleveland Browns football games (during the Otto Graham era) with his uncle and father.
"He was a very outgoing guy and he liked everybody and race never made a difference to him. I can remember as a kid he had a tobbogian sled... and everybody that was there was welcome to ride down the hill and it would hold maybe 12 people and it would be black and white there was no difference he just received everybody he was that kind of a guy."
That guy was a boxing coach and a judge years down the road, giving kids an outlet and a way to stay off the streets.
"He became a role model, not only was he respected in the African American community he was respected in the majority community as an outstanding contributor, as an outstanding human being really," Glen says.
Glen also says that in the eyes of some people, Jimmy had a victory that was never put down in the record books.
"I've heard a lot of stories, a lot of people say that Jimmy beat Joe but Joe got the decision."
If you go and visit Bivins, he will likely invite you to play checkers. Just don't expect to play by the standard rules or ask questions about his.