Lol @ Neil's comment. McCallum beats the Hell outta Kelly. Most likely by wide UD, but a late stoppage is possible. I'd give Kelly the benefit of the doubt and say he lasts the distance though.
Pavlik is the larger man and McCallum liked to fight close, which would suit Kelly. McCallum wins clearly with his craft but it would be a bruising bout and pretty even for six rounds or so
Not that I can't see your scenario happening, but it's worth pointing out that Watson was also a bigger man than McCallum. Watson, like Pavlik, was really a super middle campaigning at 160. It didn't save him from getting battered.
McCallum would be the fighter that does the second most damage to Kelly's liver. Kelly is champ in that category.