Against regular people in regular situations, that's all it's going to take. A big bastard like that gets a hold of you in a bar fight and you are fucked like that little bully against the giant nerd. In a controlled setting, lactic acid sets in and then the big muscles are a big liability.
Of course if some giant freak of a human gets a hold of a regular size dude it's not going to be pretty. Even fat dudes are strong as fuck. I had a drunken boxing match with a 500lbs dude one night and that fucker hit me hard and was strong as an ox. We were just going to the body though so we didn't actually kill eachother.