Hitting a wall will hurt more than hitting a piece of sponge. This can be explained as follows:
Wall is hard. When you hit a wall, an equal reaction force acts on the hand and hurts it.
When you hit a piece of sponge, a part of momentum of hand is transferred to the sponge. As a result, it gets deformed (squeezed), and the reaction is much less than the action.
