When a stone is going around a circular path, the instantaneous velocity of stone is acting tangentially to the circle.
When the string breaks, the centripetal force stops to act.
Due to inertia, the stone continue to move along the tangent to circular path.
That is why, the stone flies off tangentially to the circular path.