social sanction

Sanction refers to either a penalty or a punishment so when you put the word "social" in front of it, it refers to a group or organization that has norms and boundaries and you either get punished from crossing the norms and boundaries, but can also be rewarded for staying and complying with the norm. The reward is also a factor of social sanctioning.