In practice I think that’s what legislators and executives do all the time, or are supposed to. When they ban handguns from schools are they violating the Second Amendment? Some would say yes, but the majority of us would disagree. When they purge voter rolls of literally hundreds of thousands of people with foreign sounding names they claim to be protecting our system against voter fraud. Almost any decision trades off rights vs impacts in some way — and government done right balances them well, in the interests of the people rather than the party.