If you spend time with those who are behind you, your average will go down, and with it, your success.
13 Things You Should Give Up If You Want To Be Successful
Zdravko Cvijetic

Yes, give up toxic people.

That is a very different thing from spending time with people less accomplished than you.

If the people more accomplished than you adopt this approach, then you will spend zero time with them.

Be a mentor to the less accomplished while you are being mentored by those who are more accomplished.