The key is to refactor in the right places — the places where you think you will need simplicity and flexibility in the future.
Managing Technical Debt
Carl Tashian

But you already correctly noted that one can never guess the future. What is the point of refactoring based on our guess of the future?

The best time to refactor is when you are implementing new requirements.

What you need to ensure for the future is that after each change the code is still clean and simple. If it is easy to understand the code, it will be easy to refactor it.