But you already correctly noted that one can never guess the future. What is the point of refactoring based on our guess of the future?
The best time to refactor is when you are implementing new requirements.
What you need to ensure for the future is that after each change the code is still clean and simple. If it is easy to understand the code, it will be easy to refactor it.