That is a possibility and it presents a further complication to consider.

Well, yes, I guess…but the universe changes. Our measurement of time (units) is purely arbitrary and a human construct, but the situation that something was once over there and is now here is external to ourselves. Measurement units aside, light defines time. The past is an artifact of all the light we’ve observed, and memory is the catalog of those observations. The future is all the photons we haven’t observed yet, as they enter our forward light cone and we eventually observe them.

In effect, we’re going down the river in a canoe, facing backwards. (Feel free to invent your own “without a paddle” punchline. :) )