Yes, the ball could have rolled either way. There have indeed been white slaves with black owners, in history. The way the ball rolls, can indeed be one way in one case, and the other in another case, in the course of history.
Identity politics and identity-group guilt and identity-group victim status is absurd on multiple levels, including this one. Perhaps this is part of the intention of your argument.
Slavery was an almost universal norm; “working for nothing more than food and lodging” was merely one possible outcome apart from starving to death, for much of humanity through much of history. Even “independent” subsistence living was little better.
One of the biggest conceptual mistakes that plagues historical-political argument today, at least from among the ignorant (who are far too numerical and far too influential) is the projection of comfortable modern conditions of existence, with its surpluses of production, back onto the entire past.
