I agree with most of what you are saying except for two points.
Firstly, the fact that both major parties support offshore detention does not necessarily mean that the majority of Australians do, just that the political parties believe that a significant number of swinging voters support the policy.
Indeed, opinion polls suggest that Australians are pretty evenly split between supporting and opposing offshore detention. See for instance: http://www.abc.net.au/news/2016-06-09/election-2016-vote-compass-asylum-seekers-immmigration/7493064
I also cannot agree with you that this issue is one for which only “white” Australians should be held accountable. Again, opinion polls (like the vote-compass one linked to above) suggest that migrants actually support offshore detention slightly more strongly than do native-born Australians.