Great points, but keep in mind I’m less concerned about the specifics of IQ or whatever and more concerned about big shifts we may not like, that we can’t anticipate. Other examples might be Epigenetics. We have strong views on nature vs nurture, what if an A.I. could untangle the relationships and tell us we are 97% nature 3% nurture, or 97% nurture and 3% nature? It’s unlikely but, if it happens, it would shatter much of the way we think about the world.
I was only making specific cases to ground the generalization, not to point out what is a likely area to have an A.I. prove us wrong.