Mother Nature never gave a fig for fairness.
I’d guess that most English speakers today would assume that “fig” in this context is simply a phonetic replacement for the F-bomb, but there’s more to it than that. The word “fig” was once commonly understood as a reference to a rude hand gesture that more or less conveyed the idea “I wish you would go away and get ass-raped” — similar to “showing the middle finger,” but with the fingers in a different position.
So “I don’t give a fig” essentially meant “I care so little that I can’t even be bothered to make a fuck-off gesture.”
Using the local word for “fruits of the tree Ficus carica” to mean “a certain obscene hand-gesture” goes back at least to the ancient Greeks (their word for “fig” in both senses was sykos), and persists in various languages today.