The first step towards this problem is data augmentation. By rotating, blurring, or cropping image data, synthetic images can be created that approximately mirror the distribution of images in the original dataset. This method is not perfect, however — it provides a regularizing effect that may be unwanted if the network is already not performing well in training.
I was reminded of this attitude recently when I came across a video by Gary Vaynerchuk, an entrepreneur and internet business celebrity. He was asked if he was "invested into blockchain and bitcoin." His response provides a good illustration of common misconceptions and fallacies about this topic.