How one mistake turned 100k records into 56M records, caused multiple parts of a production environment to fail for hours and what we’ve learned from it.