Your brightness augmentation is very smart. I am going to try it.
I note that there is a possibility for “washed out” artifacts , so I modify the code as:

perturb = cv2.cvtColor(perturb, cv2.COLOR_RGB2HSV)
current_bright = np.average(perturb[:, :, 2])
dbright = illumin_limit * np.random.uniform()
dbright = 1-dbright if current_bright>40 else 1+dbright
perturb[:, :, 2] = perturb[:, :, 2] * dbright
perturb = np.clip(cv2.cvtColor(perturb, cv2.COLOR_HSV2RGB),0.,255.)

you can see my examples at:
https://github.com/hengck23-udacity/udacity-driverless-car-nd-p2/blob/master/data/data_argument.jpg

I have another idea. Spatial transformation network automatically adjust the affine transform. In theory, we can make a “illumination transformation network”.