Sorry, couldn’t resist.
I have this problem – it’s especially bad when I have more than one highlighter colour. My Kindle makes extensive use of the 4 colours – although I guess I use them to classify the different highlights (red = important key point, yellow = interesting point, blue = key story/explanation/case study, orange = amusing anecdote) but I still use all of them WAY too much, and as you say it stalls my reading speed no end.
Perhaps in the future when wearables are common place, rather than having to stop to highlight bits we found interesting, the sensors that measure arousal/concentration/happiness will track our overall response to a piece of text (mapped against which bit of text we were reading at the time, based on which part of a page the sensors on our head deem our eyes to be looking at) – and then we might be able to search a book for “that bit that made me laugh out loud”/ “that paragraph I had to read three times because it was so profound” / “that description of the old lady that made me weep” and our smart device would be able to immediate pull up that bit onto our screen?