Is the idea original or an adaptation of an existing idea? If it’s an adaptation, is the idea at least a 10x improvement over the competition?
Follow your heart. Do it, even if it doesn’t 100% make sense. It might not work out exactly how you imagine, but it’s probably less risky than you think. And either way, you’ll learn something.
If you have to ask “will this annoy users?” the answer is always yes.