Poor preservation can be an issue. If you leave a fish out in the sun for two hours right after it’s caught, it’ll taste pretty gross, compared to one that’s been frozen, butchered, and aged properly.
Also, when the fish is butchered, if the fish flops around a lot when it dies, it uses up ATP, which in the process, produces lactic acid, which gives it a kind of sour taste. Also, the more ATP there is in the muscles, the more aging can decompose into a umami flavor. That’s why Ikejime is used, so the fish doesn’t flop around when it gets butchered.
If there are other factors, I’m currently unaware of it, but if I come across more, I’ll write about it. Thanks for the comment!