I appreciate this. My husband and I have an open relationship and it has manifested a number of different ways, including, early on in our exploration, a few situations much like this. My feelings about it sound much like yours too. I’ve come to the conclusion that sex without connection is unfulfilling for me, too.
To be clear though, having a casual foursome would not be considered polyamory by most people who consider themselves polyamorous. It is ethical nonmonogamy, yes. But polyamory, in my understanding and use of the word, is the practice of having multiple intimate, love-based connections that are not centered around sex alone.
I, myself, am married to my long term partner, and have another serious partner that I am very much in love with. Just wanted to help make the distinction.
It took my husband and I years of trial and error (some big errors) with nonmonogamy to be brave enough to really let each other be free to fall in love with others. Maybe you’ll experience a similar journey. Maybe not. But kudos to being brave enough to be honest with each other and try something new together. Happy adventuring!