The issue is not specifically why they can be sued, but that they in fact can be. You haven’t contained your corn, and as a result you’ve exposed oh, maybe, 64 city blocks to GMOs. This isn’t even a problem because GMOs are dangerous, it’s a problem because as you mention, there’s a lot of emotion around GMOs and many people have rejected them. There were 101 bills introduced at the state level in 2015 around GMOs — labeling, containment, appropriate notification. It would not occur to your neighbors that someone near has gone so far as to purchase these exotic factory farm seeds and plant them, uncontained, nearby.
One of the reasons people have backyard gardens is to control their food chain; they want fresh produce without pesticides, GMOs or other complications of the factory farm. You’ve wrecked that, for no particular reason. Did your neighbors ask for your advice? Have you put a sign on your fence to inform them of your little ‘experiment’ at their expense? Did they elect you to the Illuminati? Is your political opinion more important than their feelings?
Your planting isn’t illegal, it’s just arrogant, presumptuous, and poor science. Of course you can grow what you want — we don’t legislate around mere rudeness, and we expect neighbors to be respectful of each others homes, culture, and yes, feelings. Obviously this isn’t going to work with you. If it is a concern for them, hopefully one of your neighbors is smart enough to bring a civil suit based on your uncontained crop and the risk it poses to other backyard gardens. Several localities do have legislation demanding notification of the location of GMO crops; insensitive ‘experiments’ like your own practically ensure we’ll have more of the same.