Mike, feelings like pain and pleasure are biochemical feedback aren’t they, similar to any feedback into a system? Like the ancient creatures we evolved from we are very strongly cued by interpretation of emotion — split between automatic responses and reflective responses. Sense on the skin or other receptor, respond with instant chemistry, feel, act, reflect, act, “ouch!”/”wow!”
I don’t think that building intelligence systems that have this sort of chemistry feedback would be monstrous, although Mary Shelley disagreed. Probably a bit hard to fine tune the process, as our nature suggests…may need a bill years or so of experience and even then not always so great on a Monday morning before your flat white kicks in..
NN’s will probably produce emergent features that seem like consciousness soon but I don’t think they ever be humanlike, in the same way that Squid Thinking is probably a bit of a different experience to People Thinking.
Interesting topic, I enjoyed your reply and the article.