The problem with that is pricing/power. Vita was powerful, way more than 3DS which meant more expensive. That then put the handheld in console pricing which made it a tough sell. At the same time, phones each year are getting more capable and powerful, leaving Vita behind.
The advantage Nintendo enjoys with their handheld line is that nobody is expecting killer hardware at that price. If Sony could treat the Vita line like mobile and advance/refresh it every 12–14 months, it may have had a chance because it would have been a different beast but their current strategy never allowed for it, hence the terrible sales numbers.