I think a lot of the problem for the Vita especially was that Sony tried to make it a system with smaller PS3/4 games, as you said. But not only do those cost more, that also is a market that got cut up by super smartphones that can easily emulate up to the N64/PS1, and can play games that are as good as the Vita can, but also do so much more. It's a continuation of the PSP's biggest weakness, but now the smartphone hardware is enough to eat it up. Yeah Sony's PS2 rereleases have been kind of atrocious ( I think they're still PS3 only, not even working on PS4), but one thing they did do right that Nintendo hasn't is making Minis and PS1 games cross platform, so I don't need to buy it again to use it on the Vita, unlike Nintendo who will still take my $5 or whatever for Super Mario Bros on the Wii, Wii U (since the Wii menu on the Wii U is so bad) and 3DS. But Nintendo has been making unique experiences for their handhelds, games like Pokemon or the Zelda games for it aren't things you could see easily on the Wii U, whereas something like Wipeout 2048 or Uncharted...something something are games I could totally see on the PS4.
hell, a portable PS3 is a great idea but, Sony threw out all of that effort on either cut down ports of PS3 games, Visual Novel shovelware, or just rehashes (and even at that, not very good rehashes) of older PS2 games. It's not that Sony didn't try to make the games unique, it's just that none of the games were worth playing and any creative ideas were wasted with poor execution.
