But you're talking about graphics only. If that was the only issue, it wouldn't really matter. But Skyrim had serious game breaking stuff happening having to do with the increasing size of save files and the PS3's different structure for system memory. The longer you played the bigger your save file became, the more the game slowed down, and the more the game would just keep freezing until the whole thing became unplayable. In fact, Bethesda was initially unable to release the first expansion for Skyrim on the PS3 because they couldn't make it work without making these issues even worse (it took them eight months after the release of the first DLC for the Xbox version to release the first DLC for the PS3 version, purely due to the performance issues the PS3 was experiencing). Worth noting too that Bethesda never sent PS3 copies of the game to critics for review, as is standard practice these days -- they were withholding it from the critics because they knew there were serious issues with it that the Xbox version didn't have.
All of this is just to say again that there is the occasional game that has serious problems on one platform and runs well on the other. But truly, these cases are rare. 98% of the time differences are minor, and would probably only be noticed by doing side-by-side comparisons. Moreover, I expect these rare cases to be even rarer next-gen, as the system architecture for both consoles is more similar than ever.