For me, 'that one album' is Pet Sounds by The Beach Boys. If you are a fan of this album you may want to stop reading, I will understand if you find this a bit painful to read. I would find the same if someone did this to one of my favourites.
The great and good of the music world have continually praised this album (including people like Roger Waters and Eric Clapton), and it's very often at the top of lists of Rock's greatest ever albums. I was impressed by this praise for the album and listened to the album. I tried to like it, but I was really underwhelmed. I tried again, but found it no better second time around. I was confused - why was this genius record not even succeeding in holding my attention?
The harmonies really are the only thing I can see as being truly great in this album - I can see why people praise it for being the forebear of the concept album, but it is just a collection of love songs (a very common theme), and with the exception of Caroline No, a lot of the songs just seem twee and child-like to my ears - for me an album cannot be genius if you have to suspend your usual expectations of the standard of the lyrics. The music doesn't really grab me either, although as someone who's primarily a rock (rather than pop) fan this doesn't bother me quite as much as I can rationalise this.
I accept that some will be horrified by this comparison, but if I stack Pet Sounds up against Fleetwood Mac's Rumours, one album gives me a number of variations on the same song, the other gives me a great variety of tracks from rockers to stripped back piano ballads. One album gives me teenager in love platitudes, the other gives me lyrics which describe a real adult relationship falling apart. I could go on.
The irony is that I don't think most people would say this album is more challenging than Pink Floyd, yet I struggle with Pet Sounds while enjoying The Dark Side Of The Moon or The Wall.
Part of the problem is that Pet Sounds is widely considered to be one of the greatest albums in rock history. Even though I don't get it, I wouldn't disagree with anyone who considered it to be one of the greatest albums in pop history, but this doesn't even count as soft rock - I consider Sloop John B to be as pop as anything else on here. I have no problem with the fact that so many other people love this record, my main issue is with its position at the top of 'Greatest Ever Rock Album' lists.