Why haven't we gone back? You mean, aside from all the times we have? Maybe because it's dangerous, expensive, and the public as a whole has shown no interest in the idea?
It couldn't be any more dangerous than forty years ago. The public would show interest if our space program hinted at the idea. The only reason in your post that applies is that of expense. What isn't expensive these days?
I, for one, want to witness a moon landing during my lifetime. Who cares, right? I'm just a little ole taxpayer. :