I meet once every two weeks with some good Japanese friends from Mill Creek, and we do language exchange for about 90 minutes (25 minutes Japanese only, 25 minutes English only, then free-form). It's good practice, but I'm continually flustered by my inability at times to follow key parts of the conversation. The words just fly by too quickly, and my mind can't connect the dots quickly enough. I can ask my friends to slow down and repeat - which I do, and they do, because (1) they're awesome people and (2) they ask me to do the same thing with my English. :) But it's still frustrating.
So I'm working like hell to bring my comprehension up to speed. Fortunately, I have a ton of material on hand with which to work: a full 24-episode season of K-On!! with 日本語字幕, the Nippon VoiceBlog material, and NHK ニューズ with accompanying video clips. When I don't feel like studying per se, I can drown my attention in audio-only experiences, such as the HOTCAST and そこあに podcasts, or new アニメ on CrunchyRoll.
Am I worried that listening won't fall into place within the next couple of months? Not really. One of the first things I worked on when re-starting my Japanese studies was listening to and reading 花咲じいさん。Despite having studied Japanese years ago, I could barely make out anything in that story without intensive studying. But I slogged through it. And another after that. And another, and...etc. Now, six months later, I can listen to an arbitrary 童話 and usually make out at least 70% of what's being said, without reference to the text.
I know that a lot of my "problem" is simply that my brain can't process Japanese quickly enough, because it doesn't have enough familiarity with the language. The only solution to that is 練習、練習、もっと練習.
Japanese - or any language, really - isn't out of anyone's reach. It just takes time and persistence.