The Accidental Tech Podcast is a podcast that's entered my regular rotation. Beyond its intelligent personalities, I'm always impressed by how clean the audio is, which is a rare thing to find in podcasts these days.
Co-host Casey Liss has recently published a detailed description of what they use for their setup, and it's impressive. While not super expensive, it clearly shows a lot of thought. Anyone who wants to make a high-quality sounding podcast should bookmark it as a valuable reference guide.
For the record, the podcasts I produce use a far cruder setup than the one described. In fact, my situation is not too far removed from what ATP co-host Marco Arment derogatorily refers to as "recording people over Skype with mediocre USB microphones and exporting it with nearly zero editing."
That being said, as Liss points out in his write-up, my setup also confers a big advantage: Timeliness. It's what allows people to get their Game of Thrones recaps in their podcast feed within 24 hours of the episode airing.
But beyond that, over time, I've personally found that you rapidly experience diminishing returns when it comes to podcast audio. There is a baseline level that I (and most people) will deem to be acceptable: everyone on a podcast should have a dedicated microphone that is placed somewhat close to their mouths. Edit out Skype glitches, which are a dead giveaway that your podcast is not being recorded in a professional studio. Beyond that, incremental improvements in audio quality do not bring incremental improvements in listenership.
But those who value audio quality will always be able to tell the difference. And while it's something I do value, my personal podcast projects are not the medium that I'll choose to chase it through.