I will defend the IRS a bit on this, basically auditing the rich requires high skill because of all the tricks they use, and I think it averages 2-3 years of work per return. And that's before they get into the legal fights over whether the tax payer did anything wrong. The poor? A day or two's work at worst typically, as its often a mistake and not intentional, so they're not prepared to go to war over it.

Yes, I still think the IRS should make the effort, but conveniently Congress doesn't give them enough money and politician's like to demonize any budget increase as an attack on the poor.