How do I keep flies off of my butterfly feeder?

I made a nifty feeder, with orange, banana, and nectar, but as soon as I put it out, the orange and banana are attacked with gusto by a swarm of houseflies. How can I feed the butterflies without feeding the flies. I live in central Arkansas, if that makes a difference. It stays very hot (90's+) most of the summer here, and it's somewhat humid. The feeder hangs about 5' from the ground.