The brown, falling leaves at the bottom sound like spider mites, which ivy is especially prone to. Spider mites are rarely visible to the naked eye, but look for very fine, mini spider-webs. But you may not even find those unless the mite population is quite high. If it is indeed mites,I would recommend thoroughly spraying the leaves and stems, top and bottom and undersides, with a mixture of 2 parts water to 1 part of rubbing alcohol. Every 3 days for 1 month. But I must add that I've never myself defeated spider mites on an English Ivy, though I've eradicated them on other plants. E.I. seems to be the perfect food for the monsters, rendering them impervious to assault! I've given up on E. Ivies for this reason.