By making this distinction he is not saying “This is a bad apple; that is a good apple.” He is saying something more like, “This is a transitive verb, and that one is intransitive.” The servile arts are utilitarian -- that is, there is an object outside of themselves, just as when “Sally throws the ball” the action is performed on some object other than herself. But if “Sally smiles,” the action remains with herself, even though it may have an effect on the people around her.
We have trouble with his distinction because he is saying that the servile arts are inferior to the liberal. Egalitarianism is in the very air that we breathe and so we tend to view this kind of hierarchical distinction as being one of good v. bad, or better v. worse. But I think he's saying that the servile arts are inferior to the liberal in the same way that a foundation is inferior to the walls of a house, which are in turn inferior to the roof.
The foundation is, well, fundamental. But it's not there for its own sake; it's there to hold up the rest of the house.
Likewise, the servile arts are absolutely necessary to life. But we don't live to work, do we? We work so that we may be at leisure. Or, as John Gould Fletcher put it:
We feed and clothe and exercise our bodies, for example, in order to be able to do something with our minds. We employ our minds in order to achieve character…. We achieve character, personality, gentlemanliness in order to make our lives an art and to bring our souls into relation with the whole scheme of things, which is the divine nature.