Taste is the only one of the senses that requires the destruction of the object. Sight and sound don't even require contact. Smell flits molecules away from the object, but only those that would be released into the air regardless. Touch can have a corrosive or destructive element, but in many circumstances the natural oil of the fingers will not be enough to harm an object's surface.
Taste, however...depends upon the dissolving through saliva of some small part of an object, having it come away on your tongue, stealing it. Taste takes a little part away for its own. Not to mention the destructive connotations of eating, teeth, and digestion. Tasteless things like stainless steel are impervious to the destruction of our tongues, and so escape flavor.
Thoughts from a visual art thesis upstairs, entitled "Taste" and showing permutations of apples.