Do you know what type of crystals you have ? If you knew it would help narrow down the cleaning process. In most cases some mild soap diluted with water with a soft cloth would suffice in most cases, but some crystals are water soluble and can get damaged this way. Even with such a crystal it might be worth doing still, rather than have the ink from the marker on it. I would use a cloth though and only enough water as needed.
Depending on the type of crystal, certain acids can work wonders, but again this depends on the mineral in question. Try a little sunlight soap on the end of a Q-tip with a tiny bit of water just to moisten it a bit, and then go over a small part of the ink very lightly and see if there is an improvement.