Size can help, as far as boosting the energies of a stone, but size isn't the only consideration. Even small stones can be potent.

Basically, if the pendant that you want to wear is a comfortable size for you, and if the stone's energies resonate with you, then that's the right size to wear.