The summoner will become the bane of your existence once you encounter them in Gladius. They love to charge up affinity, then summon a creature to give you hell while they lob spells at you. In the beginning, they aren't hard to kill, with ranged units being your go-to for summoner problems. Once they hit a certain level though, they start hitting you with a powerful spell called Far-Strike. Because it is a Masher type spell, even when you receive your own Summoner Sutehk in the Southern Expanse, you can't take advantage of it yourself, making them hard to kill with being really lucky and dodging their hits.
When you finally get your own summoner Sutehk in the final region, he starts with Earth Affinity all the way to LV 4, so you probably aren't switching him. Because of the fast recharge of his Summon Affinity skill, he is good for summoning a creature, then spamming the Affinity 4 spell. His real power lays with his summons though, with the Affinity Beasts giving you stat boost depending on the element. You are not limited to earth Beast for summoning, as his affinity doesn't affect his summoning capabilities.