Sounds like the termites are coming from the ground up through the hole in the bottom of your pot. Maybe u could treat the ground under the pot with some chemical or just move the pot to an area, such as a slab of concrete or asphalt where they cannot penetrate.
If they are just a few renagade termites, they really pose no danger whatsoever, having been "cutoff" from their nest.

* * * *
Bob

I'd get rid of them anyway. Submerge the pot for an hour in water. Then allow to drain. If you add a little fish fertiliser or similar to the water you'll fertilise the plant at the same time.

I soaked the pot for about an hour and a half. It's been a few weeks now and no termites. :^}

Thanks Longy!

Yeah you should ensure your house is protected from infestation. If it is you have little to worry about. There are termites everywhere and we'll never kill them all. Same as black ants or pretty much any insect. The ones in the pot would've been workers or whatever looking for a feed. The queen is still alive somewhere with her plethora of slaves. She could be right under you or hundreds of metres away. Possibly under your neighbours woodshed or at the base of that big old dead tree down the road a bit. Anyway, you killed the ones in the pot and that was your query:)

* * * *

The secret is the soil.