Without seeing the insects more clearly we cannot be positive, but from your description they might be fungus gnats. If you see the adult insects fly away when you water, this is another indication that they are probably fungus gnats. Resembling fruit flies, their larvae feed on organic matter in the soil, and damage the roots of young seedlings. Soil that is kept constantly wet is especially susceptible to fungus gnats. To break the cycle and to keep the adults from laying eggs in the soil, reduce watering as much as possible without damaging the plant and cover with a layer of horticultural sand available at your local garden center (do not use beach sand). You can also use yellow sticky traps (again available at your local nursery) to capture the adults. The larvae in the soil are killed by putting Spinosad in the watering can. Spinosad is the active bacteria, an organic treatment so fine for edible herbs, and it’s available in a variety of products at your local garden center.
The combination of yellow sticky cards and Spinosad is very effective. There are a couple of other products that are applied on the surface of house plants that also treat this problem organically. Ask at your garden center if you’d like to explore those options.