Next time you have fruit flies, break the cycle between the adults and the eggs in the soil, use yellow sticky traps to get rid of the adults and a layer of horticultural sand over the soil surface to keep the adults from laying eggs and to suffocate the existing eggs in the soil. Yours is one of the more cold tolerant citrus species, hardy down to -6.66 C., but irregular temperatures such as in your area could have caused the plant to discolor and defoliate. It also may have suffered transplant shock when you replaced the potting soil medium with fresh soil. Hopefully it will recover. Also taking outdoors suddenly without acclimating it gradually is like going to the beach without any protection after being indoors all season long. It is best to keep it indoors until mild temperatures for day and evening is consistent. When indoors, place in bright, indirect light, water regularly and wait a few weeks before feeding with a slow release or organic fertilizer formulated for citrus. At its current stressed state, keep the plant moist but not soggy, do not allow the plant to sit in water and do not feed until the plant recovers somewhat. Good luck!