Thinning gelcoat should only be done when it needs to be, such as for spray applications. For most applications, gelcoat should only be thinned with styrene (which creates the very recognizable smell that gelcoat and polyester resin emit). There are also specialty thinning products designed for gelcoat application.
One of the most commonly used products for thinning gelcoat is acetone, and though this product will lower the viscosity, it has a strong tendency to adjust the color of the material as it cures.
Products such as Sea Hawk Patch Aid 8185 and Dura Technologies Duratec® are specialty products designed to improve the working and cured properties of gelcoat, while thinning it.
Each application and brand of gelcoat will require different amounts of thinner for a specific application, so there really is no short answer. For small repairs, a small percentage of thinner may be needed for flow or self-leveling, but some gelcoat guns may require more to spray the material properly. Patch Aid and Duratec also have specific directions on how much to use.