The choice of brush type for varnishing is dependent both on the varnish material, and the preference of the person varnishing. Wider brushes are better for varnishing wider pieces with greater consistency, and narrow brushes provide more precision for finer pieces.
Oil-based varnishes and spar varnishes are the most common varnishes in the marine industry. Oil-based marine wood finishes also fall under this category, though they are truly different than a varnish. These products apply the best with the control of a natural bristle brush. Badger hair and other brush types fall within this category. The higher the bristle count, the smoother the finish will lay down. Thicker brushes will hold more varnish and allow you to reload the brush fewer times while applying. A successful varnish job is still a balance of varnish viscosity, brush width, having the right brush type, and having a feel or technique to match.
Many people ask if you can just use a foam brush when varnishing or applying an oil-based wood finish. Just about every professional varnish applicator will cringe at the thought, but many will say a foam brush will be suitable on the sealer coats or on some build coats—but the final coats should always be with a high-quality, natural bristle brush. A foam brush will never have the feel or control that a bristle brush will give.
Water-based wood finish products generally require nylon or poly (synthetic) bristles as they do not absorb the water and swell up as a natural type bristle would. Foam brushes can also be used with water-based products, but they lack the feel and controlled flow of a bristle brush. Water-based products have a distinct flow characteristic. Bristle brushes give you a little more control with how much material you apply at a time, but over-brushing will remove material, and tends to make it look inconsistent. Apply enough material for consistency and self-leveling, but not enough to sag or drip.
Two-component varnishes are also a little trickier to apply by 'brushing' as you would a regular oil-based varnish. Natural bristle brushes are commonly used for brushing applications though foam brushes can be used for smaller pieces. Keep in mind that the foam may be dissolved or it may swell up from the solvents in two-part varnishes. Two-part varnishes are a little more like water-based products in that they do not really do well when over-brushed. These products commonly like to flow off the brush on their own, as opposed to being overworked and constantly brushed. Their self-leveling properties tend to be chemically engineered into the products, so you simply need to apply the product evenly, to the recommended application thickness, and control any drips or sags.