There are a few major differences between using epoxy resin vs. polyester resin, and for many applications, they are not interchangeable.
Substrate compatibility is more diverse for epoxy products and epoxies generally will have better secondary bond characteristics than polyester resin products. Polyester resin products generally stick to wood, other fiberglass or polyester resin substrates, and metals. Epoxy generally sticks to all of the same substrates, as well as some plastics, when properly treated. Linear polyurethane and acrylic urethane paints will also bond to epoxy-based primers.
Primary or secondary bond: Polyester resin and epoxy resin both create very good primary bonds, so new construction using either resin works very well. Secondary bonds occur when you add onto an existing item, such as making a repair to a fiberglass boat. Polyester resin products have a lower bond strength than epoxy resins.
Strength, dimensional stability, rigidity, weight, and water resistance: Epoxy has an advantage in all of these categories. Polyester resin has lower strength, is more prone to changing shape after it has cured, requires more resin to saturate glass mat and cloth, adding extra weight (also requires more cloth material and resin to reach the same physical properties as an epoxy laminate), and is more susceptible to water intrusion through osmosis (many people recognize this as osmotic blisters below the waterline).
The cost of epoxy resin is a bit higher than polyester resin products, which is commonly the deciding factor in which resin to use.