Always check the directions on the paint can first for recommendations on roller type.
Many bottom paint manufacturers recommend using a 3/8" nap roller. A 3/8" nap will apply a thick coat and an orange peel-type finish because of the heavy nap of the roller. On ablative paints, that orange peel texture will smooth out some as the boat moves in the water.
A smooth, slick finish results in faster speed through the water. If you want a smoother finish, use a 3/16" to 1/4" nap roller.
A foam roller can be used, but foam rollers lay down a thinner coat of paint. You'll have to build up proper mil thickness, as specified by the bottom paint manufacturer, in order to get proper antifouling protection from your bottom paint. Building mil thickness with a foam roller will require applying an additional coat(s). Ideally, you should use a solvent-safe foam roller cover. This type of roller cover is constructed so it won't fall apart when applying solvent-based paints.