Don't paternity rights only apply to men in married couples?

No. Although this leave and pay is called ‘Statutory Paternity Leave’ and ‘Statutory Paternity Pay’ it applies to all couples, whether heterosexual or same-sex and whether married, in a civil partnership or co-habiting where one person in the couple is giving birth to, or adopting, the child.