If you are only getting the legal minimum then it makes no difference what your usual working days are, your entitlement is based on the length of your normal working week.
But if you work for an employer who does better than the legal minimum, it is entirely possible, if unfair, that you could lose out as long as you are getting more than the legal minimum.
Entitlement to bank holidays, and any extra leave above the four weeks' statutory minimum, are governed by your contract of employment. You should check your staff handbook, and discuss this with your employer, but be aware that if your contract does not specify anything to the contrary, your employer's decision on this will be final.
Some employers may simply not have realised that they have this unfairness built into their system, and a simple request may secure change.
Others may not be so sympathetic, and this is an obvious case to involve a trade union if you, and preferably your colleagues too, are members.
Just One More Click: Doggy drama about staying safe online at work.