Yes. You do not have the absolute right to choose when you take your holiday, and your employer can tell you when to take your leave.
However, unless your contract specifies otherwise, your employer must give you at least two days' notice for every day they want you to take. So if your employer is telling you that you must take four days' holiday, they need to tell you this at least eight days' in advance. Your employer can refuse your request for a specific holiday date (provided it has a legitimate reason), but you cannot so easily turn down your employer's instruction to take leave.