Hi Bab,
You need to write a custom action to validate the data. We provide a sample project (check your QWP installation folder for Sample Code folder) that can be used to perform your validation.
When you select a checkbox of a choice field the data is stored in the following format:
;#One;#Two;#Three;#
where One, Two, Three… are the choice values.
You just need to parse his string and enforce your validation for this (any 5 values).
Please check the docs (see Custom action Help section) for more details.
If you have any questions please let me know.