Hi adamst,
We have just recently implemented a new feature leveraging the dialog framework new to SharePoint 2010. The user will be able to choose between opening a form launched by a custom action in the current window, a new browser window, or a SharePoint dialog. This feature is brand new and it will be available in the next release (5.7) of Quest Web Parts so it is not supported just yet but will be once you upgrade to v5.7.
Thanks,
Yuan