5.7 will be available sometimes around end of March to beginning of April introducing many new features and enhancements including support for SP modal dialog with custom action.
RE: Opening Newform.aspx in Modal Dialog Window with Custom Action
5.7 will be available sometimes around end of March to beginning of April introducing many new features and enhancements including support for SP modal dialog with custom action.