Hi Nate,
Thanks for reporting your findings. The "Check User Access" option when enabled is supposed to hide menu items pointing to site pages to which the current user does not have permission to access from that user.
However I can confirm this feature is not currently working in our latest code streams, so I have logged this issue (no:213711) and it will be addressed in an upcoming release.
Thanks,
Yuan