Hi Edward
It is sure a surprise to hear that it took 20 to 40 seconds to load a page with couple of web parts fetching 500 some list items. I would first isolate the components involved by removing the web parts on the page and measure the performance one by one to find out which Web Part on the page is actually causing the performance issue. Then we can look closely to see what could be the cause.
In the past we have cutomers experienceing performance issue related to qListView. There are tips and best practices that can go with the configugration of the qListView and design of the lists connected. We can get you that. In one particular case, we also found out that a particular version of IE browser plays an important role, where using a different version of IE or chrome or Firefox does not have the same performance issue but that is the only case we have run into.