[Anonymous]

I wanted to provide a bit more information about the environment.  There are 3000+ subsites and most sites have on average 3 – 5 lists / libraries that will have content that could be included.  We are looking at design architecture options that will efficiently scale.  We are open to setting up a set of master lists that would be used to collect the information involved in the rollups, but this would add complexity to our design.  We are intending on using a flag within our metadata to mark content eligible for the rollup process.  The simplest solution would be to use the web parts to handle the rollups across the site structure, but I am concerned with the overall scalability within this simplistic design.

Thanks,

Jeff Prewitt

[Anonymous]

Hi Jeff,

  Sorry for getting back to you so late.

Rolling up 3000+ sites is a big process. Then the size of those 3-5 lists in your 3000+ sites is important. If the lists are big (thousands of items) then the rollup process will be costly.

I'd suggest to use QWP CAML filter feature to filter out all the information you don't want to display. If you could implement such architecture that you can split the sites/lists based on a logic that will help.

If you already started the implementation please let us know how that goes, any issue that you see, any tips that you could share with the community will be appreciated.

[Anonymous]

We are designing our architecture for the rollup processes using Quest Web Parts for SharePoint and curious to know if anyone has suggestions or advice upon how to identify any upper limits for the number of child sites to rollup content….or best practices for scaling the rollups for large site structures.  Our rollups are occuring across web applications, but they are on the same farm.  Most of the rollups are from our Teams Web Application into the Enterprise Portal Web Application. We are working on several design options, but are looking for guidance to avoid scalability pitfalls.

Thank you for any guidance.

[Anonymous]

Quest List View Web Part offers the capability to roll up content from lists/libraries of different sites. There are many different ways to roll up the content.

• You can define multiple lists in the ViewedList property to allow qListView to explicitly rollup content from each list specified.
• You can define list with "Is Rollup List" to true, where the list contains info about the actual lists/libraries that you want to roll up content. You may specify one or more rollup lists (master lists), the list view will build the final rollup lists by querying these roll up lists. Comparing to explicitly specifying lists to be rolled up, you would incur some overhead querying the rollup list. The more rollup list (master list) you specify, the more overhead.
• Specify multiple lists in the ViewedList property and set "Include Subsites" to true, the list view will search all subsites for list whose name matched with the ListName property and roll up data in matched list. This simplifies configuration but comes with expense as list view has to recursively search through sub sites. We have Web application level setting "Max Rollup Depth" to limit the level of recursive search for rolling up data. But needless to say, if you have a deep site structure to cover, using this recursive method can add a lot of processing overhead.

For your SharePoint environment, the first option is impossible to manage and not scalable either. The second option may work better if the rollup list can be built and maintained. The third option may kill the application just to search through 3000+ subsites and the total number of lists to be aggregated could be in the range of 9,000 to 15,000, a number that exceed the default threshold Max Query Row Count of 5,000 items by SharePoint. So you would have to increase this threshold number.

Since large amount of data will be aggregated, there are a few suggestion for the potential performance hits:

• Use Next/Previous paging machanism and specify a Page Size. There is no need to bring large amount of data at once to crowd the page, instead only the specified Page Size number of rollup items will be returned. The Next/Previous style paging allows to query SP database to retrieve specific range of data where extra processing for Numeric paging on the SharePoint application side in order to get the true count of items to be returned.
• When possible, use the Turn On List Searcher Mode under the qListView display options. This option does not display any data until the user provides search criteria and helps to minimize SharePoint database hits.
• Define CAML filter in the list view to reduce the number of returns rows. 
  • Avoid defining Complex Filter unless it is absolutely needed. Complex Filter incur extra processing after large amounts of data is returned from the SharePoint database, whereas CAML filters restrict large amount of data before the data is returned from SQL Server.
  • Avoid using Sorting for the same reason above. If user changes the sort order for any column, sorting will happen to all pages. This can be a performance hog especially with roll up list–avoid roll up list, which is convenient but expensive.
  • Avoid grouping for the same reason given above as grouping is handled on the client side (Web Part processing logic).
  • Avoid putting too many number of Web Parts on one page. Each Quest Web Part has its own logic of processing data and if you have 8 or 10 Web Parts on a single page, not only they crowd the page but also the amount of work involved to render 8 or 10 Web Parts on the page can drag down the performance resulting a slow page loading and bad user experience.

[Author]

Posts