10 Tips to Optimize Data Warehouse Reporting

Unless your marketing data warehouse is stored in a massively parallel appliance such as Netezza, your data most likely resides on a Symmetric Multi-Processing (SMP) computer in a relational database engine, such as Microsoft SQL Server.

SQL Server is a highly scalable, versatile database product that can serve the needs of a wide range of applications, from the tiniest 100MB database all the way up to multi-terabyte data warehouses.  Along with its versatility comes the responsibility of the data architects to optimize the configuration of the database for the specific application.  The configuration of a transactional database, such as an order processing system, will look much different than a heavy hitting operational data warehouse.

For large-scale data warehouses, such as marketing automation databases, optimization often means providing the end users with the fastest reporting times possible.  Slow response times will quickly frustrate them, and inevitably, result in your phone ringing off the desk. 

If you are running a large data warehouse, and don’t like having the red phone light up all day long, there are a few simple rules that must be followed in order to keep your customers happy:

  1. Spread your data across as many physical disk drives as possible and as evenly as possible.  If you are building a large data warehouse, don’t put all of your data in a single file group that contains a single file.  In order to do this you will need to:
    • Create multiple file groups.  File GROUPS are logical database containers that are used to hold database files.
    • Create multiple database files for each of the file GROUPS.  These are the physical database files.  Ideally they will be spread across multiple physical drives. 
    • Rule #1: A database RECORD is always contained in a single database file.  It is NEVER spread across multiple database files, even if those database files are in the same file GROUP.
    • Rule #2: A database TABLE is always contained in a single database file GROUP.
    • Rule #3: SQL Server uses a proportional fill algorithm to spread writes across the multiple database files (within a file group).  If all database files in the FILE GROUP are the same size the same number of records will be written to each individual file (desired affect!!!).  Hence, always create database files within the same file group to have the same size.
  2. Initialize your database files to the max size that will be needed to hold your data.  This is way easier said than done.  It’s very easy to grossly under estimate the amount of data that will be flowing into your data warehouse.  When this happens the data files will auto grow, which is a SLOW process.  Even more important in the context of reporting, the new disk space added to the file group will probably not be contiguous with the original file, which will lead to slower reporting capabilities.
  3. Use the fewest number of threads possible when loading data.  If you set the MAXDOP (max degree of parallelism) to 1 the incoming data will be written to contiguous chunks of disk space.  Data warehouse reporting is typically grabbing lots of related records, usually over a time range.  This results in pulling records that are all loaded at the same time.  This is known as a RANGE SCAN.  Range scans work best when the data is stored on contiguous blocks of disk.  It’s really that simple…  By using a single thread to lay the data down on the disk, we are sacrificing load performance for improved reporting performance.  That’s OK, you only load the data once and hopefully no people are involved in the load process.  The data will be read many times over by people waiting for reports to come out of the system.  A slower load time is a fair trade for a faster user experience any day of the week.
  4. Maintain the highest quality statistics on your data as possible.  If your tables tend to grow incrementally over time, the auto statistics feature of SQL Server will work fine, for a while.  At some point the low sample rate of the auto stats will result in inaccurate statistics.  For that reason it is absolutely imperative that a rock solid statistics maintenance plan be put in place.  And don’t be fooled into thinking that the stock “maintenance plan” wizard in SQL Server will suffice.  What happens if a statistic becomes severely out of date?  The query optimizer will generate a bad plan for the SQL query and the query will appear “hung”.  The consumer of the data will think the server is busy or that you have a bad design, when in fact you just have outdated statistics.
  5. Give SQL Server as much memory as you can afford.  Whatever you give it, it will use.  If your server is a dedicated database  server, turn on Lock pages in memory.  This will prevent SQL Server from ever giving up memory after it has allocated it.  If your server is used for other applications however, locking memory pages may end up starving those applications of memory, which may end up making the database look slow.
  6. On a related note, separate services when ever possible.  SQL Server Integration Services (SSIS), like the SQL Server engine, will grab as much memory as is available.  If you just gave all of the system memory to SQL Server (minus a few Gig so you don’t starve the OS) what will be left for your ETL/loads?  Put SSIS on a separate server.
  7. If you have tables that are loaded once a month and then used for reporting purposes the rest of the month, consider placing these tables in READ ONLY file groups.  The SQL engine will reward you with a drastic reduction in locks locks on your data, and your data consumers will think you are a genius.
  8. For very large tables, such as more than 100M records, consider a horizontal partitioning strategy.  SQL Server does not have a nice wizard to setup partitioning.  It is very technical to implement and deploy.  But if you have a table with 200M records, but usually don’t care about any of the data except for the most recent month, horizontal partitioning will allow correctly designed queries to ignore all but that most recent month’s data.  This is called partition elimination, and can/will make a huge difference in query performance.
  9. Do your housecleaning…  Get rid of indexes that are no longer needed.  Add new indexes sparingly and with great caution.  But do so when it makes sense.  I’ve never seen a perfectly indexed database in the initial design.  Drop “temp”/”test” tables.  Archive old data.  Better yet, deploy an automated archiving process.  There’s no point in keeping 10 years’ worth of data when your end users only need the current year plus last year.
  10. And last but not least, my personal favorite, NEVER SHRINK a database or database file.  If you do, you just fragmented your data and indexes, probably FOREVER.  The only way to get rid of that fragmentation will be to move the data to a new file group that isn’t fragmented.  The catch 22… if you just did a shrink on your database it’s probably because you were running low on disk space.  If you don’t have disk space to auto-grow your existing data files, you probably don’t have the disk space to move the data to a new file group.  The data will remain fragmented, FOREVER.

Your reporting customers want their data as simply and as quickly as possible.  Making that happen only requires following some basic design and maintenance strategies.

What strategies have you employed to avoid a data scare-house?


Leave a Reply

You must be logged in to post a comment.

Using Gravatars in the comments - get your own and be recognized!

XHTML: These are some of the tags you can use: <a href=""> <b> <blockquote> <code> <em> <i> <strike> <strong>

popular personal essay writers for hire usa buy astronomy presentation write my religious studies thesis statement help me write professional phd essay on trump esl dissertation results writing for hire best custom essay ghostwriter for hire for college best admission essay writers service au professional dissertation hypothesis proofreading services for college professional masters blog post topic custom academic essay editor for hire for mba top curriculum vitae editing service au help with my custom critical thinking online pay to write cheap admission essay best blog post writing services cheap paper ghostwriters services for university professional homework writer website gb top dissertation hypothesis editing services uk custom essay ghostwriter for hire for university esl problem solving editor site us type my classic english literature report help me write custom reflective essay esl critical analysis essay proofreading for hire ca cheap curriculum vitae ghostwriting site us best reflective essay ghostwriters websites for mba top dissertation hypothesis editor sites usa cheap content editing for hire for college popular report ghostwriting services usa popular admission essay editor services usa descriptive essay writer site uk custom dissertation abstract editing sites cheap research proposal editing service usa best school essay writers service online cheap college cheap essay advice buy english report professional blog editor website for masters professional dissertation ghostwriting websites online esl case study ghostwriting service us pay to do top problem solving online cheap critical analysis essay proofreading sites for college cheap article review ghostwriters sites uk soft shell lobster top article ghostwriting websites au custom creative essay writing service gb popular blog post ghostwriter site for university top expository essay writers websites custom university essay editing sites for university cheap mba admission essay ideas do my human resource management creative writing dissertation ghostwriter websites au professional speech editor websites for masters best writing service online esl analysis essay writer website uk professional school homework assistance best dissertation methodology writing for hire for phd best literature review ghostwriting websites for phd popular application letter writing websites for college esl letter writer website gb top definition essay writer site best creative writing writers sites online popular biography editor sites for university culture essay writing for hire best cover letter ghostwriter website for mba top presentation ghostwriting services usa professional dissertation methodology proofreading services for college esl blog post writers sites for masters esl speech editor website usa popular university essay ghostwriting sites for phd professional dissertation methodology writers websites for masters cheap papers writing service gb top papers writer sites for phd esl descriptive essay editor sites for university top home work writer website for university top dissertation results editing service for mba popular thesis proposal writers for hire online cheap definition essay ghostwriter services usa professional scholarship essay editor site us esl bibliography editing site online type my family and consumer science curriculum vitae cheap expository essay proofreading for hire critical analysis essay proofreading services au law essay sample top personal statement editor site buy professional letter top academic essay ghostwriters service for school popular blog ghostwriters sites us best mba essay writer sites ca reflective essay for university write my engineering application letter cheap masters essay writers services for mba professional presentation ghostwriter website ca ancient egyptian paper best admission essay writing websites for university custom thesis writer for hire us custom reflective essay ghostwriter service for school popular college essay ghostwriter site online custom definition essay writing for hire gb write environmental studies bibliography popular essays ghostwriter site uk cheap assignment ghostwriters service for mba custom paper writing sites gb pay to write admission essay on lincoln professional personal statement proofreading websites for masters cheap scholarship essay writing sites ca top content ghostwriting service gb popular analysis essay editing website online esl report writing service for mba help with my ancient civilizations dissertation chapter top problem solving ghostwriters site online popular course work ghostwriters website for university best definition essay ghostwriter site for university cheap analysis essay writer service for school help with trigonometry dissertation methodology custom editor service for masters preschool child observation report pay to do communication term paper pay to do health course work popular essays ghostwriting for hire online esl thesis proposal writing websites for mba type my top best essay professional research paper ghostwriter service usa professional research proposal proofreading service esl dissertation proposal ghostwriter service for university popular creative writing ghostwriter websites ca best blog ghostwriting site for college best content writer service gb professional presentation ghostwriting site uk esl university dissertation chapter example popular mba essay writers sites online popular dissertation introduction ghostwriters site au liberal view on poverty order top scholarship essay on usa top scholarship essay proofreading service ca top annotated bibliography writers for hire for phd cheap mba dissertation methodology examples the buried life summary popular argumentative essay writer sites for masters best research proposal editor sites for phd professional research proposal proofreading websites usa top papers editor sites uk esl biography proofreading service us cheap college cover letter samples popular college creative writing examples pay for custom problem solving online personal proofreading service au custom dissertation abstract writers websites online cheap research paper writing site online best bibliography editing services gb hamlet comic relief top university biography topics top scholarship essay writers website gb esl term paper editor website for masters top term paper ghostwriters services us esl admission essay writers for hire for university personal statement editor for hire online top blog writers site au cheap paper editing services for college business essay writer sites top speech editing for hire us pay for my custom reflective essay on donald trump cheap dissertation chapter writers websites for school professional report ghostwriter service for college best admission paper ghostwriting services for masters custom blog editor site for mba julius caesar paper help with my esl persuasive essay on civil war custom thesis writing service for university custom scholarship essay writer services ca a seperate peace theme write esl expository essay on usa community identity stability best movie review ghostwriters sites for university custom article review editing websites for school pay for professional rhetorical analysis essay on brexit community essay sample top resume ghostwriting site for phd esl article review writing services ca best book review proofreading website for mba custom thesis writing site au popular admission paper ghostwriter website au best book review editing websites usa cheap homework editor website for university cheap cover letter writer websites for mba just lather esl university academic essay example esl paper editing services ca best university cv example pay to write popular admission essay on lincoln professional research paper editing sites online top case study ghostwriter site for college top speech ghostwriting website for mba professional speech writing website for university custom course work proofreading for hire for college professional dissertation methodology editor for hire for masters popular rhetorical analysis essay proofreading service au critical essay ghostwriting for hire uk help with my esl custom essay on usa popular dissertation hypothesis ghostwriter for hire for mba best thesis ghostwriters service for mba pros of racial profiling custom critical analysis essay ghostwriting site usa pay to get phd essay on usa cheap cv ghostwriters for hire usa esl research proposal ghostwriting service gb write me esl book review esl best essay proofreading site help me write shakespeare studies cover letter help me write best critical essay on presidential elections professional cv ghostwriting sites for mba write my best bibliography online best masters thesis advice top analysis essay writers website for university website review essay best analysis essay writer site for university pay for my professional creative essay on presidential elections custom content writing sites au best reflective essay ghostwriting for hire usa professional curriculum vitae editor website au esl dissertation chapter writer websites usa custom phd biography topic cheap phd curriculum vitae advice popular book review ghostwriter service for university write top business plan online blog ghostwriters sites online esl university assignment sample custom essays ghostwriters for hire for school custom essay ghostwriters service top persuasive essay ghostwriting service for school custom reflective essay editor websites ca top academic essay writers service for phd best writer site au professional dissertation proposal writers service for college top admission essay editor for hire for school best dissertation results editor service for mba popular rhetorical analysis essay editor site au cheap phd essay ghostwriters website for university custom scholarship essay writing service usa blog post ghostwriters service au best phd essay ghostwriter website for school cheap biography writers website us custom argumentative essay proofreading websites for mba cheap mba business plan topics help writing life science thesis write medicine letter popular academic essay ghostwriting services au pay to do popular phd essay on presidential elections professional admission paper ghostwriting for hire gb professional content writers sites for school help me write best masters essay on lincoln esl course work ghostwriter site uk popular article ghostwriter for hire for phd custom critical essay ghostwriting sites uk scholarship editor website usa mom is best popular blog post ghostwriter sites au cheap academic essay writer service au cheap scholarship essay writers services au cheap dissertation writing for hire usa top thesis proposal ghostwriter service gb professional academic essay ghostwriting service for university dissertation abstract writing for hire gb popular content writer website au esl annotated bibliography writers website uk best critical analysis essay writers services for university custom dissertation results editing websites for masters love in la sparknotes top biography writing for hire uk best academic essay ghostwriters site for university top report ghostwriters service for college popular dissertation introduction writer services for masters professional best essay editor services us popular article review ghostwriters services for college custom personal essay on usa custom thesis proposal ghostwriters websites for mba top annotated bibliography proofreading service for university best critical analysis essay writing website ca popular dissertation abstract ghostwriting services us popular blog writing websites best school essay writing sites for masters pay to write art & architecture bibliography top research paper writers website for phd top literature review ghostwriter website gb cheap college movie review socialization and culture top letter editing site au top dissertation results writers sites for school popular cv editing site for university popular thesis writing websites au best creative essay ghostwriting service online cheap literature review editor site au custom college course work example help me write popular creative essay on hacking top academic essay ghostwriter service best critical thinking editor website us professional masters expository essay help best rhetorical analysis essay ghostwriting services uk popular expository essay writer service for school professional homework proofreading website for mba cheap article editing site for college popular school cv examples logic essay ghostwriting site professional mba dissertation proposal esl dissertation ghostwriter services for phd phd essay on history best dissertation methodology ghostwriter website au popular book review proofreading site for university popular course work ghostwriter website usa cheap creative essay ghostwriting for hire for phd esl creative writing writing for hire for school elvis presleys wife cheap dissertation hypothesis ghostwriting for hire for university what are western religions top blog ghostwriting sites ca help writing best analysis essay on hillary cheap dissertation proposal writing for hire au help with custom admission paper buy shakespeare studies thesis top critical essay editor service online esl expository essay writers site custom personal essay editor service for school popular blog ghostwriter sites au type my professional critical essay on shakespeare custom persuasive essay writing sites us esl paper editing site us pay to write health speech cheap essays writers website usa esl university essay ghostwriting website for phd cheap best essay writer services gb cheap scholarship essay proofreading website for phd help with best literature review online custom creative writing writers service au professional annotated bibliography ghostwriter services ca dissertation chapter ghostwriter sites us professional dissertation proposal writer website for mba literature review writers services ca order classic english literature resume persuasive editor services us popular best essay writing for hire for university rhetorical analysis ghostwriter websites au top curriculum vitae proofreading service gb professional research proposal proofreading site for masters popular business plan ghostwriting site for college custom university essay ghostwriting website for mba type my women and gender studies creative writing best mba essay ghostwriters for hire us cheap phd essay ghostwriter service best dissertation abstract ghostwriting website for phd esl speech proofreading sites for school custom argumentative essay editing site for mba custom school letter assistance best dissertation conclusion editor websites for phd do my custom masters essay on shakespeare esl curriculum vitae editor service custom academic essay writing for hire au type my custom academic essay on usa professional critical thinking ghostwriters site for masters esl book review writers websites au professional creative essay ghostwriter websites for university pay to write investments admission essay physics writing websites top critical thinking writers sites ca case study ghostwriter site ca professional presentation ghostwriter site for school cheap argumentative essay editor websites for school popular course work ghostwriters website uk top course work ghostwriter service for school help writing statistics dissertation conclusion popular personal essay writing services online life science essay writers for hire custom thesis proposal writer services uk esl bibliography ghostwriting service cheap university business plan topic best dissertation results editor service ca cheap curriculum vitae editor sites for school best biography editor website au professional argumentative essay writing service top book review writer websites for school popular dissertation ghostwriter site usa custom phd essay ghostwriter website top college essay ghostwriter site online esl blog ghostwriter websites online best best essay writers services us buy logic cover letter professional scholarship essay ghostwriting for hire for university help writing film studies cv esl proofreading website au pay for tourism bibliography top annotated bibliography writing website uk business analysis paper popular rhetorical analysis essay writing websites for phd an image of africa meaning of true friendship cheap cover letter ghostwriters services popular dissertation ghostwriters site usa write me ancient civilizations movie review order world literature book review professional admission essay proofreading website uk popular course work ghostwriter sites for university biography proofreading for hire online ahab in moby dick nature vs nurture research paper custom college dissertation ideas cheap school annotated bibliography advice best course work ghostwriter for hire uk esl article review editing for hire for mba custom definition essay writing site for phd pay to write anthropology article review professional phd dissertation sample cheap book review writer websites custom dissertation results ghostwriter for hire us pay to write communication book review popular article ghostwriter sites online esl speech proofreading website for phd cheap essays writers services au professional masters essay ghostwriting service for phd top business plan writing services for university best curriculum vitae writing website au professional university admission essay advice popular expository essay ghostwriter websites pay to get theater studies dissertation results popular custom essay ghostwriter website for masters custom application letter editing websites for college symbols in the metamorphosis professional thesis proposal ghostwriter service for school popular mba term paper topic breathing underwater sparknotes great depression essay introduction earth science ghostwriters sites cheap personal statement writing services for college custom speech ghostwriting website us dissertation writing services gb popular home work ghostwriters services usa custom descriptive essay writing websites usa professional blog post writers sites for mba popular letter proofreading services for college pay to write custom academic essay on usa best blog editing sites for phd esl admission paper proofreading services uk professional dissertation chapter ghostwriting websites for masters cheap university essay ghostwriter site au custom papers editing site for mba top university essay proofreading website ca help with top dissertation conclusion esl dissertation conclusion editor site au popular blog post writer site for masters esl scholarship essay writers services custom business plan ghostwriter website online esl essays proofreading for hire for mba professional thesis statement writer site uk best article ghostwriters sites ca united methodist communion order earth science admission essay professional personal statement proofreading service for school custom problem solving writers services for school professional homework ghostwriter website ca college ghostwriting websites au logic writing site popular best essay editor services au cheap bibliography ghostwriter for hire uk best report editor site au cheap thesis proofreading site gb professional thesis statement ghostwriter services gb my personal philosophy professional report editing service for university custom ghostwriters website for masters popular scholarship essay proofreading sites usa help me write top blog online top thesis statement writer websites for university cheap assignment ghostwriter services top research paper proofreading site for college cheap research proposal ghostwriter websites online best academic essay proofreading services au pay to get english dissertation hypothesis cheap dissertation abstract ghostwriters sites professional phd resume assistance help with music thesis proposal buy human resource management admission paper professional blog post writer for hire ca best college curriculum vitae samples esl paper ghostwriter service best best essay writing for hire for phd write me best analysis essay online cheap application letter writer sites for mba custom rhetorical analysis essay editing site for college professional problem solving proofreading service for phd esl blog post editing site au reflective editing sites usa esl dissertation conclusion proofreading sites au help with my music home work communication writers services professional phd dissertation hypothesis advice help with my top masters essay on hacking best term paper ghostwriter service for school cheap curriculum vitae editing sites for mba pay to do popular school essay on trump help me write esl university essay on brexit custom research paper writing website gb esl academic essay proofreading for hire top argumentative essay writing services usa popular argumentative essay proofreading for hire for university pay to do history annotated bibliography