Using EVENT NOTIFCATION to ALERT on user query CPU usage

We often have customers running ad hoc queries on the databases that we host.  Most of the time the queries run fine… sometimes they don’t.

The first step we took towards managing rogue queries was to deploy Resource Governor (only available in Enterprise or Developer editions) to limit their MAXDOP.  In general, a rule of thumb is to limit ALL queries to the number of physical cores on a single socket.  (The basis for this rule is related to NUMA, and SQL Server’s “NUMA awareness”, which I will not go into here.  If you really want the authority on these matters check out Glenn Berry’s work.)  By limiting the MAX DOP we can at least take comfort in knowing that all of the CPUs will not be consumed by a bad query.

The next step was to create an alert that fires when the user query (not system, an individual query) reaches a specified CPU threshold, in seconds.

It turns out there is a simple way to do this using the standard alert configurations in SSMS.  Unfortunately, the counter behind the alert never changes from zero (at least not in SQL 2008R2SP2CU3).  So I scrapped that approach.

The next approach involved SERVICE BROKER, EVENT NOTIFICATIONS and in this case, RESOURCE GOVERNOR.  That sounds like a lot of moving parts to create an alert.  Fortunately, I found an article by Aaron Bertrand, who leveraged work by Jonathan Kehayias, that provided me with a good foundation to develop my own, extensible alerting framework.

The following is an excerpt from an email that was sent to my internal team.  I had to tweak it a little to make it more generic, but the concepts remain the same.

I’ve added the TSQL in the New Downloads section (to the right of this screen) called Event Notifications Framework.  You can use that code to build your own set of trace event alerts.  I only request that you not remove any of the credits, myself as well as the link to Aaron’s post.

—————————————————————————————————

OBJECTIVE: Create a database alert that will fire an email to support whenever an external user query exceeds a CPU threshold, such as 1 minute or 1 hour.

Q: I thought we already had a “longest running transaction” alert.  What’s the difference?

A:  We do.  The LONGEST RUNNING TRANSACTION alert fires when a TRANSACTION (INSERT, UPDATE, DELETE, MERGE) runs for longer than a fixed threshold.  For HFC that threshold is 8 hours.  There are intrinsic weaknesses in this alert:

  • It does NOT apply to SELECT queries, which are usually not part of a TRANSACTION.
  • If multiple transactions are running above the 8 hour threshold, the alert will only fire for the first transaction that crosses the threshold.  If the second transaction finishes before the first transaction, it will never be recognized.
  • Simply put, this counter does nothing to warn us of runaway, ad hoc SELECT queries.

Q: Why didn’t we create a simple “long running SELECT query alert” in the first place?

A:  It should be possible to create a high CPU alert using a standard SQL Server alert configuration.  The configuration screen below “should” work… but it doesn’t.  It turns out that the counter is always zero.  It is a bug that was identified in the original release of SQL Server 2008 and has not been corrected in any of the subsequent service packs (SP) or cumulative updates (CU).  Don’t worry… I tried it anyway.

clip_image002

Q: How does this “alternative” approach work?

A:  First, a couple of SQL Server subsystems need to be configured.

  • Deploy Resource Governor (RG).  RG makes it real easy to:
  • Categorize user logins into certain “buckets” using a “classifier function”.  We then use these buckets to tell the SQL Engine how much resources the users in those groups can consume.  (In most cases, only when there is contention on the system.)
  • “Limit” any user’s query to a specified number of CPU seconds. We can do it when we configure RG by setting the REQUEST_MAX_CPU_TIME_SEC.  (In SQL Server, a “request” is just a confusing term for a query of any sort.)

clip_image004

  • Next we need to configure SERVICE BROKER (another subsystem of the SQL Server engine) to send out the alert.  SERVICE BROKER is a fancy (misleading) name for an internal, queue-based, asynchronous messaging system.  Specifically to this alert, when the CPU time for a user query goes above the 5 second threshold, an XML “message” is placed on a special SB queue.  Once it is on the queue, it will stay there until some other process comes along and “consumes” it.
  • EVENT NOTIFICATIONS, is what makes it possible for the trace event that was set by resource governor to be placed on the SB queue.  There is nothing to configure… it just runs.  Event notifications was introduced in SQL Server 2005 so it’s hardly new technology.
  • A special stored procedure “service” was written that acts as the consumer of the XML events that are placed on the SB queue.  Ultimately, it sends out the email alert with the details of the event.

Q: Why do we need SERVICE BROKER/EVENT NOTIFICATIONS to process the event?  Shouldn’t Resource Governor have some way to flag an administrator if the CPU threshold is exceeded?

A: For starters, we really would not want RG to be automatically killing queries that exceed a certain CPU threshold.  The ONLY response RG does when the CPU threshold is exceeded is raise a “trace event”.  Once the trace event is raised, EVENT NOTIFICATIONS places the event on the SB queue.  What we DO with that information is entirely up to US, the watchdogs of our systems.  There are other “limits” that RG does place a hard stop on, such as the MAX DOP.  But in the case of the CPU threshold, it doesn’t make sense.  The response to the event simply needs to be customized.

Q: What will the email alert look like?

A: The email message will prominently include:

  • The originating server
  • The CPU time in milliseconds that the query has been running before the ALERT was processed from SB
  • The SPID of the offending query session.

SERVICE BROKER and RESOURCE GOVERNOR are only available in the ENTERPRISE edition of SQL Server.  They are also at the top of the list of subsystems that are too often not used (aka, least understood).  But their potential value is immense.

clip_image006

Q: If the alert fires, how can we determine which user queries were running at the time of the event?  We may not see the alert until several hours after the event occurs.

A:  One of the “responses” that has been configured is to call sp_whoisactive and save the results into a table in DBA_BASELINES.  This will capture all requests (active queries) that are running on the system.  From there we should be able to narrow down the offending query/user very quickly. (Amplification: We have a SQL Agent job that calls Adam Machanic’s sp_whoisactive, and writes the result set into a table.)

Q: Can the CPU threshold fire even if the elapsed time (wall clock) of the query is less than the threshold?

A: Based on my knowledge of query analysis, YES.  A CPU bound query, such as one that is dominated by a large SORT operation, that is running on multiple CPUs, could easily use up the 5 seconds of CPU in less than 5 seconds of ELAPSED time.  In an extreme case, 5 seconds could be used by 4 CPUs running in parallel in 1.25 seconds.

Q: In the sample email message the reported CPU time is 8.376 seconds.  Why didn’t the event fire on 5 seconds, which is the preset threshold value in RESOURCE GOVERNOR?

A: Service Broker is an ASYNCHRONOUS queue manager.  This means that it will process the incoming event when it can; which may not be immediately.  The delay could be UP TO 5 seconds after the trace event fires.  In the real world we will be concerned with queries that are running for at least 15 minutes.  Another 5 seconds (max) added to the runtime is insignificant.

Q: If the user queries runs for hours and hours through the night will my inbox be inundated with this alert?

A: No.  This alert will fire ONCE and ONLY ONCE for each query that exceeds the CPU threshold.

Q: Will this approach create a high overhead on these busy database servers?

A: Two out of the three components of this system are always running anyways… SERVICE BROKER and EVENT NOTIFICATIONS.

SQL Server uses SB for its own management of processes.  All instances of SQL Server have SB running in the background.  It’s engineered into the SQL Engine.  EVENT NOTIFICATIONS is very lightweight and consumes minimal system resources.  It’s running all the time for SQL Trace, Profiler and the DEFAULT trace.

The only “optional” component of this solution, RESOURCE GOVERNOR, which has already been proven to be a valuable tool with managing system resources, is already in place.  By all accounts, this approach to converting trace events into alerts has a very low overhead on the database engine. (Note: Resource Gov is NOT needed to process trace events, only the trace event that is used by RG to limit REQUEST_MAX_CPU_TIME_SEC.)

Q: If we now have a FRAMEWORK to process alerts, what other “events” can be added to our pool of interesting alerts?

A: Anything that can be monitored in SQL Profiler (or SQL TRACE) can be monitored and alerted on using EVENT NOTIFICATIONS.  For the CPU Usage alert we only needed RESOURCE GOVERNOR to set the max threshold.  Otherwise, it isn’t really needed for this framework to function.  [I believe DDL triggers can also be processed with this framework but I have not tested any.] Adding any event that is available from sys.trace_events requires two simple steps:

1. Create a new notification.  Such as…

clip_image008

2. Add a block of code to the Service Broker “Service” dbo.dba_processEventNotifications.  This custom “service”:

  • Pops the trace events off of the SB queue.
  • Reads the XML out of the SB message.
  • Formats the email SUBJECT and BODY content and sends out the DB Mail message that contains the alert information.

Basically, the following block of code would need to be added.  The @message section would need to be tweaked to pull out the relevant attributes from the XML message.  For example, CPU would not be relevant to an alert triggered by a database file autogrowth, but the database file name would be of interest.

clip_image010

Once again, the source code for this is available in the NEW DOWNLOADS section.

Stay curious, keep learning…

0 Comments

Leave a Reply

You must be logged in to post a comment.

Using Gravatars in the comments - get your own and be recognized!

XHTML: These are some of the tags you can use: <a href=""> <b> <blockquote> <code> <em> <i> <strike> <strong>

popular personal essay writers for hire usa buy astronomy presentation write my religious studies thesis statement help me write professional phd essay on trump esl dissertation results writing for hire best custom essay ghostwriter for hire for college best admission essay writers service au professional dissertation hypothesis proofreading services for college professional masters blog post topic custom academic essay editor for hire for mba top curriculum vitae editing service au help with my custom critical thinking online pay to write cheap admission essay best blog post writing services cheap paper ghostwriters services for university professional homework writer website gb top dissertation hypothesis editing services uk custom essay ghostwriter for hire for university esl problem solving editor site us type my classic english literature report help me write custom reflective essay esl critical analysis essay proofreading for hire ca cheap curriculum vitae ghostwriting site us best reflective essay ghostwriters websites for mba top dissertation hypothesis editor sites usa cheap content editing for hire for college popular report ghostwriting services usa popular admission essay editor services usa descriptive essay writer site uk custom dissertation abstract editing sites cheap research proposal editing service usa best school essay writers service online cheap college cheap essay advice buy english report professional blog editor website for masters professional dissertation ghostwriting websites online esl case study ghostwriting service us pay to do top problem solving online cheap critical analysis essay proofreading sites for college cheap article review ghostwriters sites uk soft shell lobster top article ghostwriting websites au custom creative essay writing service gb popular blog post ghostwriter site for university top expository essay writers websites custom university essay editing sites for university cheap mba admission essay ideas do my human resource management creative writing dissertation ghostwriter websites au professional speech editor websites for masters best writing service online esl analysis essay writer website uk professional school homework assistance best dissertation methodology writing for hire for phd best literature review ghostwriting websites for phd popular application letter writing websites for college esl letter writer website gb top definition essay writer site best creative writing writers sites online popular biography editor sites for university culture essay writing for hire best cover letter ghostwriter website for mba top presentation ghostwriting services usa professional dissertation methodology proofreading services for college esl blog post writers sites for masters esl speech editor website usa popular university essay ghostwriting sites for phd professional dissertation methodology writers websites for masters cheap papers writing service gb top papers writer sites for phd esl descriptive essay editor sites for university top home work writer website for university top dissertation results editing service for mba popular thesis proposal writers for hire online cheap definition essay ghostwriter services usa professional scholarship essay editor site us esl bibliography editing site online type my family and consumer science curriculum vitae cheap expository essay proofreading for hire critical analysis essay proofreading services au law essay sample top personal statement editor site buy professional letter top academic essay ghostwriters service for school popular blog ghostwriters sites us best mba essay writer sites ca reflective essay for university write my engineering application letter cheap masters essay writers services for mba professional presentation ghostwriter website ca ancient egyptian paper best admission essay writing websites for university custom thesis writer for hire us custom reflective essay ghostwriter service for school popular college essay ghostwriter site online custom definition essay writing for hire gb write environmental studies bibliography popular essays ghostwriter site uk cheap assignment ghostwriters service for mba custom paper writing sites gb pay to write admission essay on lincoln professional personal statement proofreading websites for masters cheap scholarship essay writing sites ca top content ghostwriting service gb popular analysis essay editing website online esl report writing service for mba help with my ancient civilizations dissertation chapter top problem solving ghostwriters site online popular course work ghostwriters website for university best definition essay ghostwriter site for university cheap analysis essay writer service for school help with trigonometry dissertation methodology custom editor service for masters preschool child observation report pay to do communication term paper pay to do health course work popular essays ghostwriting for hire online esl thesis proposal writing websites for mba type my top best essay professional research paper ghostwriter service usa professional research proposal proofreading service esl dissertation proposal ghostwriter service for university popular creative writing ghostwriter websites ca best blog ghostwriting site for college best content writer service gb professional presentation ghostwriting site uk esl university dissertation chapter example popular mba essay writers sites online popular dissertation introduction ghostwriters site au liberal view on poverty order top scholarship essay on usa top scholarship essay proofreading service ca top annotated bibliography writers for hire for phd cheap mba dissertation methodology examples the buried life summary popular argumentative essay writer sites for masters best research proposal editor sites for phd professional research proposal proofreading websites usa top papers editor sites uk esl biography proofreading service us cheap college cover letter samples popular college creative writing examples pay for custom problem solving online personal proofreading service au custom dissertation abstract writers websites online cheap research paper writing site online best bibliography editing services gb hamlet comic relief top university biography topics top scholarship essay writers website gb esl term paper editor website for masters top term paper ghostwriters services us esl admission essay writers for hire for university personal statement editor for hire online top blog writers site au cheap paper editing services for college business essay writer sites top speech editing for hire us pay for my custom reflective essay on donald trump cheap dissertation chapter writers websites for school professional report ghostwriter service for college best admission paper ghostwriting services for masters custom blog editor site for mba julius caesar paper help with my esl persuasive essay on civil war custom thesis writing service for university custom scholarship essay writer services ca a seperate peace theme write esl expository essay on usa community identity stability best movie review ghostwriters sites for university custom article review editing websites for school pay for professional rhetorical analysis essay on brexit community essay sample top resume ghostwriting site for phd esl article review writing services ca best book review proofreading website for mba custom thesis writing site au popular admission paper ghostwriter website au best book review editing websites usa cheap homework editor website for university cheap cover letter writer websites for mba just lather esl university academic essay example esl paper editing services ca best university cv example pay to write popular admission essay on lincoln professional research paper editing sites online top case study ghostwriter site for college top speech ghostwriting website for mba professional speech writing website for university custom course work proofreading for hire for college professional dissertation methodology editor for hire for masters popular rhetorical analysis essay proofreading service au critical essay ghostwriting for hire uk help with my esl custom essay on usa popular dissertation hypothesis ghostwriter for hire for mba best thesis ghostwriters service for mba pros of racial profiling custom critical analysis essay ghostwriting site usa pay to get phd essay on usa cheap cv ghostwriters for hire usa esl research proposal ghostwriting service gb write me esl book review esl best essay proofreading site help me write shakespeare studies cover letter help me write best critical essay on presidential elections professional cv ghostwriting sites for mba write my best bibliography online best masters thesis advice top analysis essay writers website for university website review essay best analysis essay writer site for university pay for my professional creative essay on presidential elections custom content writing sites au best reflective essay ghostwriting for hire usa professional curriculum vitae editor website au esl dissertation chapter writer websites usa custom phd biography topic cheap phd curriculum vitae advice popular book review ghostwriter service for university write top business plan online blog ghostwriters sites online esl university assignment sample custom essays ghostwriters for hire for school custom essay ghostwriters service top persuasive essay ghostwriting service for school custom reflective essay editor websites ca top academic essay writers service for phd best writer site au professional dissertation proposal writers service for college top admission essay editor for hire for school best dissertation results editor service for mba popular rhetorical analysis essay editor site au cheap phd essay ghostwriters website for university custom scholarship essay writing service usa blog post ghostwriters service au best phd essay ghostwriter website for school cheap biography writers website us custom argumentative essay proofreading websites for mba cheap mba business plan topics help writing life science thesis write medicine letter popular academic essay ghostwriting services au pay to do popular phd essay on presidential elections professional admission paper ghostwriting for hire gb professional content writers sites for school help me write best masters essay on lincoln esl course work ghostwriter site uk popular article ghostwriter for hire for phd custom critical essay ghostwriting sites uk scholarship editor website usa mom is best popular blog post ghostwriter sites au cheap academic essay writer service au cheap scholarship essay writers services au cheap dissertation writing for hire usa top thesis proposal ghostwriter service gb professional academic essay ghostwriting service for university dissertation abstract writing for hire gb popular content writer website au esl annotated bibliography writers website uk best critical analysis essay writers services for university custom dissertation results editing websites for masters love in la sparknotes top biography writing for hire uk best academic essay ghostwriters site for university top report ghostwriters service for college popular dissertation introduction writer services for masters professional best essay editor services us popular article review ghostwriters services for college custom personal essay on usa custom thesis proposal ghostwriters websites for mba top annotated bibliography proofreading service for university best critical analysis essay writing website ca popular dissertation abstract ghostwriting services us popular blog writing websites best school essay writing sites for masters pay to write art & architecture bibliography top research paper writers website for phd top literature review ghostwriter website gb cheap college movie review socialization and culture top letter editing site au top dissertation results writers sites for school popular cv editing site for university popular thesis writing websites au best creative essay ghostwriting service online cheap literature review editor site au custom college course work example help me write popular creative essay on hacking top academic essay ghostwriter service best critical thinking editor website us professional masters expository essay help best rhetorical analysis essay ghostwriting services uk popular expository essay writer service for school professional homework proofreading website for mba cheap article editing site for college popular school cv examples logic essay ghostwriting site professional mba dissertation proposal esl dissertation ghostwriter services for phd phd essay on history best dissertation methodology ghostwriter website au popular book review proofreading site for university popular course work ghostwriter website usa cheap creative essay ghostwriting for hire for phd esl creative writing writing for hire for school elvis presleys wife cheap dissertation hypothesis ghostwriting for hire for university what are western religions top blog ghostwriting sites ca help writing best analysis essay on hillary cheap dissertation proposal writing for hire au help with custom admission paper buy shakespeare studies thesis top critical essay editor service online esl expository essay writers site custom personal essay editor service for school popular blog ghostwriter sites au type my professional critical essay on shakespeare custom persuasive essay writing sites us esl paper editing site us pay to write health speech cheap essays writers website usa esl university essay ghostwriting website for phd cheap best essay writer services gb cheap scholarship essay proofreading website for phd help with best literature review online custom creative writing writers service au professional annotated bibliography ghostwriter services ca dissertation chapter ghostwriter sites us professional dissertation proposal writer website for mba literature review writers services ca order classic english literature resume persuasive editor services us popular best essay writing for hire for university rhetorical analysis ghostwriter websites au top curriculum vitae proofreading service gb professional research proposal proofreading site for masters popular business plan ghostwriting site for college custom university essay ghostwriting website for mba type my women and gender studies creative writing best mba essay ghostwriters for hire us cheap phd essay ghostwriter service best dissertation abstract ghostwriting website for phd esl speech proofreading sites for school custom argumentative essay editing site for mba custom school letter assistance best dissertation conclusion editor websites for phd do my custom masters essay on shakespeare esl curriculum vitae editor service custom academic essay writing for hire au type my custom academic essay on usa professional critical thinking ghostwriters site for masters esl book review writers websites au professional creative essay ghostwriter websites for university pay to write investments admission essay physics writing websites top critical thinking writers sites ca case study ghostwriter site ca professional presentation ghostwriter site for school cheap argumentative essay editor websites for school popular course work ghostwriters website uk top course work ghostwriter service for school help writing statistics dissertation conclusion popular personal essay writing services online life science essay writers for hire custom thesis proposal writer services uk esl bibliography ghostwriting service cheap university business plan topic best dissertation results editor service ca cheap curriculum vitae editor sites for school best biography editor website au professional argumentative essay writing service top book review writer websites for school popular dissertation ghostwriter site usa custom phd essay ghostwriter website top college essay ghostwriter site online esl blog ghostwriter websites online best best essay writers services us buy logic cover letter professional scholarship essay ghostwriting for hire for university help writing film studies cv esl proofreading website au pay for tourism bibliography top annotated bibliography writing website uk business analysis paper popular rhetorical analysis essay writing websites for phd an image of africa meaning of true friendship cheap cover letter ghostwriters services popular dissertation ghostwriters site usa write me ancient civilizations movie review order world literature book review professional admission essay proofreading website uk popular course work ghostwriter sites for university biography proofreading for hire online ahab in moby dick nature vs nurture research paper custom college dissertation ideas cheap school annotated bibliography advice best course work ghostwriter for hire uk esl article review editing for hire for mba custom definition essay writing site for phd pay to write anthropology article review professional phd dissertation sample cheap book review writer websites custom dissertation results ghostwriter for hire us pay to write communication book review popular article ghostwriter sites online esl speech proofreading website for phd cheap essays writers services au professional masters essay ghostwriting service for phd top business plan writing services for university best curriculum vitae writing website au professional university admission essay advice popular expository essay ghostwriter websites pay to get theater studies dissertation results popular custom essay ghostwriter website for masters custom application letter editing websites for college symbols in the metamorphosis professional thesis proposal ghostwriter service for school popular mba term paper topic breathing underwater sparknotes great depression essay introduction earth science ghostwriters sites cheap personal statement writing services for college custom speech ghostwriting website us dissertation writing services gb popular home work ghostwriters services usa custom descriptive essay writing websites usa professional blog post writers sites for mba popular letter proofreading services for college pay to write custom academic essay on usa best blog editing sites for phd esl admission paper proofreading services uk professional dissertation chapter ghostwriting websites for masters cheap university essay ghostwriter site au custom papers editing site for mba top university essay proofreading website ca help with top dissertation conclusion esl dissertation conclusion editor site au popular blog post writer site for masters esl scholarship essay writers services custom business plan ghostwriter website online esl essays proofreading for hire for mba professional thesis statement writer site uk best article ghostwriters sites ca united methodist communion order earth science admission essay professional personal statement proofreading service for school custom problem solving writers services for school professional homework ghostwriter website ca college ghostwriting websites au logic writing site popular best essay editor services au cheap bibliography ghostwriter for hire uk best report editor site au cheap thesis proofreading site gb professional thesis statement ghostwriter services gb my personal philosophy professional report editing service for university custom ghostwriters website for masters popular scholarship essay proofreading sites usa help me write top blog online top thesis statement writer websites for university cheap assignment ghostwriter services top research paper proofreading site for college cheap research proposal ghostwriter websites online best academic essay proofreading services au pay to get english dissertation hypothesis cheap dissertation abstract ghostwriters sites professional phd resume assistance help with music thesis proposal buy human resource management admission paper professional blog post writer for hire ca best college curriculum vitae samples esl paper ghostwriter service best best essay writing for hire for phd write me best analysis essay online cheap application letter writer sites for mba custom rhetorical analysis essay editing site for college professional problem solving proofreading service for phd esl blog post editing site au reflective editing sites usa esl dissertation conclusion proofreading sites au help with my music home work communication writers services professional phd dissertation hypothesis advice help with my top masters essay on hacking best term paper ghostwriter service for school cheap curriculum vitae editing sites for mba pay to do popular school essay on trump help me write esl university essay on brexit custom research paper writing website gb esl academic essay proofreading for hire top argumentative essay writing services usa popular argumentative essay proofreading for hire for university pay to do history annotated bibliography