• Skip to main content
  • Skip to after header navigation
  • Skip to site footer

Shaping Software

Enduring Ideas in the Realm of Software

  • About
  • Topics
  • Best Software Books
  • Archives
  • JD Meier.com

Performance Hot Spots for Organizing Performance Knowledge

Apr 13, 2009 by JD

Performance Hot Spots

“The key to improving application performance is to identify and focus on the hot spots that have the greatest impact on performance.” — Bruce Dawson

The Performance Hot Spots framework is a powerful tool that can help your organization better manage and advance software performance.

As a business leader, you know that performance is critical for your organization. However, it can be overwhelming to navigate the vast amount of information out there on performance principles, patterns, and practices.

That’s where the Performance Hot Spots framework comes in. I created the Performance Hot Spots framework a lens for performance that can help you find, organize, and share software performance knowledge in a more actionable way.

I had worked on several large knowledge bases and knowledge management systems at Microsoft, so I wanted to apply that knowledge to advance the software performance space.  I figured the best way to advance performance would be to create a set of categories for knowledge, similar to how the Project Management Body of Knowledge created Knowledge Categories to advance project management.

And so the Performance Hot Spots framework was born.

The Power of Performance Hot Spots

From a software performance standpoint, “hot spots” are an area of intense activity.  They’re hot spots because they’re frequently executed code paths with some sort of friction or bottleneck.

They represent potential optimization paths for improving the performance of your code or design.  You find the hot spots by measuring and analyzing the system.

Stepping back, we can use “Hot Spots” more loosely.  We can use them to gather, organize, and share principles, patterns, and practices for performance.

You can think of Performance Hot Spots as a map or frame.

Remember that even with a map, you still need to set goals and measure.  You need to know what good looks like and you need to know when you’re done.

The benefit of a map or frame is that you have a set of potential areas to help you focus your exploration as well as find and share knowledge.

Visual Overview of Software Performance Hot Spots

Here is a simple whiteboard overview of the Software Performance Hot Spots Framework:

PerformanceHotSpots

Benefits of Performance Hot Spots 

The Performance Hot Spots framework can help businesses and organizations to optimize their applications, deliver better user experiences, and increase their bottom line.

The Performance Hot Spots framework provides several benefits for tackling performance issues:

  1. Firstly, it chunk up the performance space, making it easier to navigate and understand the different components that affect performance.
  2. Secondly, by creating more meaningful filters, it helps to identify the areas that require immediate attention, reducing the risk of getting lost in the vast sea of performance-related information.
  3. Thirdly, it provides a living map to help you navigate through performance issues and solutions, keeping you up-to-date with the latest trends and strategies.
  4. Fourthly, Performance Hot Spots can guide your inspections, whether it’s in the performance design, code, or deployment stages, helping you to identify potential performance bottlenecks early on.
  5. Lastly, the framework helps to divide and conquer performance issues, making it easier to prioritize and address the most critical performance hot spots first.

Performance Hot Spot Categories

Performance is a crucial aspect of software development, especially as user expectations for speed and efficiency continue to rise.

To address performance issues effectively, it’s essential to have a clear understanding of the key areas that impact performance.

This is where Performance Hot Spot Categories come in, providing a structured framework for identifying and addressing performance-related concerns.

The Performance Hot Spots at the application level are:

  • Caching
  • Communication
  • Concurrency
  • Coupling / Cohesion
  • Data Access
  • Data Structures / Algorithms
  • Exception Management
  • Resource Management
  • State Management

Performance Hot Spots for Applications

Here are some common performance issues organized by Performance Hot Spots:

Category Issues
Caching
  • Round trips to data store for every single user request, increased load on the data store
  • Increased client response time, reduced throughput, and increased server resource utilization
  • Increased memory consumption, resulting in reduced performance, cache misses, and increased data store access
Communication
  • Multiple round trips to perform a single operation
  • Increased serialization overhead and network latency
  • Cross-boundary overhead: security checks, thread switches, and serialization
Concurrency
  • Stalls the application, and reduces response time and throughput
  • Stalls the application, and leads to queued requests and timeouts
  • Additional processor and memory overhead due to context switching and thread management overhead
  • Increased contention and reduced concurrency
  • Poor choice of isolation levels results in contention, long wait time, timeouts, and deadlocks
Coupling and Cohesion
  • Mixing functionally different logic (such as presentation and business) without clear, logical partitioning limits scalability options
  • Chatty interfaces lead to multiple round trips
Data Access
  • Increased database server processing
  • Reduced throughput
  • Increased network bandwidth consumption
  • Delayed response times Increased client and server load
  • Increased garbage collection overhead
  • Increased processing effort required
  • Inefficient queries or fetching all the data to display a portion is an unnecessary cost, in terms of server resources and performance
  • Unnecessary load on the database server
  • Failure to meet performance objectives and exceeding budget allocations
Data Structures / Algorithms
  • Reduced efficiency; overly complex code
  • Reduced efficiency; overly complex code
  • Passing value type to reference type causing boxing and unboxing
  • Complete scan of all the content in the data structure, resulting in slow performance
  • Undetected bottlenecks due to inefficient code.
Exception Management
  • Round trips to servers and expensive calls
  • Expensive compared to returning enumeration or Boolean values
  • Increased inefficiency
  • Adds to performance overhead and can conceal information unnecessarily
Resource Management
  • Can result in creating many instances of the resources along with its connection overhead
  • Increase in overhead cost affects the response time of the application; Not releasing (or delaying the release of) shared resources, such as connections, leads to resource drain on the server and limits scalability
  • Retrieving large amounts of data from the resource increases the time taken to service the request, as well as network latency
  • Increase in time spent on the server also affects response time as concurrent users increase
  • Leads to resource shortages and increased memory consumption; both of these affect scalability
  • Large numbers of clients can cause resource starvation and overload the server.
State Management
  • Holding server resources and can cause server affinity, which reduces scalability options
  • Limits scalability due to server affinity
  • Increased server resource utilization Limited server scalability
  • In-process and local state stored on the Web server limits the ability of the Web application to run in a Web farm
  • Large amounts of state maintained in memory also create memory pressure on the server
  • Increased server resource utilization, and increased time for state storage and retrieval
  • Inappropriate timeout values result in sessions consuming and holding server resources for longer than necessary.

Examples / Case Studies of the Performance Hot Spots in Action

Using Performance Hot Spots can help organizations achieve better performance results by providing a structured approach to identifying and addressing performance issues.

The evidence is clear, as seen in examples such as the Improving .NET Application Performance and Scalability book and the Performance Testing Guidance for Web Applications, which were framed using Performance Hot Spots.

The Performance Hot Spots approach is also used in Performance Engineering, Performance Design Guidelines, Performance Inspections, Performance Checklists, and Performance Modeling, making it a versatile and effective tool for optimizing performance.

Using Performance Hot Spots produces results.  Here are examples of Performance Hot spots in Action:

  • Performance Guides / Books.  We used Performance Hot Spots to help frame the patterns & practices book Improving .NET Application Performance and Scalability and Performance Testing Guidance for Web Applications.
  • Performance Engineering.  The heart of our patterns & practices Performance Engineering is Performance Hot Spot driven.  We focus on the high ROI activities and each activity uses Performance Hot Spots to focus results.
  • Performance and Scalability Frame.   We use the Performance Hot Spots to create the Performance and Scalability Frame.
  • Performance Design Guidelines.  We used Performance Hot Spots to create the Performance Frame, which is an organizing backdrop for Performance Design Guidelines.
  • Performance Inspections.  We used Performance Hot Spots to drive our patterns & practices Performance Design Inspection prescriptive guidance
  • Performance Checklists.  We organize our Performance Design Checklist using the Performance Hot Spots.
  • Performance Modeling.  We use the Performance Hot Spots as a way to guide and structure our patterns & practices Performance Modeling approach.

Questions for Reflection

Performance Hot Spots can offer numerous benefits for your organization. Here are some questions to help you leverage them to improve your performance results:

  • How can you use Performance Hot Spots to identify and address performance issues in your applications?
  • How can you organize your performance knowledge using Performance Hot Spots to make it more actionable?
  • How can you use Performance Hot Spots to share best practices and anti-patterns among your team members?
  • How can you improve your performance checklists using Performance Hot Spots to focus on key areas?
  • How can you use Performance Hot Spots to optimize your performance inspections, such as code reviews and deployment inspections?

You Might Also Like

Performance Threats and Countermeasures Framework
Security Hot Spots for Organizing Security Knowledge

Category: Architecture, Performance

About JD

Previous Post:Software Security Hot SpotsSecurity Hot Spots for Organizing Security Knowledge
Next Post:Customer-Connected Engineering Slides (PDF Download)Customer-Connected Engineering Slides

Reader Interactions

Comments

  1. Mike Dunlavey

    May 25, 2009 at 3:24 pm

    This blog seems to be very high-level, and I would suggest there is room for a low-level perspective as well, especially for the problem of finding and removing performance problems in specific software.

    First, I would suggest not using terms unless they have a clear definition, not hand-waving. For “bottleneck” I have never heard a clear definition. For “hotspot” I think there is a clear definition, though it less important than commonly regarded.

    I have a lot of practical experience doing performance tuning of software, and depending on how bad the problems are, often achieve speedup factors between 10x and 200x.

    Rather than the terms “hotspot” or “bottleneck”, I prefer the term “slowness bug”, or “slug” for short, although it sounds more professional to say “performance problem”. In any case, they are bugs. What they do wrong is take too much time. They are like bugs in that they must be found, and fixed. The fix always consists of changing some lines of code, and the finding consists of finding those lines.

    Here’s my method, which is tried-and-true, even though it flies in the face of accepted wisdom. (Accepted wisdom has gone without being questioned for far too long.) Your first reaction to this method may be that it is naively simple, but in fact that is why it is so effective.

    1. While the program is being slow, I manually interrupt it at a random time, and make a record of the call stack. The call stack is a list of instruction addresses, all of which are function call instructions except the “bottom” one. (The distinction between a function, and a function CALL, cannot be stressed enough. The latter is packed with useful information.)

    2. Then I take another sample of the call stack in the same way. Then I compare the two, looking for any instruction address that appears on both of them.

    Now, here is the key point: any instruction appearing on both samples, _if it can be avoided_ will save a large fraction of execution time.

    3. In the typical case that I do _not_ find a repeated instruction that can be avoided, I repeat step 2, as many times as necessary, building up a collection of call stack samples, and each time looking for instruction addresses that appear on multiple samples, _and_ that can be avoided. As soon as I find one, I fix it, rebuild, and re-run. The program is now faster, guaranteed. (If I go past 20 samples without finding something to fix, I’ve hit the point of diminishing returns, and stop.)

    4. I repeat this overall cycle 1,2,3, each time finding and removing a slowness bug until it is difficult to get samples because the program runs too fast. Then I insert a temporary outer loop to make it take longer, and keep at it, until I can no longer find things that I can fix. Then I remove the outer loop, and the program is much, much faster than it was to begin with, depending on how many problems it had to begin with.

    There are problems for which the call stack alone is not enough information to identify the problem, and for those I augment this with other techniques, but that’s too complicated to explain here.

    When I try to teach this method I run into standard objections, none of which are shared by the small group of people who actually use it.

    Objection 1: It is just a sampling method, with a very small sample size, so it could give spurious results.

    Answer: Yes it is sampling, but the key difference is in _what_ it samples. Retaining the program counter alone does not tell you much, except in very small programs. Retaining only the identities of the functions in the call stack, without the line-number or call site address information, is discarding key information.

    As far as the results being spurious, consider the smallest useful sample size, two samples of the call stack. If a call instruction (or any instruction) appears on the first sample, that does not tell you much. However, if it appears on both samples, that is _highly unlikely_ to happen by chance. Some instructions will obviously be on both samples, like “call _main()”, but those cannot be avoided. You are looking for instructions that _can_ be avoided.

    If you change the code so that such an avoidable function call is performed less often or not at all, the amount of time saved is approximately the percentage of samples that contained that function call. If this is not understood, please think about it until it is.

    Objection 2: It will stop someplace random, and will miss the _real_ problem.

    Answer: This comes from having a preconception of what the _real_ problem is. You don’t have to think about this very hard to understand that a) if it finds a problem, that problem is _real_, and 2) if a problem is _real_, it will find it.

    Objection 3: It will not work if there is recursion.

    Answer: Call-graph profilers are confused by recursion, but this method is not. Remember the method is to locate avoidable call instructions that appear on multiple samples, and the amount of time that could be saved is approximated by the percentage of samples containing such an instruction. This remains true even if any sample contains duplicate instructions. If this is not understood, please think about it until it is.

    Objection 4: What about multi-thread programs?

    Answer: Each thread by itself can be tuned. Then, there can be cross-thread dependencies, where one thread waits for another, or for a message from a remote process. In these cases, stack sample information is not sufficient, and more information needs to be captured. I will not go into this here, for lack of space, but it is quite do-able.

    Finally I offered to define the term “hot-spot”. It is a small set of addresses found at the bottom of the call stack on a significant percentage of samples. A hot-spot can be a performance bug, but usually it is not. The most common type of performance bug is an avoidable function call somewhere mid-stack. So if it were up to me I would downplay the emphasis on hot-spots in discussing performance tuning.

    The term “bottleneck” implies a constriction of some sort, like the narrow end of a funnel. Maybe in a communication network such a thing makes sense. In a single thread of execution it makes no sense, because the time taken is more like an automobile driver taking a cross-country route. If he is efficient, he goes straight from A to B by the most direct routine. If he is not, he makes lots of side trips, gets lost, dallies about, and takes his sweet time. Does it make sense to call that a “bottleneck”?

    Thanks for the chance to air this point of view.

  2. JD

    May 26, 2009 at 4:15 am

    @ Mike

    Good stuff.

    For lower level stuff, I keep that on MSDN (for example, http://msdn.microsoft.com/en-us/library/ms998530.aspx). This is stepping back to look at a simpler map.

    Hot Spots in this case isn’t a performance term. It’s the concept of framing out common areas.

    I think it’s helpful in software to think in terms of app level and code level. The Hot Spots examples I gave are more at the app level, though they apply to lower level as well (http://msdn.microsoft.com/en-us/library/ms998547.aspx.)

Trackbacks

  1. Interesting Finds: 2009 04.10~04.15 - gOODiDEA.NET says:
    Apr 15, 2009 at 12:40 am

    […] Performance Hot Spots […]

  2. This Week on C9: Office 2010, PowerShell for OpenXML, and would you help a robot - ComponentGear.com Feed - ComponentGear.com says:
    Apr 18, 2009 at 12:09 am

    […] you know if you are spamming an email alias or emailing someone out of the office – J.D. Meier – Performance Hot Spots, via Alvin Ashcraft – Arian Kulp – Coding4Fun Extensible System Tray Framework – Larry Larsen – […]

  3. This Week on C9: Office 2010, PowerShell for OpenXML, and would you help a robot « Bloggin says:
    Apr 23, 2009 at 5:33 pm

    […] you know if you are spamming an email alias or emailing someone out of the office – J.D. Meier – Performance Hot Spots, via Alvin Ashcraft – Arian Kulp – Coding4Fun Extensible System Tray Framework – Larry Larsen – […]

Sidebar

Recent Posts

  • What is ChatGPT?
  • Agile Performance Engineering
  • What is Cybersecurity?
  • Software Security Threats: A Comprehensive Guide
  • What is Software Security?

Popular Posts

Best Software Books of All Time
Best Practices for Project Management
Best Practices for Software Development
Customer-Connected Engineering
How To Frame Problems Better
How To Pitch Business Ideas Better
How To Structure Vision Scope Presentations
Intro to Lean Software Development
Lean Principles for Software Development
The Enterprise of the Future