“The key to improving application performance is to identify and focus on the hot spots that have the greatest impact on performance.” — Bruce Dawson
The Performance Hot Spots framework is a powerful tool that can help your organization better manage and advance software performance.
As a business leader, you know that performance is critical for your organization. However, it can be overwhelming to navigate the vast amount of information out there on performance principles, patterns, and practices.
That’s where the Performance Hot Spots framework comes in. I created the Performance Hot Spots framework a lens for performance that can help you find, organize, and share software performance knowledge in a more actionable way.
I had worked on several large knowledge bases and knowledge management systems at Microsoft, so I wanted to apply that knowledge to advance the software performance space. I figured the best way to advance performance would be to create a set of categories for knowledge, similar to how the Project Management Body of Knowledge created Knowledge Categories to advance project management.
And so the Performance Hot Spots framework was born.
The Power of Performance Hot Spots
From a software performance standpoint, “hot spots” are an area of intense activity. They’re hot spots because they’re frequently executed code paths with some sort of friction or bottleneck.
They represent potential optimization paths for improving the performance of your code or design. You find the hot spots by measuring and analyzing the system.
Stepping back, we can use “Hot Spots” more loosely. We can use them to gather, organize, and share principles, patterns, and practices for performance.
You can think of Performance Hot Spots as a map or frame.
Remember that even with a map, you still need to set goals and measure. You need to know what good looks like and you need to know when you’re done.
The benefit of a map or frame is that you have a set of potential areas to help you focus your exploration as well as find and share knowledge.
Visual Overview of Software Performance Hot Spots
Here is a simple whiteboard overview of the Software Performance Hot Spots Framework:
Benefits of Performance Hot Spots
The Performance Hot Spots framework can help businesses and organizations to optimize their applications, deliver better user experiences, and increase their bottom line.
The Performance Hot Spots framework provides several benefits for tackling performance issues:
- Firstly, it chunk up the performance space, making it easier to navigate and understand the different components that affect performance.
- Secondly, by creating more meaningful filters, it helps to identify the areas that require immediate attention, reducing the risk of getting lost in the vast sea of performance-related information.
- Thirdly, it provides a living map to help you navigate through performance issues and solutions, keeping you up-to-date with the latest trends and strategies.
- Fourthly, Performance Hot Spots can guide your inspections, whether it’s in the performance design, code, or deployment stages, helping you to identify potential performance bottlenecks early on.
- Lastly, the framework helps to divide and conquer performance issues, making it easier to prioritize and address the most critical performance hot spots first.
Performance Hot Spot Categories
Performance is a crucial aspect of software development, especially as user expectations for speed and efficiency continue to rise.
To address performance issues effectively, it’s essential to have a clear understanding of the key areas that impact performance.
This is where Performance Hot Spot Categories come in, providing a structured framework for identifying and addressing performance-related concerns.
The Performance Hot Spots at the application level are:
- Caching
- Communication
- Concurrency
- Coupling / Cohesion
- Data Access
- Data Structures / Algorithms
- Exception Management
- Resource Management
- State Management
Performance Hot Spots for Applications
Here are some common performance issues organized by Performance Hot Spots:
| Category | Issues |
|---|---|
| Caching |
|
| Communication |
|
| Concurrency |
|
| Coupling and Cohesion |
|
| Data Access |
|
| Data Structures / Algorithms |
|
| Exception Management |
|
| Resource Management |
|
| State Management |
|
Examples / Case Studies of the Performance Hot Spots in Action
Using Performance Hot Spots can help organizations achieve better performance results by providing a structured approach to identifying and addressing performance issues.
The evidence is clear, as seen in examples such as the Improving .NET Application Performance and Scalability book and the Performance Testing Guidance for Web Applications, which were framed using Performance Hot Spots.
The Performance Hot Spots approach is also used in Performance Engineering, Performance Design Guidelines, Performance Inspections, Performance Checklists, and Performance Modeling, making it a versatile and effective tool for optimizing performance.
Using Performance Hot Spots produces results. Here are examples of Performance Hot spots in Action:
- Performance Guides / Books. We used Performance Hot Spots to help frame the patterns & practices book Improving .NET Application Performance and Scalability and Performance Testing Guidance for Web Applications.
- Performance Engineering. The heart of our patterns & practices Performance Engineering is Performance Hot Spot driven. We focus on the high ROI activities and each activity uses Performance Hot Spots to focus results.
- Performance and Scalability Frame. We use the Performance Hot Spots to create the Performance and Scalability Frame.
- Performance Design Guidelines. We used Performance Hot Spots to create the Performance Frame, which is an organizing backdrop for Performance Design Guidelines.
- Performance Inspections. We used Performance Hot Spots to drive our patterns & practices Performance Design Inspection prescriptive guidance
- Performance Checklists. We organize our Performance Design Checklist using the Performance Hot Spots.
- Performance Modeling. We use the Performance Hot Spots as a way to guide and structure our patterns & practices Performance Modeling approach.
Questions for Reflection
Performance Hot Spots can offer numerous benefits for your organization. Here are some questions to help you leverage them to improve your performance results:
- How can you use Performance Hot Spots to identify and address performance issues in your applications?
- How can you organize your performance knowledge using Performance Hot Spots to make it more actionable?
- How can you use Performance Hot Spots to share best practices and anti-patterns among your team members?
- How can you improve your performance checklists using Performance Hot Spots to focus on key areas?
- How can you use Performance Hot Spots to optimize your performance inspections, such as code reviews and deployment inspections?
You Might Also Like
Performance Threats and Countermeasures Framework
Security Hot Spots for Organizing Security Knowledge
Security Hot Spots for Organizing Security Knowledge
This blog seems to be very high-level, and I would suggest there is room for a low-level perspective as well, especially for the problem of finding and removing performance problems in specific software.
First, I would suggest not using terms unless they have a clear definition, not hand-waving. For “bottleneck” I have never heard a clear definition. For “hotspot” I think there is a clear definition, though it less important than commonly regarded.
I have a lot of practical experience doing performance tuning of software, and depending on how bad the problems are, often achieve speedup factors between 10x and 200x.
Rather than the terms “hotspot” or “bottleneck”, I prefer the term “slowness bug”, or “slug” for short, although it sounds more professional to say “performance problem”. In any case, they are bugs. What they do wrong is take too much time. They are like bugs in that they must be found, and fixed. The fix always consists of changing some lines of code, and the finding consists of finding those lines.
Here’s my method, which is tried-and-true, even though it flies in the face of accepted wisdom. (Accepted wisdom has gone without being questioned for far too long.) Your first reaction to this method may be that it is naively simple, but in fact that is why it is so effective.
1. While the program is being slow, I manually interrupt it at a random time, and make a record of the call stack. The call stack is a list of instruction addresses, all of which are function call instructions except the “bottom” one. (The distinction between a function, and a function CALL, cannot be stressed enough. The latter is packed with useful information.)
2. Then I take another sample of the call stack in the same way. Then I compare the two, looking for any instruction address that appears on both of them.
Now, here is the key point: any instruction appearing on both samples, _if it can be avoided_ will save a large fraction of execution time.
3. In the typical case that I do _not_ find a repeated instruction that can be avoided, I repeat step 2, as many times as necessary, building up a collection of call stack samples, and each time looking for instruction addresses that appear on multiple samples, _and_ that can be avoided. As soon as I find one, I fix it, rebuild, and re-run. The program is now faster, guaranteed. (If I go past 20 samples without finding something to fix, I’ve hit the point of diminishing returns, and stop.)
4. I repeat this overall cycle 1,2,3, each time finding and removing a slowness bug until it is difficult to get samples because the program runs too fast. Then I insert a temporary outer loop to make it take longer, and keep at it, until I can no longer find things that I can fix. Then I remove the outer loop, and the program is much, much faster than it was to begin with, depending on how many problems it had to begin with.
There are problems for which the call stack alone is not enough information to identify the problem, and for those I augment this with other techniques, but that’s too complicated to explain here.
When I try to teach this method I run into standard objections, none of which are shared by the small group of people who actually use it.
Objection 1: It is just a sampling method, with a very small sample size, so it could give spurious results.
Answer: Yes it is sampling, but the key difference is in _what_ it samples. Retaining the program counter alone does not tell you much, except in very small programs. Retaining only the identities of the functions in the call stack, without the line-number or call site address information, is discarding key information.
As far as the results being spurious, consider the smallest useful sample size, two samples of the call stack. If a call instruction (or any instruction) appears on the first sample, that does not tell you much. However, if it appears on both samples, that is _highly unlikely_ to happen by chance. Some instructions will obviously be on both samples, like “call _main()”, but those cannot be avoided. You are looking for instructions that _can_ be avoided.
If you change the code so that such an avoidable function call is performed less often or not at all, the amount of time saved is approximately the percentage of samples that contained that function call. If this is not understood, please think about it until it is.
Objection 2: It will stop someplace random, and will miss the _real_ problem.
Answer: This comes from having a preconception of what the _real_ problem is. You don’t have to think about this very hard to understand that a) if it finds a problem, that problem is _real_, and 2) if a problem is _real_, it will find it.
Objection 3: It will not work if there is recursion.
Answer: Call-graph profilers are confused by recursion, but this method is not. Remember the method is to locate avoidable call instructions that appear on multiple samples, and the amount of time that could be saved is approximated by the percentage of samples containing such an instruction. This remains true even if any sample contains duplicate instructions. If this is not understood, please think about it until it is.
Objection 4: What about multi-thread programs?
Answer: Each thread by itself can be tuned. Then, there can be cross-thread dependencies, where one thread waits for another, or for a message from a remote process. In these cases, stack sample information is not sufficient, and more information needs to be captured. I will not go into this here, for lack of space, but it is quite do-able.
Finally I offered to define the term “hot-spot”. It is a small set of addresses found at the bottom of the call stack on a significant percentage of samples. A hot-spot can be a performance bug, but usually it is not. The most common type of performance bug is an avoidable function call somewhere mid-stack. So if it were up to me I would downplay the emphasis on hot-spots in discussing performance tuning.
The term “bottleneck” implies a constriction of some sort, like the narrow end of a funnel. Maybe in a communication network such a thing makes sense. In a single thread of execution it makes no sense, because the time taken is more like an automobile driver taking a cross-country route. If he is efficient, he goes straight from A to B by the most direct routine. If he is not, he makes lots of side trips, gets lost, dallies about, and takes his sweet time. Does it make sense to call that a “bottleneck”?
Thanks for the chance to air this point of view.
@ Mike
Good stuff.
For lower level stuff, I keep that on MSDN (for example, http://msdn.microsoft.com/en-us/library/ms998530.aspx). This is stepping back to look at a simpler map.
Hot Spots in this case isn’t a performance term. It’s the concept of framing out common areas.
I think it’s helpful in software to think in terms of app level and code level. The Hot Spots examples I gave are more at the app level, though they apply to lower level as well (http://msdn.microsoft.com/en-us/library/ms998547.aspx.)