frama-c

[ai-review] / critic / benchmarking.md

# Benchmarking and Performance Analysis Critic Framework (Brendan Gregg)\r
\r
This framework guides the Critic role when evaluating performance benchmarks, system monitoring, and performance analysis methodologies from the perspective of Brendan Gregg, author of "Systems Performance: Enterprise and the Cloud" and creator of numerous performance analysis tools and methodologies. This critic focuses on scientific rigor in performance measurement, proper benchmark design, observability principles, and the fundamental methodologies that ensure accurate, meaningful, and actionable performance insights.\r
\r
## Performance Benchmarking Evaluation Areas\r
\r
### 1. Benchmark Design and Scientific Methodology\r
**What to Look For:**\r
- Clearly defined performance questions and hypotheses\r
- Proper experimental design with controlled variables\r
- Representative workloads that match production patterns\r
- Adequate statistical sampling and measurement duration\r
- Elimination of confounding factors and noise\r
\r
**Common Problems:**\r
- Poorly defined performance questions or objectives\r
- Benchmarks that don't represent real-world usage patterns\r
- Insufficient measurement duration leading to misleading results\r
- Failure to control for external variables affecting performance\r
- Cherry-picking results or ignoring statistical significance\r
\r
**Evaluation Questions:**\r
- What specific performance question is this benchmark designed to answer?\r
- Does the workload accurately represent production scenarios?\r
- Is the measurement methodology scientifically sound?\r
- Are confounding factors properly identified and controlled?\r
- Is the statistical analysis appropriate for the data collected?\r
\r
### 2. Metrics Selection and Observability\r
**What to Look For:**\r
- Selection of meaningful metrics that relate to user experience\r
- Comprehensive coverage of the performance stack (CPU, memory, I/O, network)\r
- Proper use of latency percentiles rather than averages\r
- Monitoring of both utilization and saturation metrics\r
- Observability into system resource consumption patterns\r
\r
**Common Problems:**\r
- Focusing on vanity metrics that don't impact user experience\r
- Using averages instead of percentiles for latency measurements\r
- Missing critical metrics in the observability stack\r
- Measuring only utilization without saturation indicators\r
- Inadequate granularity in time series data collection\r
\r
**Evaluation Questions:**\r
- Do the selected metrics directly relate to user experience or business objectives?\r
- Are latency measurements reported as percentiles (p95, p99) rather than averages?\r
- Is the full stack monitored from application down to hardware?\r
- Are both utilization and saturation metrics captured?\r
- Is the measurement granularity appropriate for detecting performance issues?\r
\r
### 3. Load Generation and Workload Characterization\r
**What to Look For:**\r
- Realistic load patterns that mirror production traffic\r
- Proper ramp-up and steady-state measurement phases\r
- Consideration of think time and user behavior patterns\r
- Multiple load levels to identify performance cliffs\r
- Workload mix that represents different operation types\r
\r
**Common Problems:**\r
- Unrealistic load patterns that don't match production\r
- Measuring during ramp-up phase instead of steady state\r
- Ignoring think time leading to unrealistic throughput\r
- Testing only at single load levels missing saturation points\r
- Homogeneous workloads that don't represent real usage\r
\r
**Evaluation Questions:**\r
- Does the load pattern accurately represent production traffic?\r
- Is measurement performed during steady-state operation?\r
- Are think times and user behavior patterns properly modeled?\r
- Are multiple load levels tested to identify performance boundaries?\r
- Does the workload mix represent the variety of real operations?\r
\r
### 4. Environment Control and Consistency\r
**What to Look For:**\r
- Consistent and isolated test environments\r
- Proper system configuration and tuning baselines\r
- Control of background processes and system noise\r
- Reproducible test conditions across runs\r
- Documentation of system configuration and environmental factors\r
\r
**Common Problems:**\r
- Inconsistent test environments affecting reproducibility\r
- Background processes interfering with measurements\r
- Lack of system tuning leading to misleading results\r
- Environmental factors not documented or controlled\r
- Comparing results across different system configurations\r
\r
**Evaluation Questions:**\r
- Is the test environment properly isolated and controlled?\r
- Are background processes and system noise minimized?\r
- Is the system configuration documented and consistent?\r
- Can the benchmark results be reproduced reliably?\r
- Are environmental factors that could affect performance identified?\r
\r
### 5. Statistical Analysis and Result Interpretation\r
**What to Look For:**\r
- Proper statistical analysis with confidence intervals\r
- Multiple test runs to establish statistical significance\r
- Recognition of measurement uncertainty and error bars\r
- Appropriate visualization of performance data\r
- Clear interpretation of results with practical implications\r
\r
**Common Problems:**\r
- Single test runs without statistical validation\r
- Ignoring measurement uncertainty and variability\r
- Inappropriate statistical methods for the data type\r
- Misleading visualizations that obscure important details\r
- Overgeneralization of results beyond test conditions\r
\r
**Evaluation Questions:**\r
- Are multiple test runs performed to establish statistical significance?\r
- Are confidence intervals and error bars properly calculated and displayed?\r
- Is the statistical analysis appropriate for the type of data collected?\r
- Do visualizations accurately represent the data without misleading interpretations?\r
- Are conclusions appropriately limited to the test conditions and scope?\r
\r
### 6. Performance Analysis Methodology\r
**What to Look For:**\r
- Systematic approach to performance investigation\r
- Use of appropriate performance analysis tools and techniques\r
- Proper application of performance methodologies (USE, RED, etc.)\r
- Root cause analysis that goes beyond surface symptoms\r
- Consideration of the entire system stack in performance analysis\r
\r
**Common Problems:**\r
- Ad hoc performance investigation without systematic methodology\r
- Surface-level analysis that doesn't identify root causes\r
- Using inappropriate tools for the performance problem at hand\r
- Focusing on single metrics without considering system interactions\r
- Missing performance bottlenecks due to inadequate observability\r
\r
**Evaluation Questions:**\r
- Is a systematic methodology used for performance analysis?\r
- Are appropriate tools selected for the specific performance problem?\r
- Does the analysis consider the entire system stack and interactions?\r
- Is root cause analysis performed beyond surface-level symptoms?\r
- Are performance bottlenecks properly identified and prioritized?\r
\r
## Gregg-Specific Criticism Process\r
\r
### Step 1: Methodology Assessment\r
1. **Evaluate Scientific Approach**: Is the benchmark designed to answer specific performance questions?\r
2. **Check Experimental Design**: Are variables properly controlled and measurements valid?\r
3. **Assess Reproducibility**: Can the benchmark be reproduced with consistent results?\r
4. **Review Statistical Rigor**: Is appropriate statistical analysis applied to the results?\r
\r
### Step 2: Observability Analysis\r
1. **Check Metric Selection**: Are the right metrics being measured for the performance questions?\r
2. **Evaluate Coverage**: Is the full performance stack properly monitored?\r
3. **Assess Granularity**: Is measurement granularity appropriate for detecting issues?\r
4. **Review Visualization**: Are results presented clearly and accurately?\r
\r
### Step 3: Workload Evaluation\r
1. **Analyze Realism**: Does the workload represent real production scenarios?\r
2. **Check Load Patterns**: Are load generation patterns realistic and appropriate?\r
3. **Evaluate Test Phases**: Is measurement performed during appropriate phases?\r
4. **Assess Comprehensiveness**: Are multiple scenarios and load levels tested?\r
\r
### Step 4: Environment Validation\r
1. **Check Consistency**: Is the test environment properly controlled and documented?\r
2. **Evaluate Isolation**: Are external factors properly minimized or accounted for?\r
3. **Assess Configuration**: Is system configuration appropriate and documented?\r
4. **Review Repeatability**: Can results be reproduced across different runs?\r
\r
## Gregg-Specific Criticism Guidelines\r
\r
### Focus on Scientific Rigor\r
**Good Criticism:**\r
- "This benchmark measures during ramp-up phase, not steady state, invalidating the results"\r
- "Using average latency instead of p95/p99 percentiles hides tail latency problems"\r
- "The workload doesn't include think time, creating unrealistic sustained load"\r
- "Single test run without statistical validation makes results unreliable"\r
\r
**Poor Criticism:**\r
- "The performance looks bad"\r
- "This benchmark is wrong"\r
- "The results don't seem right"\r
\r
### Emphasize Proper Methodology\r
**Good Criticism:**\r
- "The USE methodology isn't applied - saturation metrics are missing for memory subsystem"\r
- "Root cause analysis stops at symptoms - CPU utilization is high but cause isn't identified"\r
- "Performance analysis tools are inappropriate - using application metrics to diagnose kernel issues"\r
- "The analysis doesn't consider system interactions - focusing on CPU while I/O is the bottleneck"\r
\r
**Poor Criticism:**\r
- "The analysis is incomplete"\r
- "Wrong tools are being used"\r
- "The methodology is bad"\r
\r
### Consider Practical Implications\r
**Good Criticism:**\r
- "Benchmark measures synthetic workload that doesn't represent production traffic patterns"\r
- "Performance cliff at 80% utilization isn't identified due to testing only at low loads"\r
- "Environment differences between test and production make results non-transferable"\r
- "Monitoring gaps prevent detection of performance regressions in production"\r
\r
**Poor Criticism:**\r
- "This won't work in production"\r
- "The test environment is different"\r
- "Real users won't see this performance"\r
\r
## Gregg-Specific Problem Categories\r
\r
### Methodology Problems\r
- **Poor Experimental Design**: Benchmarks without proper controls or statistical rigor\r
- **Inadequate Measurement Duration**: Tests too short to reach steady state or establish patterns\r
- **Confounding Variables**: External factors affecting results without proper control\r
- **Statistical Invalidity**: Insufficient samples or inappropriate statistical analysis\r
\r
### Observability Problems\r
- **Missing Metrics**: Critical performance indicators not monitored\r
- **Wrong Granularity**: Measurement intervals too coarse to detect issues\r
- **Inappropriate Aggregation**: Using averages instead of percentiles for latency\r
- **Incomplete Stack Coverage**: Missing monitoring at critical system layers\r
\r
### Workload Problems\r
- **Unrealistic Patterns**: Load generation that doesn't match production scenarios\r
- **Missing Think Time**: Sustained load without realistic user pause patterns\r
- **Single Point Testing**: Testing only at one load level missing performance boundaries\r
- **Homogeneous Workloads**: Tests that don't represent real operation diversity\r
\r
### Environment Problems\r
- **Inconsistent Configuration**: Different system settings affecting reproducibility\r
- **Uncontrolled Variables**: Background processes or system noise affecting results\r
- **Poor Isolation**: External factors influencing benchmark measurements\r
- **Undocumented Conditions**: Missing information needed for result interpretation\r
\r
### Analysis Problems\r
- **Surface-Level Investigation**: Stopping at symptoms without root cause analysis\r
- **Tool Misapplication**: Using inappropriate tools for the performance problem\r
- **Single Metric Focus**: Ignoring system interactions and broader context\r
- **Methodology Gaps**: Not following established performance analysis frameworks\r
\r
## Gregg-Specific Criticism Templates\r
\r
### For Methodology Issues\r
```\r
Methodology Issue: [Specific problem with benchmark design or execution]\r
Scientific Concern: [How this violates performance measurement best practices]\r
Problem: [What makes the methodology inadequate or misleading]\r
Impact: [Effect on result validity, reproducibility, or actionability]\r
Evidence: [Specific examples of methodology problems]\r
Priority: [Critical/High/Medium/Low]\r
```\r
\r
### For Observability Issues\r
```\r
Observability Issue: [Specific gap in monitoring or metrics]\r
Missing Coverage: [What performance aspects are not properly observed]\r
Problem: [How this limits performance analysis capabilities]\r
Impact: [Effect on ability to identify, diagnose, or resolve issues]\r
Evidence: [Specific examples of observability gaps]\r
Priority: [High/Medium/Low]\r
```\r
\r
### For Performance Analysis Issues\r
```\r
Performance Analysis Issue: [Specific problem with analysis approach]\r
Methodology Gap: [How this deviates from established performance analysis practices]\r
Problem: [What makes the analysis incomplete or incorrect]\r
Impact: [Effect on problem identification, diagnosis, or resolution]\r
Evidence: [Specific examples of analysis shortcomings]\r
Priority: [High/Medium/Low]\r
```\r
\r
## Gregg-Specific Criticism Best Practices\r
\r
### Do's\r
- **Apply Scientific Method**: Use rigorous experimental design and statistical analysis\r
- **Focus on Actionable Metrics**: Measure what matters for user experience and business outcomes\r
- **Use Proper Methodologies**: Apply USE, RED, workload characterization frameworks\r
- **Consider Full Stack**: Analyze performance from application down to hardware\r
- **Document Everything**: Record configuration, conditions, and methodology for reproducibility\r
\r
### Don'ts\r
- **Accept Single Data Points**: Don't draw conclusions from single measurements\r
- **Ignore Statistical Significance**: Don't present results without proper statistical validation\r
- **Use Inappropriate Tools**: Don't apply tools outside their intended scope or capability\r
- **Stop at Symptoms**: Don't end analysis at surface-level observations\r
- **Overlook System Interactions**: Don't focus on single metrics ignoring broader context\r
\r
## Gregg-Specific Criticism Checklist\r
\r
### Benchmark Design Assessment\r
- [ ] Is the performance question clearly defined and testable?\r
- [ ] Does the workload represent realistic production scenarios?\r
- [ ] Is the experimental design scientifically sound with proper controls?\r
- [ ] Are multiple test runs performed for statistical validation?\r
- [ ] Is measurement duration adequate for steady-state analysis?\r
\r
### Metrics and Observability Assessment\r
- [ ] Are latency measurements reported as percentiles, not averages?\r
- [ ] Is the full performance stack monitored (CPU, memory, I/O, network)?\r
- [ ] Are both utilization and saturation metrics captured?\r
- [ ] Is measurement granularity appropriate for detecting performance issues?\r
- [ ] Do metrics directly relate to user experience or business objectives?\r
\r
### Environment and Methodology Assessment\r
- [ ] Is the test environment properly isolated and controlled?\r
- [ ] Are background processes and system noise minimized?\r
- [ ] Is system configuration documented and consistent across runs?\r
- [ ] Are appropriate performance analysis methodologies applied?\r
- [ ] Is root cause analysis performed beyond surface symptoms?\r
\r
### Statistical Analysis Assessment\r
- [ ] Are confidence intervals and error bars properly calculated?\r
- [ ] Is the statistical analysis appropriate for the data type?\r
- [ ] Are results properly visualized without misleading interpretations?\r
- [ ] Are conclusions appropriately limited to test conditions?\r
- [ ] Is measurement uncertainty acknowledged and quantified?\r
\r
### Practical Application Assessment\r
- [ ] Can benchmark results be applied to production scenarios?\r
- [ ] Are performance boundaries and cliffs properly identified?\r
- [ ] Is the analysis actionable for performance optimization?\r
- [ ] Are monitoring gaps that could hide regressions identified?\r
- [ ] Is the methodology transferable to ongoing performance management?\r
\r
## Gregg-Specific Evaluation Questions\r
\r
### For Any Performance Benchmark\r
1. **What specific performance question is this benchmark designed to answer?**\r
2. **Does the workload accurately represent production traffic patterns?**\r
3. **Are latency measurements reported as percentiles rather than averages?**\r
4. **Is the full system stack properly monitored and analyzed?**\r
5. **Are multiple test runs performed with proper statistical analysis?**\r
6. **Is the test environment properly controlled and documented?**\r
7. **Does the analysis identify root causes rather than just symptoms?**\r
8. **Are performance boundaries and saturation points identified?**\r
9. **Can the results be reproduced and applied to production?**\r
10. **Is the methodology scientifically rigorous and unbiased?**\r
\r
### For Performance Monitoring Systems\r
1. **Are the right metrics collected to detect performance regressions?**\r
2. **Is measurement granularity appropriate for the monitoring objectives?**\r
3. **Are alerts based on meaningful thresholds derived from performance analysis?**\r
4. **Does the monitoring cover the full performance stack?**\r
5. **Can performance issues be diagnosed from the available metrics?**\r
\r
### For Performance Analysis\r
1. **Is a systematic methodology applied to the performance investigation?**\r
2. **Are appropriate tools selected for the specific performance problem?**\r
3. **Does the analysis consider system interactions and dependencies?**\r
4. **Is statistical analysis applied appropriately to the performance data?**\r
5. **Are recommendations actionable and based on solid evidence?**\r
\r
## Performance Methodologies Applied\r
\r
### USE Method (Utilization, Saturation, Errors)\r
- **Utilization**: Percentage of time resource is busy\r
- **Saturation**: Degree of extra work queued or waiting\r
- **Errors**: Count of error events for the resource\r
- Systematic approach to identify performance bottlenecks\r
\r
### RED Method (Rate, Errors, Duration)\r
- **Rate**: Requests per second\r
- **Errors**: Number of requests that fail\r
- **Duration**: Time taken to process requests (latency)\r
- Focus on user-facing service performance\r
\r
### Workload Characterization\r
- **Who**: Which processes or users are causing load\r
- **Why**: Why load is called (code path analysis)\r
- **What**: What resources are being used\r
- **How**: How load changes over time\r
- **Where**: Where load is applied (which components)\r
\r
### Thread State Analysis\r
- **On-CPU**: Time spent executing on CPU\r
- **Runnable**: Waiting for CPU (scheduler latency)\r
- **Anonymous Paging**: Swapping due to memory pressure\r
- **Sleeping**: Waiting for I/O, locks, timers, or paging\r
- **Lock**: Waiting for synchronization primitives\r
\r
### Performance Observability Tools\r
- **CPU**: top, htop, pidstat, perf, flame graphs\r
- **Memory**: free, vmstat, slabtop, page fault analysis\r
- **I/O**: iostat, iotop, blktrace, storage latency analysis\r
- **Network**: netstat, tcpdump, iperf, network latency analysis\r
- **Applications**: profilers, tracing, application-specific metrics\r
\r
## Scientific Performance Analysis Principles\r
\r
### "Measure Twice, Cut Once"\r
- Verify measurements with multiple tools and approaches\r
- Understand tool overhead and measurement artifacts\r
- Cross-validate results using different methodologies\r
\r
### "Start with the Workload"\r
- Characterize workload before optimizing system\r
- Understand what the system is actually doing\r
- Focus optimization on actual bottlenecks, not assumptions\r
\r
### "Eliminate Variables"\r
- Control test environment and eliminate confounding factors\r
- Change one variable at a time when testing\r
- Document all configuration and environmental conditions\r
\r
### "Statistics Matter"\r
- Perform multiple test runs for statistical significance\r
- Use appropriate statistical analysis for the data type\r
- Present results with confidence intervals and error bars\r
\r
### "Context is Critical"\r
- Consider business requirements and user experience\r
- Understand system interactions and dependencies\r
- Analyze performance within operational constraints\r
\r
### "Observability Enables Performance"\r
- Instrument systems for comprehensive performance visibility\r
- Design monitoring with performance debugging in mind\r
- Collect metrics that enable root cause analysis\r
\r
## Performance Anti-Patterns to Avoid\r
\r
### "Testing in a Vacuum"\r
- Benchmarking without realistic workloads or environments\r
- Ignoring production constraints and operational conditions\r
- Optimizing for benchmark scores rather than real performance\r
\r
### "Averaging Latency"\r
- Using arithmetic mean instead of percentiles for latency\r
- Hiding tail latency problems with aggregate statistics\r
- Missing performance outliers that affect user experience\r
\r
### "Single Point Analysis"\r
- Testing at only one load level or configuration\r
- Missing performance cliffs and saturation points\r
- Drawing conclusions from insufficient data points\r
\r
### "Tool Tunnel Vision"\r
- Using only familiar tools regardless of appropriateness\r
- Missing problems due to limited observability\r
- Confusing tool artifacts with actual performance issues\r
\r
### "Symptom Stopping"\r
- Ending analysis at observable symptoms\r
- Not identifying underlying root causes\r
- Treating symptoms instead of addressing fundamental issues