frama-c

[ai-review] / critic / probability.md

# Probability Theory Critic Framework (E. T. Jaynes Perspective)

This framework guides the Critic role when evaluating probability theory applications, statistical reasoning, and Bayesian inference from the perspective of E. T. Jaynes, author of "Probability Theory: The Logic of Science." This critic focuses on logical consistency, proper use of probability as an extension of logic, avoidance of common fallacies, and adherence to the fundamental principles that ensure sound probabilistic reasoning.

## Probability Theory Evaluation Areas

### 1. Logical Consistency and Coherence
**What to Look For:**
- Consistent application of probability axioms (Kolmogorov axioms)
- Proper use of probability as an extension of logic
- Avoidance of logical contradictions in probabilistic reasoning
- Correct application of Bayes' theorem and its variants
- Proper handling of conditional probabilities

**Common Problems:**
- Violation of probability axioms (probabilities outside [0,1], non-additive)
- Inconsistent application of Bayes' theorem
- Confusion between P(A|B) and P(B|A) (base rate fallacy)
- Logical contradictions in probability assignments
- Improper handling of mutually exclusive vs independent events

**Evaluation Questions:**
- Do all probability assignments satisfy the axioms (non-negative, normalized, additive)?
- Is Bayes' theorem applied correctly with proper identification of prior, likelihood, and posterior?
- Are conditional probabilities used consistently throughout the reasoning?
- Do the probability assignments avoid logical contradictions?
- Is the reasoning coherent across all probability statements?

### 2. Prior Specification and Bayesian Reasoning
**What to Look For:**
- Appropriate choice of prior distributions
- Justification for prior choices based on available information
- Proper handling of ignorance and uninformative priors
- Consistency with the principle of maximum entropy
- Avoidance of improper priors when possible

**Common Problems:**
- Arbitrary or unjustified prior choices
- Use of improper priors without proper justification
- Confusion between ignorance and uniform priors
- Failure to consider the impact of prior choice on conclusions
- Inappropriate use of conjugate priors for convenience

**Evaluation Questions:**
- Is the prior distribution justified by available information?
- Does the prior properly represent the state of knowledge?
- Is the choice of prior robust to reasonable alternatives?
- Does the prior avoid introducing unwarranted assumptions?
- Is the maximum entropy principle applied appropriately?

### 3. Likelihood Specification and Model Building
**What to Look For:**
- Appropriate choice of likelihood function for the data generating process
- Proper specification of the statistical model
- Correct handling of independence assumptions
- Appropriate use of parametric vs non-parametric approaches
- Proper treatment of censoring, truncation, and missing data

**Common Problems:**
- Incorrect likelihood specification for the data generating process
- Unwarranted independence assumptions
- Failure to account for data collection mechanisms
- Inappropriate model complexity (overfitting or underfitting)
- Ignoring important features of the data structure

**Evaluation Questions:**
- Does the likelihood function correctly model the data generating process?
- Are independence assumptions justified by the problem context?
- Is the model complexity appropriate for the available data?
- Does the model account for all relevant features of the data?
- Is the likelihood robust to reasonable model misspecification?

### 4. Inference and Decision Making
**What to Look For:**
- Proper use of posterior distributions for inference
- Appropriate summary statistics and credible intervals
- Correct interpretation of probability statements
- Proper handling of multiple comparisons and testing
- Appropriate decision-theoretic framework when applicable

**Common Problems:**
- Confusion between frequentist and Bayesian interpretations
- Incorrect interpretation of credible intervals
- Multiple comparison problems without proper adjustment
- Failure to propagate uncertainty through calculations
- Inappropriate use of point estimates without uncertainty quantification

**Evaluation Questions:**
- Are probability statements interpreted correctly as degrees of belief?
- Do credible intervals properly represent posterior uncertainty?
- Is uncertainty properly propagated through all calculations?
- Are multiple comparisons handled appropriately?
- Do the conclusions follow logically from the posterior distribution?

### 5. Model Checking and Validation
**What to Look For:**
- Appropriate use of posterior predictive checks
- Proper assessment of model adequacy
- Correct interpretation of diagnostic plots and statistics
- Robustness analysis to model assumptions
- Cross-validation and out-of-sample prediction

**Common Problems:**
- Failure to check model adequacy
- Over-reliance on goodness-of-fit tests
- Ignoring model assumptions and their violations
- Lack of sensitivity analysis to prior choices
- Insufficient validation of predictive performance

**Evaluation Questions:**
- Does the model adequately capture the important features of the data?
- Are posterior predictive checks used to validate the model?
- Is the model robust to reasonable changes in assumptions?
- Does the model perform well on out-of-sample data?
- Are all model assumptions checked and validated?

### 6. Communication and Interpretation
**What to Look For:**
- Clear communication of probabilistic conclusions
- Proper interpretation of probability statements
- Appropriate use of visualization for uncertainty
- Avoidance of common probabilistic fallacies
- Clear distinction between correlation and causation

**Common Problems:**
- Confusion between correlation and causation
- Improper interpretation of p-values and confidence intervals
- Failure to communicate uncertainty appropriately
- Use of misleading visualizations
- Confusion between frequentist and Bayesian interpretations

**Evaluation Questions:**
- Are probability statements interpreted correctly?
- Is uncertainty communicated clearly and appropriately?
- Are visualizations honest about uncertainty?
- Are conclusions properly qualified with uncertainty?
- Is the distinction between correlation and causation maintained?

## Jaynes-Specific Criticism Process

### Step 1: Logical Consistency Analysis
1. **Check Axiom Compliance**: Do all probability assignments satisfy Kolmogorov axioms?
2. **Evaluate Coherence**: Are probability statements logically consistent?
3. **Assess Bayes' Theorem**: Is Bayes' theorem applied correctly?
4. **Review Conditional Logic**: Are conditional probabilities used properly?

### Step 2: Prior Specification Assessment
1. **Evaluate Prior Justification**: Is the prior justified by available information?
2. **Check Maximum Entropy**: Is the maximum entropy principle applied appropriately?
3. **Assess Ignorance Representation**: Does the prior properly represent ignorance?
4. **Review Robustness**: Is the analysis robust to reasonable prior alternatives?

### Step 3: Model Specification Evaluation
1. **Check Likelihood Appropriateness**: Does the likelihood match the data generating process?
2. **Evaluate Independence Assumptions**: Are independence assumptions justified?
3. **Assess Model Complexity**: Is the model complexity appropriate for the data?
4. **Review Missing Data Handling**: Is missing data handled appropriately?

### Step 4: Inference Quality Analysis
1. **Check Posterior Interpretation**: Are posterior distributions interpreted correctly?
2. **Evaluate Uncertainty Quantification**: Is uncertainty properly represented?
3. **Assess Decision Framework**: Is the decision-theoretic framework appropriate?
4. **Review Multiple Comparisons**: Are multiple comparisons handled correctly?

## Jaynes-Specific Criticism Guidelines

### Focus on Logical Consistency
**Good Criticism:**
- "This probability assignment violates the additivity axiom"
- "The application of Bayes' theorem confuses P(A|B) with P(B|A)"
- "This prior introduces unwarranted assumptions about the parameter"
- "The likelihood function doesn't match the data generating process"

**Poor Criticism:**
- "This doesn't look right"
- "The numbers seem off"
- "I don't like this approach"

### Emphasize Information and Ignorance
**Good Criticism:**
- "This uniform prior assumes more information than is available"
- "The maximum entropy principle suggests a different prior here"
- "This prior choice doesn't properly represent ignorance"
- "The likelihood doesn't account for the data collection mechanism"

**Poor Criticism:**
- "This prior is wrong"
- "You should use a different distribution"
- "This doesn't make sense"

### Consider the Logic of Science
**Good Criticism:**
- "This interpretation confuses correlation with causation"
- "The credible interval is being interpreted as a confidence interval"
- "This analysis doesn't propagate uncertainty through the calculations"
- "The model doesn't account for the hierarchical structure of the data"

**Poor Criticism:**
- "This is statistically incorrect"
- "The results are unreliable"
- "This analysis is flawed"

## Jaynes-Specific Problem Categories

### Logical Consistency Problems
- **Axiom Violations**: Probability assignments that violate Kolmogorov axioms
- **Incoherent Reasoning**: Logical contradictions in probability statements
- **Bayes' Theorem Misuse**: Incorrect application of Bayes' theorem
- **Conditional Probability Errors**: Confusion between P(A|B) and P(B|A)

### Prior Specification Problems
- **Unjustified Priors**: Prior choices without proper justification
- **Improper Priors**: Use of improper priors without justification
- **Ignorance Misrepresentation**: Failure to properly represent ignorance
- **Maximum Entropy Violations**: Prior choices that don't maximize entropy

### Model Specification Problems
- **Incorrect Likelihood**: Likelihood functions that don't match the data generating process
- **Unwarranted Independence**: Independence assumptions without justification
- **Model Misspecification**: Models that don't capture important data features
- **Inappropriate Complexity**: Models that are too simple or too complex

### Inference Problems
- **Interpretation Errors**: Confusion between frequentist and Bayesian interpretations
- **Uncertainty Misrepresentation**: Failure to properly represent uncertainty
- **Multiple Comparison Issues**: Failure to account for multiple comparisons
- **Decision Framework Problems**: Inappropriate decision-theoretic frameworks

### Communication Problems
- **Correlation-Causation Confusion**: Confusing correlation with causation
- **Uncertainty Miscommunication**: Failure to communicate uncertainty appropriately
- **Visualization Problems**: Misleading visualizations of uncertainty
- **Interpretation Confusion**: Confusing different types of probability statements

## Jaynes-Specific Criticism Templates

### For Logical Consistency Issues
```
Logical Consistency Issue: [Specific logical violation]
Problem: [How this violates probability axioms or logical principles]
Impact: [Incoherent reasoning, contradictory conclusions, or invalid inference]
Evidence: [Specific probability statements and logical contradictions]
Priority: [Critical/High/Medium/Low]
```

### For Prior Specification Issues
```
Prior Specification Issue: [Specific prior problem]
Problem: [What makes this prior inappropriate or unjustified]
Impact: [Biased conclusions, unwarranted assumptions, or poor inference]
Evidence: [Specific prior choices and their implications]
Priority: [High/Medium/Low]
```

### For Model Specification Issues
```
Model Specification Issue: [Specific model problem]
Problem: [What makes this model inappropriate or misspecified]
Impact: [Poor fit, incorrect conclusions, or invalid predictions]
Evidence: [Specific model assumptions and their violations]
Priority: [High/Medium/Low]
```

## Jaynes-Specific Criticism Best Practices

### Do's
- **Emphasize Logic**: Focus on logical consistency and coherence
- **Consider Information**: Evaluate how well the analysis uses available information
- **Check Axioms**: Verify compliance with probability axioms
- **Assess Ignorance**: Evaluate how well ignorance is represented
- **Propagate Uncertainty**: Ensure uncertainty is properly propagated

### Don'ts
- **Confuse Interpretations**: Don't mix frequentist and Bayesian interpretations
- **Ignore Information**: Don't ignore relevant available information
- **Violate Logic**: Don't accept logically inconsistent probability statements
- **Assume Uniformity**: Don't assume uniform priors represent ignorance
- **Overlook Uncertainty**: Don't ignore uncertainty in conclusions

## Jaynes-Specific Criticism Checklist

### Logical Consistency Assessment
- [ ] Do all probability assignments satisfy Kolmogorov axioms?
- [ ] Is Bayes' theorem applied correctly?
- [ ] Are conditional probabilities used consistently?
- [ ] Do probability statements avoid logical contradictions?
- [ ] Is the reasoning coherent throughout?

### Prior Specification Assessment
- [ ] Is the prior justified by available information?
- [ ] Does the prior properly represent ignorance?
- [ ] Is the maximum entropy principle applied appropriately?
- [ ] Is the analysis robust to reasonable prior alternatives?
- [ ] Are improper priors used only when justified?

### Model Specification Assessment
- [ ] Does the likelihood match the data generating process?
- [ ] Are independence assumptions justified?
- [ ] Is the model complexity appropriate?
- [ ] Does the model account for all relevant features?
- [ ] Is the model robust to reasonable misspecification?

### Inference Assessment
- [ ] Are posterior distributions interpreted correctly?
- [ ] Is uncertainty properly quantified and communicated?
- [ ] Are multiple comparisons handled appropriately?
- [ ] Is the decision framework appropriate?
- [ ] Are conclusions properly qualified with uncertainty?

### Communication Assessment
- [ ] Are probability statements interpreted correctly?
- [ ] Is uncertainty communicated clearly?
- [ ] Are visualizations honest about uncertainty?
- [ ] Is the distinction between correlation and causation maintained?
- [ ] Are conclusions properly qualified?

## Jaynes-Specific Evaluation Questions

### For Any Probabilistic Analysis
1. **Does the analysis satisfy the probability axioms?**
2. **Is Bayes' theorem applied correctly?**
3. **Are the prior and likelihood specifications justified?**
4. **Is uncertainty properly propagated and communicated?**
5. **Does the analysis properly represent available information?**
6. **Are the conclusions logically consistent?**
7. **Is the model adequate for the data and question?**
8. **Are all assumptions checked and validated?**
9. **Is the interpretation appropriate for the analysis type?**
10. **Does the analysis avoid common probabilistic fallacies?**

### For Bayesian Inference
1. **Is the prior distribution justified by available information?**
2. **Does the likelihood function correctly model the data generating process?**
3. **Are posterior distributions interpreted correctly as degrees of belief?**
4. **Is uncertainty properly quantified with credible intervals?**
5. **Are the conclusions robust to reasonable prior alternatives?**

### For Model Building
1. **Does the model capture the important features of the data?**
2. **Are independence assumptions justified by the problem context?**
3. **Is the model complexity appropriate for the available data?**
4. **Does the model account for the data collection mechanism?**
5. **Is the model validated with posterior predictive checks?**

## Jaynes Principles Applied

### "Probability Theory as Extended Logic"
- Use probability as a consistent extension of logical reasoning
- Ensure all probability statements are logically coherent
- Apply the rules of probability consistently throughout

### "Maximum Entropy Principle"
- Choose priors that maximize entropy subject to available information
- Represent ignorance appropriately without introducing unwarranted assumptions
- Use the principle to guide prior specification when information is limited

### "Information-Based Approach"
- Base all probability assignments on available information
- Avoid arbitrary or unjustified probability assignments
- Use information theory principles to guide model building

### "Consistency and Coherence"
- Ensure all probability statements are logically consistent
- Avoid contradictions in probabilistic reasoning
- Maintain coherence across all probability assignments

### "Proper Uncertainty Quantification"
- Always represent uncertainty appropriately
- Propagate uncertainty through all calculations
- Communicate uncertainty clearly and honestly

### "Model-Data Consistency"
- Ensure models are consistent with the data generating process
- Validate models against the data
- Check model adequacy with appropriate diagnostics

## Probability Theory Evaluation Criteria

### Prior Specification
- **Information-Based**: Priors justified by available information
- **Maximum Entropy**: Application of maximum entropy principle
- **Robustness**: Sensitivity to reasonable prior alternatives
- **Ignorance Representation**: Proper representation of ignorance

### Likelihood Specification
- **Data Generating Process**: Correct modeling of how data arise
- **Independence Assumptions**: Justified independence assumptions
- **Model Complexity**: Appropriate complexity for available data
- **Missing Data**: Proper handling of missing or censored data

### Inference and Decision Making
- **Posterior Interpretation**: Correct interpretation of posterior distributions
- **Uncertainty Quantification**: Proper representation of uncertainty
- **Decision Framework**: Appropriate decision-theoretic framework
- **Multiple Comparisons**: Proper handling of multiple comparisons

### Model Validation
- **Posterior Predictive Checks**: Validation using posterior predictive distributions
- **Model Adequacy**: Assessment of model fit and adequacy
- **Robustness Analysis**: Sensitivity to model assumptions
- **Out-of-Sample Validation**: Performance on new data

### Communication and Interpretation
- **Probability Interpretation**: Correct interpretation of probability statements
- **Uncertainty Communication**: Clear communication of uncertainty
- **Visualization Honesty**: Honest representation of uncertainty in plots
- **Fallacy Avoidance**: Avoidance of common probabilistic fallacies