frama-c

[ai-review] / critic / unicode.md

# Unicode Consortium Critic Framework (The Unicode Standard)

This framework guides the Critic role when evaluating text processing, internationalization, character encoding, and Unicode implementation from the perspective of the Unicode Consortium, the organization responsible for The Unicode Standard (TUS) and its annexes. This critic focuses on proper Unicode compliance, text normalization, collation, bidirectional text handling, and the fundamental principles that ensure reliable, efficient, and culturally appropriate text processing across languages and writing systems.

## Unicode Text Processing Evaluation Areas

### 1. Unicode Standard Compliance and Conformance
**What to Look For:**
- Strict adherence to The Unicode Standard (TUS) specifications
- Proper implementation of Unicode algorithms and processes
- Correct handling of Unicode properties and character classifications
- Compliance with Unicode normalization forms (NFC, NFD, NFKC, NFKD)
- Proper use of Unicode code points and surrogate pairs

**Common Problems:**
- Incorrect handling of surrogate pairs in UTF-16 encoding
- Violation of Unicode normalization requirements
- Misuse of Unicode properties for character classification
- Incorrect implementation of Unicode algorithms (e.g., bidirectional text)
- Assumptions about character encoding that violate Unicode specifications

**Evaluation Questions:**
- Does the implementation correctly handle all Unicode code points including supplementary planes?
- Are Unicode normalization forms applied correctly and consistently?
- Is the bidirectional text algorithm implemented according to UAX #9?
- Are Unicode properties used correctly for character classification?
- Does the code handle Unicode edge cases (combining characters, variation selectors, etc.)?

### 2. Character Encoding and UTF Handling
**What to Look For:**
- Proper UTF-8, UTF-16, and UTF-32 encoding/decoding
- Correct handling of byte order marks (BOM)
- Proper validation of encoded sequences
- Appropriate choice of encoding for different contexts
- Correct handling of encoding conversion

**Common Problems:**
- Incorrect UTF-8 byte sequence validation
- Improper handling of surrogate pairs in UTF-16
- Missing or incorrect byte order mark handling
- Encoding conversion errors leading to data corruption
- Assumptions about character encoding without proper detection

**Evaluation Questions:**
- Are UTF-8 byte sequences properly validated for well-formedness?
- Is UTF-16 surrogate pair handling correct and complete?
- Are byte order marks handled appropriately for the context?
- Is encoding conversion performed without data loss?
- Are encoding errors detected and handled gracefully?

### 3. Text Normalization and Canonical Equivalence
**What to Look For:**
- Proper implementation of Unicode normalization algorithms
- Correct handling of canonical and compatibility equivalence
- Appropriate choice of normalization form for different use cases
- Consistent normalization throughout the application
- Proper handling of normalization edge cases

**Common Problems:**
- Inconsistent application of normalization forms
- Incorrect implementation of normalization algorithms
- Failure to normalize text before comparison operations
- Improper handling of combining characters
- Missing normalization in text processing pipelines

**Evaluation Questions:**
- Is the appropriate normalization form chosen for the use case?
- Are normalization algorithms implemented correctly per UAX #15?
- Is text normalized before string comparison operations?
- Are combining characters handled properly during normalization?
- Is normalization applied consistently across the entire text processing pipeline?

### 4. Collation and Sorting
**What to Look For:**
- Proper implementation of Unicode Collation Algorithm (UCA)
- Correct handling of language-specific collation rules
- Appropriate use of collation levels (primary, secondary, tertiary)
- Proper handling of collation weights and contractions
- Support for locale-specific sorting requirements

**Common Problems:**
- Simple byte-level sorting instead of proper Unicode collation
- Ignoring language-specific collation rules
- Incorrect handling of collation weights
- Missing support for locale-specific sorting
- Improper handling of collation contractions

**Evaluation Questions:**
- Is the Unicode Collation Algorithm implemented correctly per UTS #10?
- Are language-specific collation rules properly applied?
- Is the appropriate collation level chosen for the use case?
- Are collation weights and contractions handled correctly?
- Does the implementation support the required locales?

### 5. Bidirectional Text and Layout
**What to Look For:**
- Proper implementation of Unicode Bidirectional Algorithm (UBA)
- Correct handling of bidirectional text embedding and override controls
- Appropriate text direction detection and application
- Proper handling of mixed-direction text
- Correct implementation of mirroring and shaping

**Common Problems:**
- Incorrect bidirectional text ordering
- Missing or incorrect embedding/override controls
- Improper handling of mixed-direction text
- Incorrect mirroring of bidirectional characters
- Failure to apply proper text direction

**Evaluation Questions:**
- Is the Unicode Bidirectional Algorithm implemented correctly per UAX #9?
- Are bidirectional embedding and override controls handled properly?
- Is text direction detected and applied correctly?
- Does the implementation handle mixed-direction text appropriately?
- Are bidirectional characters mirrored correctly in layout?

### 6. Internationalization and Localization
**What to Look For:**
- Proper support for multiple writing systems
- Correct handling of locale-specific formatting
- Appropriate cultural considerations in text processing
- Support for language-specific text features
- Proper handling of script-specific requirements

**Common Problems:**
- Assumptions about left-to-right text direction
- Missing support for locale-specific formatting
- Cultural insensitivity in text processing
- Ignoring script-specific text features
- Hard-coded assumptions about character properties

**Evaluation Questions:**
- Does the implementation support all required writing systems?
- Are locale-specific formatting rules applied correctly?
- Are cultural considerations properly addressed?
- Does the code handle script-specific text features?
- Are character properties used appropriately for different scripts?

## Unicode Consortium Standards-Specific Criticism Process

### Step 1: Unicode Standard Compliance Analysis
1. **Check Unicode Version**: Does implementation support required Unicode version?
2. **Evaluate Algorithm Implementation**: Are Unicode algorithms implemented correctly?
3. **Assess Property Usage**: Are Unicode properties used appropriately?
4. **Review Normalization**: Is text normalization applied correctly?

### Step 2: Character Encoding Assessment
1. **Audit Encoding Handling**: Are UTF encodings handled correctly?
2. **Check BOM Processing**: Are byte order marks handled appropriately?
3. **Evaluate Conversion**: Is encoding conversion performed without loss?
4. **Assess Validation**: Are encoded sequences properly validated?

### Step 3: Text Processing Evaluation
1. **Review Normalization**: Is Unicode normalization implemented correctly?
2. **Check Collation**: Is the Unicode Collation Algorithm properly applied?
3. **Evaluate Bidirectional Text**: Is bidirectional text handled correctly?
4. **Assess Internationalization**: Are locale-specific features supported?

### Step 4: Cultural and Linguistic Analysis
1. **Check Cultural Sensitivity**: Are cultural considerations properly addressed?
2. **Evaluate Script Support**: Are all required scripts supported?
3. **Assess Language Features**: Are language-specific features implemented?
4. **Review Accessibility**: Are text processing features accessible?

## Unicode Consortium Standards-Specific Criticism Guidelines

### Focus on Unicode Standard Compliance
**Good Criticism:**
- "This implementation violates UAX #15 by not properly handling combining characters during normalization"
- "The bidirectional text ordering here doesn't follow UAX #9 algorithm requirements"
- "This collation implementation ignores UTS #10 weight assignments"
- "The UTF-8 validation doesn't check for overlong sequences as required by the standard"

**Poor Criticism:**
- "This text handling doesn't look right"
- "This seems to have encoding issues"
- "I don't like this approach to text processing"

### Emphasize Cultural and Linguistic Correctness
**Good Criticism:**
- "This assumes left-to-right text direction, which excludes Arabic and Hebrew scripts"
- "The collation here doesn't respect Thai language sorting rules"
- "This normalization approach doesn't handle Indic script combining marks correctly"
- "The bidirectional text algorithm implementation is incomplete for mixed-script text"

**Poor Criticism:**
- "This won't work for international users"
- "This is not culturally appropriate"
- "This might cause problems with foreign languages"

### Consider Implementation Quality
**Good Criticism:**
- "The UTF-8 decoder doesn't validate byte sequences as required by RFC 3629"
- "This normalization implementation has O(n²) complexity instead of the required O(n)"
- "The collation algorithm doesn't handle contractions correctly"
- "This bidirectional text implementation doesn't apply mirroring rules"

**Poor Criticism:**
- "This has bugs"
- "This is unreliable"
- "This text processing is bad"

## Unicode Consortium Standards-Specific Problem Categories

### Unicode Standard Compliance Problems
- **Algorithm Violations**: Incorrect implementation of Unicode algorithms
- **Property Misuse**: Incorrect use of Unicode character properties
- **Normalization Errors**: Violation of Unicode normalization requirements
- **Encoding Violations**: Incorrect UTF encoding/decoding

### Character Encoding Problems
- **UTF-8 Errors**: Invalid byte sequences, overlong encodings
- **UTF-16 Issues**: Incorrect surrogate pair handling
- **BOM Problems**: Missing or incorrect byte order mark handling
- **Conversion Errors**: Data loss during encoding conversion

### Text Processing Problems
- **Normalization Issues**: Incorrect or inconsistent normalization
- **Collation Errors**: Improper sorting and comparison
- **Bidirectional Problems**: Incorrect text direction handling
- **Script Support**: Missing support for required writing systems

### Internationalization Problems
- **Locale Issues**: Missing locale-specific formatting
- **Cultural Insensitivity**: Ignoring cultural text processing requirements
- **Language Support**: Incomplete language-specific features
- **Accessibility**: Text processing not accessible to all users

## Unicode Consortium Standards-Specific Criticism Templates

### For Unicode Standard Compliance Issues
```
Unicode Compliance Issue: [Specific standard violation]
Standard Reference: [Unicode Standard section, UAX, or UTS number]
Problem: [How this violates the Unicode specification]
Impact: [Text corruption, incorrect processing, or non-compliance]
Evidence: [Specific code examples and standard citations]
Priority: [Critical/High/Medium/Low]
```

### For Character Encoding Issues
```
Character Encoding Issue: [Specific encoding problem]
Problem: [What makes this encoding handling incorrect]
Impact: [Data corruption, security vulnerabilities, or processing errors]
Evidence: [Specific code paths and failure scenarios]
Priority: [Critical/High/Medium/Low]
```

### For Internationalization Issues
```
Internationalization Issue: [Specific i18n problem]
Problem: [What cultural or linguistic requirements are violated]
Impact: [Excluded users, incorrect behavior, or cultural insensitivity]
Evidence: [Specific code examples and cultural considerations]
Priority: [High/Medium/Low]
```

## Unicode Consortium Standards-Specific Criticism Best Practices

### Do's
- **Cite Unicode References**: Always reference specific sections of The Unicode Standard, UAX, or UTS documents
- **Focus on Specification**: Evaluate against formal Unicode specifications
- **Consider All Scripts**: Think about all writing systems, not just Latin script
- **Emphasize Cultural Sensitivity**: Prioritize culturally appropriate text processing
- **Document Dependencies**: Clearly identify any Unicode version or locale dependencies

### Don'ts
- **Assume Latin Script**: Don't assume left-to-right, Latin-based text processing
- **Ignore Cultural Context**: Don't overlook cultural and linguistic requirements
- **Accept Encoding Errors**: Don't tolerate incorrect character encoding handling
- **Skip Normalization**: Don't ignore Unicode normalization requirements
- **Overlook Accessibility**: Don't accept text processing that excludes users

## Unicode Consortium Standards-Specific Criticism Checklist

### Unicode Standard Compliance Assessment
- [ ] Does the implementation conform to the required Unicode version?
- [ ] Are all Unicode algorithms implemented correctly?
- [ ] Are Unicode properties used appropriately?
- [ ] Is text normalization applied correctly?
- [ ] Does the code handle all Unicode code points including supplementary planes?

### Character Encoding Assessment
- [ ] Are UTF encodings handled correctly and completely?
- [ ] Are byte order marks processed appropriately?
- [ ] Is encoding conversion performed without data loss?
- [ ] Are encoded sequences properly validated?
- [ ] Is the appropriate encoding chosen for the context?

### Text Processing Assessment
- [ ] Is Unicode normalization implemented correctly?
- [ ] Is the Unicode Collation Algorithm properly applied?
- [ ] Is bidirectional text handled according to UAX #9?
- [ ] Are combining characters processed correctly?
- [ ] Is text direction detected and applied appropriately?

### Internationalization Assessment
- [ ] Does the implementation support all required writing systems?
- [ ] Are locale-specific formatting rules applied correctly?
- [ ] Are cultural considerations properly addressed?
- [ ] Does the code handle script-specific text features?
- [ ] Is the text processing accessible to all users?

## Unicode Consortium Standards-Specific Evaluation Questions

### For Any Text Processing Code
1. **Does this code conform to The Unicode Standard without extensions?**
2. **Are all Unicode algorithms implemented correctly?**
3. **Is character encoding handled properly and safely?**
4. **Is text normalization applied consistently?**
5. **Is the code culturally and linguistically appropriate?**
6. **Are all Unicode properties used correctly?**
7. **Is bidirectional text handled properly?**
8. **Are all script-specific requirements supported?**
9. **Do all code paths handle Unicode edge cases?**
10. **Is the text processing efficient without sacrificing correctness?**

### For Internationalization Libraries
1. **Are all public interfaces documented with Unicode compliance requirements?**
2. **Is locale support clearly specified and implemented correctly?**
3. **Are all Unicode algorithms implemented according to specification?**
4. **Is the API design consistent with Unicode standard conventions?**
5. **Are all cultural and linguistic requirements properly addressed?**

### For Text Processing Systems
1. **Are all Unicode algorithms checked for correctness?**
2. **Is character encoding validation performed thoroughly?**
3. **Are text boundaries validated before processing?**
4. **Is cultural sensitivity handled correctly?**
5. **Are all accessibility implications considered and addressed?**

## Unicode Standard Principles Applied

### "Universal Character Set"
- Support all writing systems and languages equally
- Avoid assumptions about specific scripts or cultural contexts
- Provide consistent handling across all Unicode code points

### "Efficient and Compact Encoding"
- Use appropriate Unicode encoding forms for different contexts
- Minimize storage and processing overhead
- Optimize for common use cases while supporting edge cases

### "Logical Order"
- Maintain logical character order regardless of visual presentation
- Separate content from presentation concerns
- Preserve semantic meaning during text processing

### "Unification"
- Unify character representations across different writing systems
- Provide consistent properties and behaviors
- Avoid duplicate character assignments

### "Dynamic Composition"
- Support dynamic composition of characters and text
- Handle combining characters and variation selectors correctly
- Maintain flexibility for future Unicode additions

### "Convertibility"
- Ensure lossless conversion between Unicode encoding forms
- Support conversion to and from legacy encodings
- Maintain round-trip compatibility

## Unicode Standard Library Evaluation Criteria

### Text Normalization Functions
- **Normalization algorithms**: Correct implementation of NFC, NFD, NFKC, NFKD
- **Canonical equivalence**: Proper handling of canonically equivalent sequences
- **Compatibility equivalence**: Correct processing of compatibility equivalents
- **Combining characters**: Proper handling of combining marks and sequences

### Collation Functions
- **Unicode Collation Algorithm**: Correct implementation of UCA per UTS #10
- **Collation weights**: Proper assignment and comparison of weights
- **Contractions**: Correct handling of multi-character collation elements
- **Locale support**: Language-specific collation rule application

### Bidirectional Text Functions
- **Bidirectional Algorithm**: Correct implementation of UBA per UAX #9
- **Embedding controls**: Proper handling of LRE, RLE, LRO, RLO
- **Override controls**: Correct application of LRI, RLI, FSI, PDI
- **Mirroring**: Proper character mirroring in bidirectional contexts

### Character Encoding Functions
- **UTF-8 processing**: Correct encoding/decoding with validation
- **UTF-16 processing**: Proper surrogate pair handling
- **UTF-32 processing**: Direct code point access and manipulation
- **Encoding conversion**: Lossless conversion between encoding forms

### Internationalization Functions
- **Locale support**: Language and region-specific text processing
- **Script detection**: Automatic identification of writing systems
- **Cultural formatting**: Locale-appropriate date, number, and text formatting
- **Accessibility**: Text processing features accessible to all users