chopratejas commited on
Commit
513ec0c
·
1 Parent(s): 7735dd6

fix(rust): smart_crusher scaffold review findings — hash truncation, int parse, python-repr matcher

Browse files

Code review (`/code-review` on commit `7735dd6`) caught one critical
bug, two important parity gaps, and a few quality nits. Fixed all of
them; all 135 unit tests pass; diff_compressor parity harness
unaffected (27/27 still matched).

# Critical fix — `hash_field_name` truncation length

Rust truncated SHA-256 to **16** hex chars; Python uses **8** (per
`smart_crusher.py:177`: `hashlib.sha256(...).hexdigest()[:8]`). 16-char
hashes would never collide with TOIN's 8-char `preserve_fields`,
silently disabling the entire `use_feedback_hints` cache lookup path.

Fix: `hex[..8]` instead of `hex[..16]`. Three pinning tests re-verified
against actual Python reference output. Doc comment now warns
explicitly that the length must match Python or TOIN lookups silently miss.

# Important fix — `python_int_parse` mirrors Python's `int()` semantics

`statistics.rs::detect_sequential_pattern` previously called
`s.parse::<i64>()`. Python's `int()` differs in three ways that affect
realistic payloads:
- strips ASCII whitespace (Rust's `parse` rejects)
- accepts leading `+` (Rust accepts; same)
- accepts PEP 515 underscores like `"3_000"` (Rust rejects)

A field with `[" 1 ", " 2 ", " 3 ", "4", "5"]` would parse all five
in Python (sequential = True) but only one in Rust (`nums.len() < 5`
→ False). Silent parity break.

Fix: new private `python_int_parse` helper that strips whitespace,
handles underscore separators, and rejects edge cases Python rejects.
Six new tests pin the behavior.

# Important fix — `python_repr` for `item_matches_anchors`

Python compares anchors via `anchor in str(item).lower()`. We were
using `serde_json::to_string(&item).to_lowercase()`, which differs in
three ways that affect substring matching:
- quote chars (`'` vs `"`)
- bool/null literals (`True`/`False`/`None` vs `true`/`false`/`null`)
- spacing (`key: value, ...` vs `key:value,...`)

Anchor `"none"` would match Python form but not JSON. Inverse for
`"null"`. Real divergence.

Fix: new private `python_repr` walks `serde_json::Value` and emits
Python-equivalent form. Plus enable `serde_json/preserve_order` at
workspace level so `Value::Object` preserves JSON parse order
(matching Python `dict` since 3.7).

# Suggestion fixes

- Classifier comment for `[True, False, 1] -> MIXED_ARRAY` now walks
both Python and Rust paths step by step.
- `ArrayAnalysis::field_stats` doc notes the BTreeMap vs Python-dict
order nuance for the analyzer port to resolve.
- Added regression tests for "all unparseable strings", "single int
among strings", fractional-step sequential, and the email-typo
pattern.

# Build / test

- `cargo build -p headroom-core` clean.
- `cargo clippy -p headroom-core -- -D warnings` clean.
- 135 unit tests in `headroom-core`, all passing (was 55).
- `cargo run -p headroom-parity run` — diff_compressor 27/27 still matched.

.claude-plugin/marketplace.json CHANGED
@@ -5,14 +5,14 @@
5
  },
6
  "metadata": {
7
  "description": "Headroom marketplace for Claude Code and GitHub Copilot CLI plugins.",
8
- "version": "0.10.17"
9
  },
10
  "plugins": [
11
  {
12
  "name": "headroom",
13
  "source": "./plugins/headroom-agent-hooks",
14
  "description": "Headroom startup hooks for Claude Code and GitHub Copilot CLI.",
15
- "version": "0.10.17",
16
  "author": {
17
  "name": "Headroom Contributors",
18
  "url": "https://github.com/chopratejas/headroom"
 
5
  },
6
  "metadata": {
7
  "description": "Headroom marketplace for Claude Code and GitHub Copilot CLI plugins.",
8
+ "version": "0.11.0"
9
  },
10
  "plugins": [
11
  {
12
  "name": "headroom",
13
  "source": "./plugins/headroom-agent-hooks",
14
  "description": "Headroom startup hooks for Claude Code and GitHub Copilot CLI.",
15
+ "version": "0.11.0",
16
  "author": {
17
  "name": "Headroom Contributors",
18
  "url": "https://github.com/chopratejas/headroom"
.github/plugin/marketplace.json CHANGED
@@ -5,14 +5,14 @@
5
  },
6
  "metadata": {
7
  "description": "Headroom marketplace for Claude Code and GitHub Copilot CLI plugins.",
8
- "version": "0.10.17"
9
  },
10
  "plugins": [
11
  {
12
  "name": "headroom",
13
  "source": "./plugins/headroom-agent-hooks",
14
  "description": "Headroom startup hooks for Claude Code and GitHub Copilot CLI.",
15
- "version": "0.10.17",
16
  "author": {
17
  "name": "Headroom Contributors",
18
  "url": "https://github.com/chopratejas/headroom"
 
5
  },
6
  "metadata": {
7
  "description": "Headroom marketplace for Claude Code and GitHub Copilot CLI plugins.",
8
+ "version": "0.11.0"
9
  },
10
  "plugins": [
11
  {
12
  "name": "headroom",
13
  "source": "./plugins/headroom-agent-hooks",
14
  "description": "Headroom startup hooks for Claude Code and GitHub Copilot CLI.",
15
+ "version": "0.11.0",
16
  "author": {
17
  "name": "Headroom Contributors",
18
  "url": "https://github.com/chopratejas/headroom"
Cargo.lock CHANGED
@@ -2198,6 +2198,7 @@ version = "1.0.149"
2198
  source = "registry+https://github.com/rust-lang/crates.io-index"
2199
  checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86"
2200
  dependencies = [
 
2201
  "itoa",
2202
  "memchr",
2203
  "serde",
 
2198
  source = "registry+https://github.com/rust-lang/crates.io-index"
2199
  checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86"
2200
  dependencies = [
2201
+ "indexmap",
2202
  "itoa",
2203
  "memchr",
2204
  "serde",
Cargo.toml CHANGED
@@ -26,7 +26,12 @@ authors = ["Headroom Maintainers"]
26
 
27
  [workspace.dependencies]
28
  serde = { version = "1", features = ["derive"] }
29
- serde_json = "1"
 
 
 
 
 
30
  bytes = "1"
31
  thiserror = "1"
32
  tracing = "0.1"
 
26
 
27
  [workspace.dependencies]
28
  serde = { version = "1", features = ["derive"] }
29
+ # `preserve_order` makes `serde_json::Value::Object` use IndexMap so JSON
30
+ # parse order is preserved through Value→string→Value round-trips. The
31
+ # smart_crusher port relies on this to match Python's `str(dict)` output,
32
+ # which preserves insertion order; otherwise BTreeMap's sorted-key default
33
+ # would diverge from Python on every multi-key object.
34
+ serde_json = { version = "1", features = ["preserve_order"] }
35
  bytes = "1"
36
  thiserror = "1"
37
  tracing = "0.1"
crates/headroom-core/src/transforms/smart_crusher/anchors.rs CHANGED
@@ -111,43 +111,117 @@ pub fn extract_query_anchors(text: &str) -> HashSet<String> {
111
  anchors
112
  }
113
 
114
- /// Check if a JSON object matches any query anchors.
 
115
  ///
116
- /// Direct port of `item_matches_anchors` (Python `smart_crusher.py:152-168`).
117
- /// Python uses `str(item).lower()` which produces Python's `dict.__str__`
118
- /// representation. We mirror by serializing with `serde_json` and
119
- /// lowercasing — this isn't byte-identical to Python's `str(dict)`
120
- /// (Python uses single quotes, JSON uses double; Python's bool is
121
- /// `True`/`False`, JSON's is `true`/`false`), so for cross-language
122
- /// parity we need a string form that matches Python's. We document this
123
- /// gap and fix it in the analyzer integration.
124
  ///
125
- /// **WARNING:** `str(item).lower()` in Python produces:
126
- /// `{'key': 'value', 'count': 5, 'ok': True}`
127
- /// while `serde_json::to_string(&item)` produces:
128
- /// `{"key":"value","count":5,"ok":true}`
 
129
  ///
130
- /// The anchor matching is substring-based (`anchor in item_str`), so
131
- /// this difference matters: if an anchor is `"true"` it matches the
132
- /// JSON form but not the Python form, and vice versa for `"True"`.
 
 
 
 
 
 
133
  ///
134
- /// **Resolution:** when items reach the matcher they're already
135
- /// lowercased, so `True` `true` after `.lower()`, removing one source
136
- /// of drift. The remaining drift (single vs double quotes, trailing
137
- /// whitespace) is unlikely to affect anchor matching in practice. We
138
- /// pin behavior with fixtures and move on.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
139
  pub fn item_matches_anchors(item: &Value, anchors: &HashSet<String>) -> bool {
140
  if anchors.is_empty() {
141
  return false;
142
  }
143
 
144
- // Python: `str(item).lower()`. We approximate via JSON serialization
145
- // followed by `.lower()` — see WARNING above for the gap.
146
- let item_str = match serde_json::to_string(item) {
147
- Ok(s) => s.to_lowercase(),
148
- Err(_) => return false,
149
- };
150
-
151
  anchors.iter().any(|a| item_str.contains(a))
152
  }
153
 
@@ -248,4 +322,92 @@ mod tests {
248
  let anchors: HashSet<String> = ["xyz123".to_string()].into_iter().collect();
249
  assert!(!item_matches_anchors(&json!({"a": "b"}), &anchors));
250
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
251
  }
 
111
  anchors
112
  }
113
 
114
+ /// Serialize a `serde_json::Value` to a string matching Python's
115
+ /// `str()` of the equivalent native value.
116
  ///
117
+ /// Used by `item_matches_anchors` because Python compares anchors via
118
+ /// `anchor in str(item).lower()` and `str(dict)` differs from
119
+ /// `json.dumps(dict)` in three ways that affect substring matching:
 
 
 
 
 
120
  ///
121
+ /// | Aspect | Python `str(dict)` | `serde_json::to_string` |
122
+ /// |------------------|------------------------------|-------------------------|
123
+ /// | String quotes | single `'` | double `"` |
124
+ /// | Booleans / null | `True`, `False`, `None` | `true`, `false`, `null` |
125
+ /// | Spacing | `key: value`, `a, b` | `key:value`, `a,b` |
126
  ///
127
+ /// All three matter for anchor matching:
128
+ /// - An anchor `"name': 'a"` extracted from a user phrase like
129
+ /// `find {'name': 'alice'}` would match Python's serialization but
130
+ /// never the JSON form.
131
+ /// - An anchor `"true"` (lowercased from `"True"`) matches both, but
132
+ /// the unlowercased version `"True"` is in Python output and not
133
+ /// JSON. Lowercasing both sides handles this.
134
+ /// - An anchor `"name: alice"` (with the space) would match Python
135
+ /// but never JSON.
136
  ///
137
+ /// Output is then lowercased upstream (matching Python's `.lower()`)
138
+ /// so True/False/None case is normalized away after that step.
139
+ fn python_repr(value: &Value) -> String {
140
+ let mut out = String::new();
141
+ write_python_repr(&mut out, value);
142
+ out
143
+ }
144
+
145
+ fn write_python_repr(out: &mut String, value: &Value) {
146
+ match value {
147
+ Value::Null => out.push_str("None"),
148
+ Value::Bool(true) => out.push_str("True"),
149
+ Value::Bool(false) => out.push_str("False"),
150
+ Value::Number(n) => {
151
+ // Python `str(int)` and `str(float)` produce minimal forms.
152
+ // `serde_json::Number`'s `Display` matches Python for ints
153
+ // (`5`) but for floats it can write `1.0` while Python may
154
+ // write `1.0` too — close enough for substring matching
155
+ // since anchor strings rarely contain numeric literals
156
+ // beyond the digit prefix.
157
+ out.push_str(&n.to_string());
158
+ }
159
+ Value::String(s) => {
160
+ // Python `repr(s)` chooses single or double quotes
161
+ // depending on content. Default preference is single
162
+ // quotes; switches to double if the string contains a
163
+ // single quote and no double. We emit single quotes
164
+ // always — this matches the dominant case (no quotes in
165
+ // the string) and is what Python does for `str(dict)` of
166
+ // most realistic data. The rare case where Python would
167
+ // switch to double quotes is documented as a known parity
168
+ // gap in `python_repr_string_with_single_quote_drift`.
169
+ out.push('\'');
170
+ out.push_str(s);
171
+ out.push('\'');
172
+ }
173
+ Value::Array(items) => {
174
+ out.push('[');
175
+ for (i, item) in items.iter().enumerate() {
176
+ if i > 0 {
177
+ out.push_str(", ");
178
+ }
179
+ write_python_repr(out, item);
180
+ }
181
+ out.push(']');
182
+ }
183
+ Value::Object(map) => {
184
+ out.push('{');
185
+ // Python preserves insertion order in `dict.__str__` (since
186
+ // Python 3.7). We require the workspace `serde_json` to be
187
+ // built with `preserve_order` so `serde_json::Map` uses
188
+ // `IndexMap` instead of the default `BTreeMap` — see the
189
+ // comment on `serde_json` in the workspace `Cargo.toml`.
190
+ // Without that feature, this iteration is sorted-by-key
191
+ // and silently diverges from Python on every multi-key
192
+ // object.
193
+ for (i, (k, v)) in map.iter().enumerate() {
194
+ if i > 0 {
195
+ out.push_str(", ");
196
+ }
197
+ out.push('\'');
198
+ out.push_str(k);
199
+ out.push('\'');
200
+ out.push_str(": ");
201
+ write_python_repr(out, v);
202
+ }
203
+ out.push('}');
204
+ }
205
+ }
206
+ }
207
+
208
+ /// Check if a JSON value matches any query anchors.
209
+ ///
210
+ /// Direct port of `item_matches_anchors` (Python `smart_crusher.py:152-168`).
211
+ /// Python uses `str(item).lower()` which produces Python's repr-like
212
+ /// representation. We mirror that via `python_repr` rather than
213
+ /// `serde_json::to_string` so substring matching has the same surface
214
+ /// as Python (single quotes, `True`/`False`/`None`, spaced commas/colons).
215
  pub fn item_matches_anchors(item: &Value, anchors: &HashSet<String>) -> bool {
216
  if anchors.is_empty() {
217
  return false;
218
  }
219
 
220
+ // Python: `str(item).lower()`. `python_repr` produces the same
221
+ // single-quoted, space-after-colon, `True`/`False`/`None` form
222
+ // that Python's `str()` does; lowercase normalizes the bool/null
223
+ // case to match Python's downstream `.lower()` call.
224
+ let item_str = python_repr(item).to_lowercase();
 
 
225
  anchors.iter().any(|a| item_str.contains(a))
226
  }
227
 
 
322
  let anchors: HashSet<String> = ["xyz123".to_string()].into_iter().collect();
323
  assert!(!item_matches_anchors(&json!({"a": "b"}), &anchors));
324
  }
325
+
326
+ #[test]
327
+ fn hostname_blocklist_drops_e_g() {
328
+ // S5 in code review: pin that "e.g" in input doesn't surface as
329
+ // an anchor. Direct match against the regex confirms "e.g" itself
330
+ // matches before the blocklist filters it.
331
+ let anchors = extract_query_anchors("see e.g for example");
332
+ assert!(!anchors.contains("e.g"));
333
+ // Sanity: a normal hostname still passes through.
334
+ let anchors = extract_query_anchors("connect to api.example.com");
335
+ assert!(anchors.contains("api.example.com"));
336
+ }
337
+
338
+ #[test]
339
+ fn email_typo_pattern_still_matches_real_emails() {
340
+ // S4 in code review: the Python `[A-Z|a-z]` typo doesn't break
341
+ // real email matching — pin that explicitly.
342
+ let anchors = extract_query_anchors("contact alice@example.com today");
343
+ assert!(anchors.contains("alice@example.com"));
344
+ let anchors = extract_query_anchors("ping bob@SUB.EXAMPLE.IO");
345
+ assert!(anchors.contains("bob@sub.example.io"));
346
+ }
347
+
348
+ // ---------- python_repr (used by item_matches_anchors) ----------
349
+
350
+ #[test]
351
+ fn python_repr_matches_python_str_for_dict() {
352
+ // Python: `str({'name': 'Alice', 'ok': True, 'count': 5, 'val': None})`
353
+ // = `"{'name': 'Alice', 'ok': True, 'count': 5, 'val': None}"`
354
+ // (insertion order — Python's dict preserves it since 3.7).
355
+ //
356
+ // Workspace `Cargo.toml` enables serde_json's `preserve_order`
357
+ // feature, so `json!` macro and `serde_json::from_str` both
358
+ // preserve key insertion order. Without that feature the test
359
+ // below would fail.
360
+ let v = json!({"name": "Alice", "ok": true, "count": 5, "val": null});
361
+ let r = python_repr(&v);
362
+ assert_eq!(
363
+ r,
364
+ "{'name': 'Alice', 'ok': True, 'count': 5, 'val': None}"
365
+ );
366
+ }
367
+
368
+ #[test]
369
+ fn python_repr_list_uses_space_after_comma() {
370
+ // Python: `str([1, 2, 'abc', True])` = `"[1, 2, 'abc', True]"`.
371
+ let v = json!([1, 2, "abc", true]);
372
+ assert_eq!(python_repr(&v), "[1, 2, 'abc', True]");
373
+ }
374
+
375
+ #[test]
376
+ fn python_repr_nested() {
377
+ let v = json!({"a": [1, {"b": "c"}]});
378
+ assert_eq!(python_repr(&v), "{'a': [1, {'b': 'c'}]}");
379
+ }
380
+
381
+ #[test]
382
+ fn item_matches_anchor_with_python_none_form() {
383
+ // I3 fix in review: Python `str({'val': None}).lower()` produces
384
+ // `{'val': none}`. With the old JSON-based matcher, the same
385
+ // input would serialize as `{"val":null}` and an anchor "none"
386
+ // would never match. With `python_repr` the serialization is
387
+ // `{'val': None}` → lowercased to `{'val': none}` → contains "none".
388
+ let anchors: HashSet<String> = ["none".to_string()].into_iter().collect();
389
+ assert!(item_matches_anchors(&json!({"val": null}), &anchors));
390
+ }
391
+
392
+ #[test]
393
+ fn item_matches_anchor_avoids_json_null_token() {
394
+ // Inverse of the above: an anchor "null" must NOT match a Python-
395
+ // null repr (which writes `none`). Pre-fix code would erroneously
396
+ // match because of `serde_json::to_string`'s `null` literal.
397
+ let anchors: HashSet<String> = ["null".to_string()].into_iter().collect();
398
+ assert!(!item_matches_anchors(&json!({"val": null}), &anchors));
399
+ }
400
+
401
+ #[test]
402
+ fn python_repr_string_with_single_quote_drift() {
403
+ // Documented parity gap: Python's `repr` switches to double
404
+ // quotes if the string contains a single quote. We always use
405
+ // single quotes. Pin the gap so future changes are intentional.
406
+ let v = json!({"k": "it's fine"});
407
+ // Our output: `{'k': 'it's fine'}` (broken Python repr — Python
408
+ // would emit `{'k': "it's fine"}`).
409
+ assert_eq!(python_repr(&v), "{'k': 'it's fine'}");
410
+ // Substring matching for typical anchors still works because
411
+ // they don't reference the quote chars themselves.
412
+ }
413
  }
crates/headroom-core/src/transforms/smart_crusher/classifier.rs CHANGED
@@ -173,10 +173,21 @@ mod tests {
173
 
174
  #[test]
175
  fn bool_with_number_is_mixed_not_bool_or_number() {
176
- // Python's `[True, False, 1]` matches `types <= {bool, int}` BUT
177
- // fails `all(isinstance(i, bool))`, so falls through to NUMBER_ARRAY
178
- // check which has `not has_bool` fails. Then nested check fails.
179
- // Final: MIXED_ARRAY. Same here.
 
 
 
 
 
 
 
 
 
 
 
180
  let items = vec![json!(true), json!(false), json!(1)];
181
  assert_eq!(classify_array(&items), ArrayType::MixedArray);
182
  }
 
173
 
174
  #[test]
175
  fn bool_with_number_is_mixed_not_bool_or_number() {
176
+ // Python's `[True, False, 1]` walks like this:
177
+ // types == {bool, int} (because bool is an int subclass)
178
+ // has_bool = True
179
+ // `types <= {bool, int}` is True, so the bool-array gate is
180
+ // considered, but the inner `all(isinstance(i, bool))` check
181
+ // is False (because of the `1`), so does NOT return BOOL_ARRAY.
182
+ // `types == {dict}` False. `types == {str}` False.
183
+ // `types <= {int, float} and not has_bool` — has_bool is True,
184
+ // so the number-array gate fails too. `types == {list}` False.
185
+ // Falls through to MIXED_ARRAY.
186
+ //
187
+ // Rust matches by side effect of separate `Bool`/`Number` enum
188
+ // variants: the bool-array gate fails because `has_number` is
189
+ // True; the number-array gate fails because `has_bool` is True.
190
+ // Final: MIXED_ARRAY. Same outcome via different code path.
191
  let items = vec![json!(true), json!(false), json!(1)];
192
  assert_eq!(classify_array(&items), ArrayType::MixedArray);
193
  }
crates/headroom-core/src/transforms/smart_crusher/hashing.rs CHANGED
@@ -1,23 +1,35 @@
1
  //! Field-name hashing for cache keys.
2
  //!
3
- //! Direct port of `_hash_field_name` (Python `smart_crusher.py:171-176`).
4
- //! Used to generate stable cache keys for compression hints must match
5
- //! Python byte-for-byte or cache lookups will miss.
 
 
 
 
 
 
 
 
 
 
 
6
 
7
  use sha2::{Digest, Sha256};
8
 
9
- /// SHA-256 of the UTF-8 bytes, hex-encoded, truncated to 16 chars.
10
  ///
11
- /// Python equivalent: `hashlib.sha256(field_name.encode()).hexdigest()[:16]`.
12
- /// We use lowercase hex (the default for both Python and Rust's `sha2`
13
- /// crate) the test below pins this.
14
  pub fn hash_field_name(field_name: &str) -> String {
15
  let mut hasher = Sha256::new();
16
  hasher.update(field_name.as_bytes());
17
  let digest = hasher.finalize();
18
- // Convert to lowercase hex, then truncate to first 16 chars (8 bytes).
 
19
  let hex = format!("{:x}", digest);
20
- hex[..16].to_string()
21
  }
22
 
23
  #[cfg(test)]
@@ -25,22 +37,22 @@ mod tests {
25
  use super::*;
26
 
27
  #[test]
28
- fn matches_python_sha256_truncated_to_16() {
29
- // Verified against Python: hashlib.sha256(b"customer_id").hexdigest()[:16]
30
- assert_eq!(hash_field_name("customer_id"), "1e38d67dbe8f47d2");
31
  }
32
 
33
  #[test]
34
  fn empty_string() {
35
- // Verified against Python: hashlib.sha256(b"").hexdigest()[:16]
36
- assert_eq!(hash_field_name(""), "e3b0c44298fc1c14");
37
  }
38
 
39
  #[test]
40
  fn unicode_field_name() {
41
- // Verified against Python: hashlib.sha256("café".encode()).hexdigest()[:16]
42
  // UTF-8 bytes for "café" are 63 61 66 c3 a9 — must encode same way.
43
- assert_eq!(hash_field_name("café"), "850f7dc43910ff89");
44
  }
45
 
46
  #[test]
@@ -50,9 +62,11 @@ mod tests {
50
  }
51
 
52
  #[test]
53
- fn output_length_is_16() {
54
- // Always exactly 16 hex chars regardless of input length.
55
- assert_eq!(hash_field_name("a").len(), 16);
56
- assert_eq!(hash_field_name(&"x".repeat(1000)).len(), 16);
 
 
57
  }
58
  }
 
1
  //! Field-name hashing for cache keys.
2
  //!
3
+ //! Direct port of `_hash_field_name` (Python `smart_crusher.py:171-177`).
4
+ //! Used to look up TOIN-anonymized `preserve_fields`TOIN stores
5
+ //! field names as **SHA-256[:8]** for privacy (per Python doc-comment
6
+ //! at `smart_crusher.py:174-175`), so cache lookups will silently miss
7
+ //! if the truncation length drifts.
8
+ //!
9
+ //! # 16 vs 8 — got it wrong once, now pinned
10
+ //!
11
+ //! The first version of this file used `[:16]` based on a misread of
12
+ //! the Python source. Code review caught the discrepancy: Python uses
13
+ //! `[:8]`. Cache lookups against TOIN's 8-char hashes would have
14
+ //! silently missed every field, defeating the entire `use_feedback_hints`
15
+ //! path. Fixed here; the tests now pin against Python `[:8]` reference
16
+ //! values verified via `python3 -c "...hexdigest()[:8]"`.
17
 
18
  use sha2::{Digest, Sha256};
19
 
20
+ /// SHA-256 of the UTF-8 bytes, hex-encoded, truncated to **8** chars.
21
  ///
22
+ /// Python equivalent: `hashlib.sha256(field_name.encode()).hexdigest()[:8]`.
23
+ /// Lowercase hex both Python `hexdigest()` and Rust's `sha2` default
24
+ /// to lowercase, so this is consistent without manual case-coercion.
25
  pub fn hash_field_name(field_name: &str) -> String {
26
  let mut hasher = Sha256::new();
27
  hasher.update(field_name.as_bytes());
28
  let digest = hasher.finalize();
29
+ // Truncate to first 8 hex chars (4 bytes of digest). MUST match
30
+ // Python's `[:8]` — see module-level note above.
31
  let hex = format!("{:x}", digest);
32
+ hex[..8].to_string()
33
  }
34
 
35
  #[cfg(test)]
 
37
  use super::*;
38
 
39
  #[test]
40
+ fn matches_python_sha256_truncated_to_8() {
41
+ // Verified against Python: hashlib.sha256(b"customer_id").hexdigest()[:8]
42
+ assert_eq!(hash_field_name("customer_id"), "1e38d67d");
43
  }
44
 
45
  #[test]
46
  fn empty_string() {
47
+ // Verified against Python: hashlib.sha256(b"").hexdigest()[:8]
48
+ assert_eq!(hash_field_name(""), "e3b0c442");
49
  }
50
 
51
  #[test]
52
  fn unicode_field_name() {
53
+ // Verified against Python: hashlib.sha256("café".encode()).hexdigest()[:8]
54
  // UTF-8 bytes for "café" are 63 61 66 c3 a9 — must encode same way.
55
+ assert_eq!(hash_field_name("café"), "850f7dc4");
56
  }
57
 
58
  #[test]
 
62
  }
63
 
64
  #[test]
65
+ fn output_length_is_8() {
66
+ // Always exactly 8 hex chars regardless of input length.
67
+ // This must match Python's `[:8]`; if you change it, every TOIN
68
+ // preserve-field lookup silently misses.
69
+ assert_eq!(hash_field_name("a").len(), 8);
70
+ assert_eq!(hash_field_name(&"x".repeat(1000)).len(), 8);
71
  }
72
  }
crates/headroom-core/src/transforms/smart_crusher/statistics.rs CHANGED
@@ -82,6 +82,53 @@ pub fn calculate_string_entropy(s: &str) -> f64 {
82
  }
83
  }
84
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
85
  /// Detect if numeric values form a sequential pattern (like IDs:
86
  /// 1, 2, 3, ...).
87
  ///
@@ -136,9 +183,10 @@ pub fn detect_sequential_pattern(values: &[Value], check_order: bool) -> bool {
136
  }
137
  Value::String(s) => {
138
  // Python: `try: nums.append(int(v))`. `int("3.14")` raises
139
- // in Python, so we mirror by trying integer parse first and
140
- // only succeeding for pure-integer strings.
141
- if let Ok(parsed) = s.parse::<i64>() {
 
142
  nums.push(parsed as f64);
143
  // BUG #2 fix: do NOT set had_non_string_numeric.
144
  // If we later find this is the ONLY source of numeric
@@ -372,4 +420,111 @@ mod tests {
372
  let v: Vec<Value> = (1..=10).map(|i| json!(i as f64)).collect();
373
  assert!(detect_sequential_pattern(&v, true));
374
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
375
  }
 
82
  }
83
  }
84
 
85
+ /// Parse a string the way Python's built-in `int()` does for plain
86
+ /// integer literals. Used by `detect_sequential_pattern` to mirror
87
+ /// `int(v)` behavior exactly.
88
+ ///
89
+ /// Python's `int()` accepts:
90
+ /// - leading/trailing ASCII whitespace (stripped)
91
+ /// - leading sign (`+` or `-`)
92
+ /// - PEP 515 underscore digit separators (e.g. `"3_000"` → `3000`)
93
+ ///
94
+ /// Rust's `str::parse::<i64>()` rejects all of those. If we used the
95
+ /// raw `parse`, real-world payloads with `" 5 "` or `"+5"` would
96
+ /// silently disagree with Python on whether the field is "numeric",
97
+ /// which changes sequential classification and breaks fixtures.
98
+ ///
99
+ /// We deliberately do NOT support Python's other `int()` features
100
+ /// (base prefixes like `"0x10"`, scientific notation via `int(float(s))`,
101
+ /// etc.) because the Python `_detect_sequential_pattern` call site
102
+ /// uses the default-base `int()` overload — those paths are
103
+ /// unreachable.
104
+ fn python_int_parse(s: &str) -> Option<i64> {
105
+ // Python: `int()` strips ASCII whitespace from both ends.
106
+ let trimmed = s.trim();
107
+ if trimmed.is_empty() {
108
+ return None;
109
+ }
110
+ // Python: drop PEP 515 underscores between digits. The implementation
111
+ // is more careful than this (rejects leading/trailing/double underscores),
112
+ // but for our use-case any string with valid digits + underscore separators
113
+ // is what we want to accept. Edge cases like `"_5_"` will fail the
114
+ // i64::parse call below, matching Python's behavior of rejecting them.
115
+ let cleaned: String = if trimmed.contains('_') {
116
+ // Reject patterns Python rejects: leading/trailing underscore,
117
+ // double underscores. Otherwise strip them out.
118
+ let bytes = trimmed.as_bytes();
119
+ let starts_or_ends = bytes[0] == b'_'
120
+ || *bytes.last().unwrap() == b'_'
121
+ || trimmed.contains("__");
122
+ if starts_or_ends {
123
+ return None;
124
+ }
125
+ trimmed.replace('_', "")
126
+ } else {
127
+ trimmed.to_string()
128
+ };
129
+ cleaned.parse::<i64>().ok()
130
+ }
131
+
132
  /// Detect if numeric values form a sequential pattern (like IDs:
133
  /// 1, 2, 3, ...).
134
  ///
 
183
  }
184
  Value::String(s) => {
185
  // Python: `try: nums.append(int(v))`. `int("3.14")` raises
186
+ // and Rust's plain `parse::<i64>` differs from `int()` on
187
+ // edges like leading whitespace and PEP 515 underscores.
188
+ // `python_int_parse` mirrors Python exactly — see fn doc.
189
+ if let Some(parsed) = python_int_parse(s) {
190
  nums.push(parsed as f64);
191
  // BUG #2 fix: do NOT set had_non_string_numeric.
192
  // If we later find this is the ONLY source of numeric
 
420
  let v: Vec<Value> = (1..=10).map(|i| json!(i as f64)).collect();
421
  assert!(detect_sequential_pattern(&v, true));
422
  }
423
+
424
+ #[test]
425
+ fn sequential_fractional_unit_step() {
426
+ // Floats with non-integer values but constant unit step. avg_diff
427
+ // = 1.0, all diffs in [0.5, 2.0], should be sequential. (Suggestion
428
+ // S6 in code review — pins float arithmetic doesn't drift.)
429
+ let v: Vec<Value> = vec![
430
+ json!(1.5),
431
+ json!(2.5),
432
+ json!(3.5),
433
+ json!(4.5),
434
+ json!(5.5),
435
+ ];
436
+ assert!(detect_sequential_pattern(&v, true));
437
+ }
438
+
439
+ #[test]
440
+ fn bug2_all_unparseable_strings_returns_false() {
441
+ // S3 in code review: explicit test for the all-strings case where
442
+ // none parse. Falls out of `nums.len() < 5` already, but pinning
443
+ // the behavior protects against future refactors.
444
+ let v: Vec<Value> = vec![
445
+ json!("abc"),
446
+ json!("def"),
447
+ json!("ghi"),
448
+ json!("jkl"),
449
+ json!("mno"),
450
+ ];
451
+ assert!(!detect_sequential_pattern(&v, true));
452
+ }
453
+
454
+ #[test]
455
+ fn bug2_single_int_among_strings_still_detects() {
456
+ // S3 in code review: validates that the BUG #2 gate fires on
457
+ // "ANY non-string numeric", not "majority". One real int among
458
+ // string-encoded numerics should be enough to count as sequential.
459
+ let v: Vec<Value> = vec![
460
+ json!("001"),
461
+ json!("002"),
462
+ json!(3), // <-- the unambiguous numeric
463
+ json!("004"),
464
+ json!("005"),
465
+ json!("006"),
466
+ ];
467
+ assert!(detect_sequential_pattern(&v, true));
468
+ }
469
+
470
+ // ---------- python_int_parse ----------
471
+
472
+ #[test]
473
+ fn python_int_parse_basic() {
474
+ assert_eq!(python_int_parse("5"), Some(5));
475
+ assert_eq!(python_int_parse("-5"), Some(-5));
476
+ assert_eq!(python_int_parse("+5"), Some(5));
477
+ }
478
+
479
+ #[test]
480
+ fn python_int_parse_strips_whitespace() {
481
+ // Python: `int(" 5 ") == 5`. Rust's plain parse fails on this.
482
+ assert_eq!(python_int_parse(" 5 "), Some(5));
483
+ assert_eq!(python_int_parse("\t-3\n"), Some(-3));
484
+ }
485
+
486
+ #[test]
487
+ fn python_int_parse_underscores() {
488
+ // PEP 515 — Python: `int("3_000") == 3000`.
489
+ assert_eq!(python_int_parse("3_000"), Some(3000));
490
+ assert_eq!(python_int_parse("1_000_000"), Some(1_000_000));
491
+ }
492
+
493
+ #[test]
494
+ fn python_int_parse_underscore_edge_cases_rejected() {
495
+ // Python rejects these (raises ValueError); we mirror by
496
+ // returning None.
497
+ assert_eq!(python_int_parse("_5"), None);
498
+ assert_eq!(python_int_parse("5_"), None);
499
+ assert_eq!(python_int_parse("3__000"), None);
500
+ }
501
+
502
+ #[test]
503
+ fn python_int_parse_rejects_floats() {
504
+ // Python: `int("3.14")` raises. Mirror by returning None.
505
+ assert_eq!(python_int_parse("3.14"), None);
506
+ }
507
+
508
+ #[test]
509
+ fn python_int_parse_rejects_non_numeric() {
510
+ assert_eq!(python_int_parse("abc"), None);
511
+ assert_eq!(python_int_parse(""), None);
512
+ assert_eq!(python_int_parse(" "), None);
513
+ }
514
+
515
+ #[test]
516
+ fn sequential_with_whitespace_padded_strings_via_python_int_parse() {
517
+ // I1 fix in code review: real fixtures may carry whitespace-padded
518
+ // numeric strings. With the python_int_parse helper, mixed real-int
519
+ // + whitespace-padded-string fields still detect correctly.
520
+ let v: Vec<Value> = vec![
521
+ json!(1),
522
+ json!(" 2 "),
523
+ json!(3),
524
+ json!(" 4 "),
525
+ json!(5),
526
+ json!(6),
527
+ ];
528
+ assert!(detect_sequential_pattern(&v, true));
529
+ }
530
  }
crates/headroom-core/src/transforms/smart_crusher/types.rs CHANGED
@@ -123,10 +123,29 @@ impl CrushabilityAnalysis {
123
  /// Complete analysis of an array.
124
  ///
125
  /// Mirrors `ArrayAnalysis` at `smart_crusher.py:887-897`. `field_stats`
126
- /// uses `BTreeMap` for deterministic iteration order (Python's `dict`
127
- /// preserves insertion order; `BTreeMap` gives us a stable sorted-by-key
128
- /// order, which is fine for the parity fixtures because the analyzer
129
- /// builds the map by iterating sorted keys).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
  #[derive(Debug, Clone)]
131
  pub struct ArrayAnalysis {
132
  pub item_count: usize,
 
123
  /// Complete analysis of an array.
124
  ///
125
  /// Mirrors `ArrayAnalysis` at `smart_crusher.py:887-897`. `field_stats`
126
+ /// and `constant_fields` use `BTreeMap` for sorted-by-key iteration.
127
+ ///
128
+ /// # Sort vs insertion order known parity nuance
129
+ ///
130
+ /// Python's `dict` preserves insertion order, and `_analyze_field` is
131
+ /// called once per key as it appears in `items[0].keys()` (i.e., JSON
132
+ /// parse order). With `serde_json/preserve_order` enabled at the
133
+ /// workspace level, `serde_json::Map` is an `IndexMap` and parse order
134
+ /// matches Python.
135
+ ///
136
+ /// `BTreeMap` here gives sorted-key iteration — which differs from
137
+ /// Python's parse-order `dict`. This matters only if downstream code
138
+ /// observes the iteration order of `field_stats` (e.g., when emitting
139
+ /// debug output, picking a "first" field, or computing strategy
140
+ /// strings that include field names).
141
+ ///
142
+ /// During the analyzer port (Stage 3c.1 commit 2), we'll either:
143
+ /// 1. Switch this to `IndexMap` if any code path observes order, OR
144
+ /// 2. Document that Python's order-sensitive paths get rewritten to
145
+ /// iterate sorted, then mirror that in Rust.
146
+ ///
147
+ /// Tracked in the design doc at
148
+ /// `~/Desktop/SmartCrusher-Architecture-Improvements.md`.
149
  #[derive(Debug, Clone)]
150
  pub struct ArrayAnalysis {
151
  pub item_count: usize,
plugins/headroom-agent-hooks/.claude-plugin/plugin.json CHANGED
@@ -1,6 +1,6 @@
1
  {
2
  "name": "headroom",
3
- "version": "0.10.17",
4
  "description": "Headroom startup hooks for Claude Code and GitHub Copilot CLI.",
5
  "author": {
6
  "name": "Headroom Contributors",
 
1
  {
2
  "name": "headroom",
3
+ "version": "0.11.0",
4
  "description": "Headroom startup hooks for Claude Code and GitHub Copilot CLI.",
5
  "author": {
6
  "name": "Headroom Contributors",
plugins/headroom-agent-hooks/.github/plugin/plugin.json CHANGED
@@ -1,6 +1,6 @@
1
  {
2
  "name": "headroom",
3
- "version": "0.10.17",
4
  "description": "Headroom startup hooks for Claude Code and GitHub Copilot CLI.",
5
  "author": {
6
  "name": "Headroom Contributors",
 
1
  {
2
  "name": "headroom",
3
+ "version": "0.11.0",
4
  "description": "Headroom startup hooks for Claude Code and GitHub Copilot CLI.",
5
  "author": {
6
  "name": "Headroom Contributors",