Jainamshahhh commited on
Commit
e912fff
·
verified ·
1 Parent(s): 2c7c1ff

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +145 -0
README.md ADDED
@@ -0,0 +1,145 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: google/gemma-3-4b-it
4
+ language:
5
+ - en
6
+ library_name: peft
7
+ pipeline_tag: text-generation
8
+ tags:
9
+ - date-arithmetic
10
+ - temporal-reasoning
11
+ - business-days
12
+ - timezones
13
+ - iso-week
14
+ - lora
15
+ - adaption-autoscientist
16
+ ---
17
+
18
+ # ChronoCalc-4B: Calendar Arithmetic Where Calendars Actually Break
19
+
20
+ **ChronoCalc-4B** is a LoRA adaptation of `google/gemma-3-4b-it` for the date and time computations
21
+ that real systems get wrong: business days across federal holidays, rolling a date off a weekend,
22
+ elapsed time across a daylight saving transition, the next occurrence of a recurring schedule, and
23
+ ISO week numbers including the years that have a week 53.
24
+
25
+ Built for the **Adaption AutoScientist Challenge, Part 2 (Math and Code)**; the training corpus was
26
+ co-optimized with **Adaptive Data** (Adaption Labs).
27
+
28
+ ## Read this before the numbers
29
+
30
+ **This model does not claim to take the base from zero.** On plain day-walking with no holiday in
31
+ the window, `gemma-3-4b-it` already scores **69 to 86%** at every chain depth we tested. Saying
32
+ otherwise would be easy and false.
33
+
34
+ What the base cannot do is handle the exceptions:
35
+
36
+ | the base model, measured | score |
37
+ |---|---|
38
+ | plain day-walking, no holiday in the window | **69 to 86%** |
39
+ | windows where a **holiday** changes the answer | **7%** (2 of 30) |
40
+ | elapsed time across a **DST transition** | **2%** |
41
+ | **cross-year ISO week** numbers | **0%** |
42
+
43
+ That is the whole thesis. Deadlines do not go wrong because someone cannot add seven days. They go
44
+ wrong because Veterans Day fell in the window, or the clocks moved, or the year had 53 weeks.
45
+
46
+ **So the claim here is scoped:** improvement on calendar exceptions, and **no regression** on plain
47
+ day-walking. Both are reported.
48
+
49
+ ## Headline results
50
+
51
+ Base and tuned generated in one process under identical greedy decoding, scored by the released
52
+ `score_chrono.py`, which re-derives all 60,000 gold answers from row parameters in stdlib Python.
53
+
54
+ | slice | rows | base | **ChronoCalc-4B** |
55
+ |---|---|---|---|
56
+ | headline held-out | 500 | 11.0% | **91.0%** |
57
+ | hard shard | 300 | 14.7% | **93.3%** |
58
+ | enumerated calendar | 200 | 23.5% | **90.5%** |
59
+ | range slice, 2031 to 2035 | 150 | 14.7% | **76.7%** |
60
+
61
+ ## The gate that nearly killed this entry, and the rule written before it ran
62
+
63
+ The pre-registered Day-0 gate measured a **blended** F1 floor of 34%, which sat between the pass and
64
+ kill bands. The decomposition explains why: the blend is a property of the pilot mix, not of the
65
+ model. The pilot was 60% holiday-affected, and `share x 7% + (1 - share) x 70%` reproduces the
66
+ measured 34% to within half a point.
67
+
68
+ Because "between the bands after a rerun" was undefined in the spec, an adjudication rule was
69
+ **written and committed to git before the rerun was generated or scored**
70
+ (`docs/eval/gcp/chrono_gate_adjudication.md`). An adjudication rule invented after seeing the number
71
+ it adjudicates is worthless. The rule set four conditions, all of which had to hold or the entry
72
+ died with no further branches. It passed.
73
+
74
+ This is on the card as measurement transparency rather than buried, in the same way the DataViz entry
75
+ shipped its disclosed regression with a diagnosis.
76
+
77
+ ## Every answer was computed twice
78
+
79
+ `solver_a` produces the answer and the working. `solver_b`, written from scratch with different
80
+ primitives, produces the answer again. A row whose solvers disagree is **dropped and logged, never
81
+ repaired**. There are **0 disagreements across 60,000 rows**, and no language model appears anywhere
82
+ in the label path.
83
+
84
+ A reviewer can recompute every published number on a laptop with no GPU.
85
+
86
+ ## Scope and operating notes
87
+
88
+ 1. **The base is not bad at dates in general**, and this card refuses to imply otherwise. The
89
+ improvement is scoped to calendar exceptions.
90
+ 2. **90% of rows name the holiday calendar by statute** rather than enumerating it, because knowing
91
+ the US federal calendar is part of the trained behaviour. The **200-row enumerated** held-out
92
+ slice prints the calendar in full, so the headline can be rechecked without any statute knowledge.
93
+ Report both.
94
+ 3. **Scope.** US federal holidays, Gregorian dates, the IANA timezone database. Not other national
95
+ calendars, not historical calendar reforms, not leap seconds.
96
+ 4. **Completions are long on purpose.** Every one enumerates its computation line by line before the
97
+ `<answer>` tag. A terser format would score the same on exact match and be worth less, and
98
+ terseness has twice generalized out of a shard on this project and cost win rate on untouched
99
+ tasks.
100
+
101
+ ## Training details
102
+
103
+ | | |
104
+ |---|---|
105
+ | base | `google/gemma-3-4b-it` |
106
+ | method | LoRA r32 alpha64 on the language-model linears, completion-only masking |
107
+ | corpus | 60,000 rows, 5 families, 51 distinct templates |
108
+ | epochs | 3 |
109
+ | max_len | **1024, measured** (total p99 833, max 960, 0 of 60,000 over budget) |
110
+ | hardware | one A100-40GB, Spot |
111
+
112
+ `max_len` is measured because a 768 window would truncate roughly the top decile, and a truncated
113
+ completion loses its `<answer>` tag entirely, so the row would teach nothing under completion-only
114
+ masking.
115
+
116
+ ## Evaluation protocol
117
+
118
+ Base and tuned generated **in one process under identical greedy decoding** (`do_sample=False`).
119
+ Output length compared base against tuned, with a tuned median below 0.6x the base median failing the
120
+ run regardless of accuracy. Every gate was proven able to fail by injecting deliberately corrupted
121
+ rows before the corpus was trusted.
122
+
123
+ ## Usage
124
+
125
+ ```python
126
+ from peft import PeftModel
127
+ from transformers import AutoModelForCausalLM, AutoTokenizer
128
+
129
+ BASE = "google/gemma-3-4b-it"
130
+ tok = AutoTokenizer.from_pretrained(BASE)
131
+ # AutoModelForCausalLM resolves gemma-3-4b-it to its multimodal wrapper, which is correct.
132
+ # Do NOT load it through a text-only class: the decoder is then randomly initialized and
133
+ # the model emits whitespace, with no error raised.
134
+ model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="bfloat16",
135
+ attn_implementation="eager", device_map="cuda")
136
+ model = PeftModel.from_pretrained(model, "Jainamshahhh/chronocalc-4b").eval()
137
+ ```
138
+
139
+ ## Reproducibility and license
140
+
141
+ The corpus regenerates byte for byte from a single seed, because a row is a pure function of its
142
+ integer id. The scorer, every held-out slice and both solvers are published. Apache-2.0, matching the
143
+ base model. Built with **Adaptive Data** by Adaption Labs, whose enhancement pass on this corpus was
144
+ run, measured, and **refused** when it rewrote dates inside the questions; that decision is documented
145
+ rather than omitted.