File size: 6,607 Bytes
e912fff | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | ---
license: apache-2.0
base_model: google/gemma-3-4b-it
language:
- en
library_name: peft
pipeline_tag: text-generation
tags:
- date-arithmetic
- temporal-reasoning
- business-days
- timezones
- iso-week
- lora
- adaption-autoscientist
---
# ChronoCalc-4B: Calendar Arithmetic Where Calendars Actually Break
**ChronoCalc-4B** is a LoRA adaptation of `google/gemma-3-4b-it` for the date and time computations
that real systems get wrong: business days across federal holidays, rolling a date off a weekend,
elapsed time across a daylight saving transition, the next occurrence of a recurring schedule, and
ISO week numbers including the years that have a week 53.
Built for the **Adaption AutoScientist Challenge, Part 2 (Math and Code)**; the training corpus was
co-optimized with **Adaptive Data** (Adaption Labs).
## Read this before the numbers
**This model does not claim to take the base from zero.** On plain day-walking with no holiday in
the window, `gemma-3-4b-it` already scores **69 to 86%** at every chain depth we tested. Saying
otherwise would be easy and false.
What the base cannot do is handle the exceptions:
| the base model, measured | score |
|---|---|
| plain day-walking, no holiday in the window | **69 to 86%** |
| windows where a **holiday** changes the answer | **7%** (2 of 30) |
| elapsed time across a **DST transition** | **2%** |
| **cross-year ISO week** numbers | **0%** |
That is the whole thesis. Deadlines do not go wrong because someone cannot add seven days. They go
wrong because Veterans Day fell in the window, or the clocks moved, or the year had 53 weeks.
**So the claim here is scoped:** improvement on calendar exceptions, and **no regression** on plain
day-walking. Both are reported.
## Headline results
Base and tuned generated in one process under identical greedy decoding, scored by the released
`score_chrono.py`, which re-derives all 60,000 gold answers from row parameters in stdlib Python.
| slice | rows | base | **ChronoCalc-4B** |
|---|---|---|---|
| headline held-out | 500 | 11.0% | **91.0%** |
| hard shard | 300 | 14.7% | **93.3%** |
| enumerated calendar | 200 | 23.5% | **90.5%** |
| range slice, 2031 to 2035 | 150 | 14.7% | **76.7%** |
## The gate that nearly killed this entry, and the rule written before it ran
The pre-registered Day-0 gate measured a **blended** F1 floor of 34%, which sat between the pass and
kill bands. The decomposition explains why: the blend is a property of the pilot mix, not of the
model. The pilot was 60% holiday-affected, and `share x 7% + (1 - share) x 70%` reproduces the
measured 34% to within half a point.
Because "between the bands after a rerun" was undefined in the spec, an adjudication rule was
**written and committed to git before the rerun was generated or scored**
(`docs/eval/gcp/chrono_gate_adjudication.md`). An adjudication rule invented after seeing the number
it adjudicates is worthless. The rule set four conditions, all of which had to hold or the entry
died with no further branches. It passed.
This is on the card as measurement transparency rather than buried, in the same way the DataViz entry
shipped its disclosed regression with a diagnosis.
## Every answer was computed twice
`solver_a` produces the answer and the working. `solver_b`, written from scratch with different
primitives, produces the answer again. A row whose solvers disagree is **dropped and logged, never
repaired**. There are **0 disagreements across 60,000 rows**, and no language model appears anywhere
in the label path.
A reviewer can recompute every published number on a laptop with no GPU.
## Scope and operating notes
1. **The base is not bad at dates in general**, and this card refuses to imply otherwise. The
improvement is scoped to calendar exceptions.
2. **90% of rows name the holiday calendar by statute** rather than enumerating it, because knowing
the US federal calendar is part of the trained behaviour. The **200-row enumerated** held-out
slice prints the calendar in full, so the headline can be rechecked without any statute knowledge.
Report both.
3. **Scope.** US federal holidays, Gregorian dates, the IANA timezone database. Not other national
calendars, not historical calendar reforms, not leap seconds.
4. **Completions are long on purpose.** Every one enumerates its computation line by line before the
`<answer>` tag. A terser format would score the same on exact match and be worth less, and
terseness has twice generalized out of a shard on this project and cost win rate on untouched
tasks.
## Training details
| | |
|---|---|
| base | `google/gemma-3-4b-it` |
| method | LoRA r32 alpha64 on the language-model linears, completion-only masking |
| corpus | 60,000 rows, 5 families, 51 distinct templates |
| epochs | 3 |
| max_len | **1024, measured** (total p99 833, max 960, 0 of 60,000 over budget) |
| hardware | one A100-40GB, Spot |
`max_len` is measured because a 768 window would truncate roughly the top decile, and a truncated
completion loses its `<answer>` tag entirely, so the row would teach nothing under completion-only
masking.
## Evaluation protocol
Base and tuned generated **in one process under identical greedy decoding** (`do_sample=False`).
Output length compared base against tuned, with a tuned median below 0.6x the base median failing the
run regardless of accuracy. Every gate was proven able to fail by injecting deliberately corrupted
rows before the corpus was trusted.
## Usage
```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
BASE = "google/gemma-3-4b-it"
tok = AutoTokenizer.from_pretrained(BASE)
# AutoModelForCausalLM resolves gemma-3-4b-it to its multimodal wrapper, which is correct.
# Do NOT load it through a text-only class: the decoder is then randomly initialized and
# the model emits whitespace, with no error raised.
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="bfloat16",
attn_implementation="eager", device_map="cuda")
model = PeftModel.from_pretrained(model, "Jainamshahhh/chronocalc-4b").eval()
```
## Reproducibility and license
The corpus regenerates byte for byte from a single seed, because a row is a pure function of its
integer id. The scorer, every held-out slice and both solvers are published. Apache-2.0, matching the
base model. Built with **Adaptive Data** by Adaption Labs, whose enhancement pass on this corpus was
run, measured, and **refused** when it rewrote dates inside the questions; that decision is documented
rather than omitted.
|