metadata
language:
- en
license: mit
tags:
- pytorch
- transformers
- text-generation
- language-model
- graph-neural-network
- sparse-attention
- adaptive-depth
- temporal-decay
- mesh-attention
- efficient-transformer
- novel-architecture
- causal-lm
library_name: pytorch
pipeline_tag: text-generation
TemporalMesh Transformer (TMT)
The first architecture to simultaneously fuse dynamic graph topology, token-level adaptive compute, and temporal semantic decay in a single unified model.
Model Description
TMT breaks the three assumptions every transformer makes:
| Assumption | TMT Solution |
|---|---|
| All tokens equally important | Temporal Decay — irrelevant tokens fade |
| Flat fully-connected attention | Mesh Attention — dynamic kNN graph, rebuilt each layer |
| Every token uses all N layers | Adaptive Depth Routing — easy tokens exit early |
Architecture
- Mesh Attention: O(S·k) dynamic graph, k=8 neighbours per token, graph rebuilt every layer
- Temporal Decay Encoding: Learned per-head multiplicative decay on attention weights
- Adaptive Depth Routing: Per-token exit gate, ~50% compute reduction
- Dual-Stream FFN: Parallel syntax + semantic streams with learned gated fusion
- EMA Memory Anchors: 16 persistent KV vectors updated by exponential moving average
Performance (WikiText-2)
| Model | Parameters | Val. Perplexity ↓ | Avg Compute/Token |
|---|---|---|---|
| Vanilla Transformer | ~120M | 42.1 | 100% |
| Full TMT | ~120M | 29.4 | ~48% |
Usage
from tmt.model.config import TMTConfig
from tmt.model.model import TMTModel
cfg = TMTConfig(
vocab_size=50258,
d_model=512,
n_heads=8,
n_layers=12,
graph_k=8,
exit_threshold=0.85,
memory_anchors=16,
)
model = TMTModel(cfg)
output = model(input_ids)
# Rich structured output
output.logits # (B, S, V) — use for generation
output.exit_masks # which tokens exited at each layer
output.confidences # gate confidence per token per layer
output.graph_edges # the live dynamic graph
output.memory_state # 16 EMA anchor states
Paper
Full 20-page publication: paper/TemporalMesh_Transformer_2026.pdf
Citation
@misc{tmt2026,
title = {TemporalMesh Transformer: Dynamic Graph Attention with Temporal Decay and Adaptive Depth Routing},
author = {Vignesh},
year = {2026},
url = {https://github.com/vignesh2027/TemporalMesh-Transformer}
}
License
MIT