# Differentiable Conformal Factuality

A fully differentiable approximation of Coherent Factuality for LLM outputs with reasoning structure. This framework enables gradient-based optimization of claim scoring functions while preserving conformal prediction guarantees.

## Overview

Large language models can generate factually incorrect claims. **Conformal Factuality** provides statistical guarantees on which claims to retain, but the original formulation uses hard (non-differentiable) thresholding. This project introduces a **differentiable relaxation** that allows end-to-end training of claim scoring models via gradient descent, while maintaining valid conformal coverage.

Key components:
- **Differentiable calibration** with soft quantiles (via `torchsort`)
- **Ancestor coherence** enforcement on reasoning graphs
- **Soft threshold selection** using a differentiable supremum approximation
- **Conformal prediction** with coverage guarantees at any user-specified error rate

## Installation

```bash
# Clone the repository
git clone https://github.com/NathanHitt/Differentiable_Conformal_Factuality.git
cd Differentiable_Conformal_Factuality

# Install dependencies (Python 3.10+)
pip install -r requirements.txt
```

For data generation scripts that use the OpenAI API, copy the environment template and add your keys:
```bash
cp .env.example .env
# Edit .env with your API keys
```

## Project Structure

```
├── src/                          # Core library
│   ├── differentiable_conformal_factuality.py  # Main differentiable pipeline
│   ├── models.py                 # Claim scoring models (LogisticClaimScorer, etc.)
│   ├── training.py               # Training pipeline with cross-validation
│   ├── utilities.py              # Helper functions
│   ├── reasonining_graph_dataset.py  # Dataset loading and processing
│   ├── debugger.py               # Debugging utilities
│   ├── hard/                     # Hard (non-differentiable) baseline implementation
│   ├── optimization/             # Hyperparameter optimization
│   └── ablation/                 # Ablation study methods
├── experiments/                  # Experiment scripts and analysis
├── scripts/                      # Data generation and preprocessing
├── config/                       # Configuration files
├── data/                         # Datasets
├── tests/                        # Test suite
├── prompts/                      # LLM prompt templates
├── figures/                      # Paper figures
└── results/                      # Experiment results
```

## Quick Start

```python
from src.differentiable_conformal_factuality import calibrate, predict, compute_soft_retention_loss
from src.models import LogisticClaimScorer

# Initialize scorer
scorer = LogisticClaimScorer(input_dim=num_features)

# Calibrate on held-out data
tau = calibrate(X_cal, Y_cal, noise_cal, scorer, alpha=0.1)

# Predict on test data
probs = predict(X_test, noise_test, scorer, tau)

# Compute loss for training
loss = compute_soft_retention_loss(probs, Y_test)
loss.backward()  # Fully differentiable!
```

## Datasets

This project supports evaluation on:
- **MATH** — mathematical reasoning claims
- **FELM** — factual error detection in LLM outputs
- **FinQA** — financial question answering

## Citation

If you use this code in your research, please cite:

```bibtex
@article{hitt2025differentiable,
  title={Differentiable Conformal Factuality},
  author={Hitt, Nathan},
  year={2025}
}
```

## License

MIT License