Visualizing Repo-Level Dedupe

This visualization demonstrates block-level deduplication across all models in bartowski/gemma-2-9b-it-GGUF.

Each row represents a file in the repository grouped into blocks of up to 64MB. The color of each block represents the dedupe factor for the block, which is a function of how many chunks in the block are present across all files in the repository. The darker the color, the more the block shares content. You can read more about chunks, blocks, and how the dedupe factor is calculated in this blog post.

To explore the visualization:

• Hover over a block in an individual file to highlight it and see where else it appears in the repository.
• Click any block in a file to see all other files that share blocks with it.
• Double-click anywhere on a file to reset and continue exploring.