| --- |
| license: mit |
| language: |
| - en |
| metrics: |
| - accuracy |
| - f1 |
| - precision |
| - recall |
| --- |
| |
| # CityPulse |
| [[Paper]](https://arxiv.org/abs/2401.01107) [[Citing]](https://github.com/tianyuanhuang/citypulse?tab=readme-ov-file#citing) [[Project Page]](https://tianyuanhuang.github.io/citypulse_web/) [[Github Repository]](https://github.com/tianyuanhuang/citypulse) |
| ## Model Summary |
| **CityPulse-DINOv2** is an urban structural change detection model finetuned based on the parameters of a Siamese [DINOv2 (ViT-B/14)](https://github.com/facebookresearch/dinov2) model. This model has been fine-tuned on the [CityPulse (Huang et al. 2024)](https://arxiv.org/abs/2401.01107) dataset. It achieves an accuracy of 88.85% in the urban building change detection task, which makes it suitable for fine-grained assessment of urban changes. The intention behind releasing this model is to provide the research community with more comprehensive and extensive resources, facilitate broader downstream research on urbans, and advance sustainable development goals. |
| |
| ## Intended Uses |
| Given the nature of the training data, the CityPulse model is best suited for detecting physical alterations in the built environment. |
| The model accepts two 224x224 RGB images as input. For urban change detection tasks, the input images should be taken at roughly the same location but at different times. The output of the model is a value ranging from 0 to 1, which can be interpreted as the model's inference of the probability of changes occurring in the buildings depicted in the images. |
| |
| ## Sample Code |
| The example Python code for inference using this model checkpoint is shown in the [Github Repo](https://github.com/tianyuanhuang/citypulse/blob/main/src/SiamDINOv2_inference.py). |
| |
| ## Limitations |
| Although has been tested on a dataset comprising over 12,700 image pairs, the model robustness remains unclear when faced with noisier images, more complex urban environments, and more diverse urban styles. |
| |
| ## Training |
| ### Model |
| * Architecture: a Siamese DINOv2 model with ViT-B as the backbone |
| |
| * Patch size: 14 |
| |
| * ViT feature dim: 768 |
| |
| ### Dataset |
| Our curated **CityPulse** dataset comprises 25,423 image pairs, including 4,465 images across five US cities, with a timespan exceeding 16 years. |
| |
| ### Optimize |
| * Optimizer: Adam |
| |
| * Learning rate: 1e-5 |
| |
| * Gradient clip: 0.5 |
| |
| * GPUs: 4 x Nvidia Tesla T4 |
| |
| ### Software |
| |
| * [PyTorch](https://github.com/pytorch/pytorch) |
| |
| * [Pytorch-Lightning](https://github.com/Lightning-AI/pytorch-lightning) |
| |
| ## Environment |
| Please refer to `citypulse.yaml` in the [Github Repository](https://github.com/tianyuanhuang/citypulse) for environment configuration. |
| |
| ## License |
| The model is licensed under the MIT license. |
| |
| ## Citing |
| |
| If you found this model checkpoint useful, please consider citing: |
| |
| ```bibtex |
| @article{huang2024citypulse, |
| title={CityPulse: Fine-Grained Assessment of Urban Change with Street View Time Series}, |
| author={Huang, Tianyuan and Wu, Zejia and Wu, Jiajun and Hwang, Jackelyn and Rajagopal, Ram}, |
| journal={arXiv preprint arXiv:2401.01107}, |
| year={2024} |
| } |
| ``` |
| |
