Resnet50 Test Quantization for accelerated int8-int8 compute for Apple Neural Engine
Resnet From Code:
Int8ANE.ipynb - test notebook to create
CoreMLTools 8.01b, M4 iPad Pro 16GB, iPad OS18.1 beta
LUT 4-bit FP16 1.03ms
A4W8 0.58 ms
LUT 4-bit (A8W8) 0.92ms - no Accell!
Tourch Model:
8-bit
resnet50-LUT8-iOS17.mlpackage 1.08 ms
resnet50-W8A8-iOS17.mlpackage 0.81 ms
4-bit
resnet50-LUT4-iOS17.mlpackage 0.93 ms
resnet50-W4A8-iOS18.mlpackage 0.68 ms