Add thresholding to inference to reduce false positive rate

debug

debug

debug

debug

Change some codes

Reduce max sequence length to 250 as PhoBERT allows maximum 256 tokens

First commit

Temporary fix

Temporary fix

Allow LSTM loading on CPU

Remove debugging code

Use BERT vocab size

test

test

Simplify distilling and inference

Simplify distilling

Fix LSTM tokenizer mismatch

Found a critical bug in LSTM inference when distillation using BERT Tokenizer but inference uses custom LSTM tokenizer

debug

Update inference_lstm

test

Remove debug

debug

debug

debug

debug

test

test

test

Fix a bug

Use the original training tokenizer instead of creating tokenizer from scratch

Test

Debug tokenizer import

Debug tokenizer import

Debug tokenizer import

Update tokenizer class

Test

Debug

Debug

Idk what error

Edit mismatching vocab size leading for mismatching embedding size for training and inference

Edit error parameters

Save vocab size with model_state_dict

Edit default mismatch in LSTM training and inference

Edit some saving and loading details