parakeet-realtime-eou-120m-v1-onnx

thoratsr7

ysdede commited on Jul 25

Commit

23ce103

0 Parent(s):

Duplicate from ysdede/parakeet-realtime-eou-120m-v1-onnx

Browse files

Co-authored-by: Yunus Dede <ysdede@users.noreply.huggingface.co>

Files changed (12) hide show

.gitattributes +4 -0
README.md +182 -0
config.json +14 -0
decoder_joint-model.fp16.onnx +3 -0
decoder_joint-model.int8.onnx +3 -0
decoder_joint-model.onnx +3 -0
encoder-model.fp16.onnx +3 -0
encoder-model.int8.onnx +3 -0
encoder-model.onnx +3 -0
figure-streaming.png +0 -0
vocab.txt +1026 -0
voice-agent.png +3 -0

.gitattributes ADDED Viewed

	@@ -0,0 +1,4 @@

+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.onnx.data filter=lfs diff=lfs merge=lfs -text
+*.data filter=lfs diff=lfs merge=lfs -text
+voice-agent.png filter=lfs diff=lfs merge=lfs -text

README.md ADDED Viewed

	@@ -0,0 +1,182 @@

+---
+license: other
+license_name: nvidia-open-model-license
+license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
+language:
+- en
+library_name: asrjs/speech-recognition
+pipeline_tag: automatic-speech-recognition
+tags:
+- automatic-speech-recognition
+- speech
+- audio
+- onnx
+- onnxruntime
+- asrjs
+- asrjs/speech-recognition
+- nemo
+- NeMo
+- FastConformer
+- RNNT
+- streaming-asr
+- eou
+- end-of-utterance
+- parakeet
+- voice-agent
+base_model: nvidia/parakeet_realtime_eou_120m-v1
+models:
+- nvidia/parakeet_realtime_eou_120m-v1
+widget:
+- example_title: Librispeech sample 1
+  src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
+- example_title: Librispeech sample 2
+  src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
+---
+# Parakeet Realtime EOU 120M v1 ONNX
+Converted ONNX package of [`nvidia/parakeet_realtime_eou_120m-v1`](https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1) for use with [`@asrjs/speech-recognition`](https://github.com/asrjs/speech-recognition).
+This repository is not the original NVIDIA training checkpoint repo. It contains exported runtime artifacts for browser and Node.js inference.
+## Included Artifacts
+- `encoder-model.onnx`
+- `decoder_joint-model.onnx`
+- `encoder-model.fp16.onnx`
+- `decoder_joint-model.fp16.onnx`
+- `encoder-model.int8.onnx`
+- `decoder_joint-model.int8.onnx`
+- `vocab.txt`
+- `config.json`
+## Model Summary
+Parakeet Realtime EOU 120M v1 is a streaming English ASR model with:
+- cache-aware FastConformer encoder
+- RNNT decoder
+- explicit `<EOU>` token emission for end-of-utterance detection
+- low-latency voice-agent-oriented output
+The model:
+- supports English only
+- does not emit punctuation or capitalization
+- may emit empty visible text for non-speech audio
+- keeps `<EOU>` in raw/native output while user-visible text should strip it
+Architecture details:
+- FastConformer-RNNT
+- 17 encoder layers
+- about 120M parameters
+## Frontend / Preprocessing
+The upstream model expects raw 16 kHz mono audio and uses a NeMo mel frontend internally.
+For `@asrjs/speech-recognition`, this ONNX package is intended to run with the shared in-repo JavaScript NeMo frontend. A dedicated `nemo80.onnx` or `nemo128.onnx` preprocessor is intentionally not required.
+Frontend assumptions:
+- sample rate: `16000`
+- mono audio
+- mel bins: `128`
+- valid length mode: `centered`
+- frontend output: raw log-mel features
+This matters: this model does **not** use the normalized `nemo128` frontend contract reused by some other NeMo exports.
+## Quantization Notes
+Included variants:
+- FP32
+- FP16
+- INT8 encoder
+- INT8 decoder
+Port validation summary on the smoke fixture:
+- FP32: exact token/text/raw-text parity
+- FP16: exact token/text/raw-text parity
+- decoder-only INT8: exact token/text/raw-text parity
+- encoder-only INT8: not exact
+- full `int8/int8`: not exact
+Recommended default pairings:
+- `fp32/fp32`
+- `fp16/fp16`
+- `fp32/int8` if you specifically want decoder-only INT8
+## Usage with `@asrjs/speech-recognition`
+### Preset usage
+```ts
+import { createSpeechPipeline, PcmAudioBuffer } from '@asrjs/speech-recognition';
+const pipeline = createSpeechPipeline({ cacheModels: true });
+const loaded = await pipeline.loadModel({
+  preset: 'parakeet',
+  modelId: 'nvidia/parakeet_realtime_eou_120m-v1',
+  backend: 'wasm',
+});
+const audio = PcmAudioBuffer.fromMono(pcmFloat32, 16000);
+const result = await loaded.transcribe(audio, {
+  detail: 'detailed',
+  responseFlavor: 'canonical+native',
+});
+console.log(result.canonical.text);
+console.log(result.native.rawUtteranceText);
+```
+### Direct source usage
+```ts
+const loaded = await pipeline.loadModel({
+  family: 'nemo-rnnt',
+  modelId: 'nvidia/parakeet_realtime_eou_120m-v1',
+  backend: 'wasm',
+  options: {
+    source: {
+      kind: 'huggingface',
+      repoId: 'ysdede/parakeet-realtime-eou-120m-v1-onnx',
+      preprocessorBackend: 'js',
+      encoderQuant: 'fp32',
+      decoderQuant: 'fp32',
+    },
+  },
+});
+```
+## Voice-Agent Context
+The original model card highlights voice-agent usage, especially streaming end-of-utterance detection.
+<div align="center">
+  <img src="./figure-streaming.png" width="450" />
+</div>
+<div align="center">
+  <img src="./voice-agent.png" width="750" />
+</div>
+## Upstream Model and License
+Original model:
+- [`nvidia/parakeet_realtime_eou_120m-v1`](https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1)
+This converted package follows the upstream **NVIDIA Open Model License** terms.
+## References
+- [Fast Conformer With Linearly Scalable Attention For Efficient Speech Recognition](https://arxiv.org/abs/2305.05084)
+- [Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition](https://arxiv.org/abs/2312.17279)
+- [NVIDIA NeMo](https://github.com/NVIDIA/NeMo)

config.json ADDED Viewed

	@@ -0,0 +1,14 @@

+{
+  "model_id": "nvidia/parakeet_realtime_eou_120m-v1",
+  "sample_rate": 16000,
+  "mel_bins": 128,
+  "frame_shift_seconds": 0.01,
+  "subsampling_factor": 8,
+  "vocab_size": 1026,
+  "blank_id": 1026,
+  "eou_id": 1024,
+  "eob_id": 1025,
+  "pred_hidden": 640,
+  "pred_layers": 1,
+  "max_symbols_per_step": 10
+}

decoder_joint-model.fp16.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e76d84f4f8e46a8ea38581964235850e500e16591f76619727d0ee61b71092a9
+size 10679601

decoder_joint-model.int8.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5464333fa933bf2c60952c08f01f5b5dd3fe3176e1c70d3eeddb48412514a0f9
+size 5368692

decoder_joint-model.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:50d355bc769420c506b2268ca3abe34f63dcaeb0ce0fdab80034248ea2a0ef88
+size 21348923

encoder-model.fp16.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:637ab309917f9ad3d3569f8294451750ebd1f0443a273a64dc31d1b4a3c8f429
+size 230121472

encoder-model.int8.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7c13202b4a1939e88f62054b7db02839d2fb18b8f7f694a0d4f224b3532e9465
+size 131479377

encoder-model.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:45c8e0b973fbd7e523308d058336bc1054ca05a66b215b5fa629ec17250bf7b8
+size 459170470

figure-streaming.png ADDED Viewed

vocab.txt ADDED Viewed

	@@ -0,0 +1,1026 @@

+<unk>
+▁t
+▁th
+▁a
+▁i
+▁the
+▁s
+re
+▁w
+▁o
+in
+at
+er
+nd
+ou
+▁c
+▁b
+▁h
+en
+on
+▁m
+▁f
+ing
+▁p
+▁to
+▁and
+▁d
+an
+or
+es
+▁y
+▁l
+▁of
+ll
+▁in
+ed
+it
+▁g
+is
+▁you
+▁n
+ar
+om
+as
+ve
+▁e
+ic
+▁it
+al
+us
+▁wh
+▁we
+▁be
+ion
+ow
+le
+▁is
+et
+ent
+ot
+ut
+▁re
+▁on
+ay
+▁ha
+ig
+▁so
+ct
+▁he
+▁for
+ver
+ke
+ro
+▁st
+id
+▁go
+all
+se
+ly
+▁u
+ch
+st
+ld
+▁k
+ce
+ur
+▁li
+am
+▁r
+ht
+▁j
+ith
+▁se
+ir
+▁as
+▁an
+im
+▁do
+ad
+▁was
+ight
+th
+▁are
+▁but
+▁sh
+ust
+ally
+▁not
+▁or
+▁com
+▁can
+▁me
+op
+▁mo
+▁at
+ill
+▁ch
+▁ne
+ant
+▁de
+▁kn
+▁one
+il
+ol
+▁con
+ter
+▁ab
+▁fr
+ere
+ck
+▁al
+▁all
+qu
+▁pro
+▁som
+ould
+▁tw
+ul
+ra
+od
+ers
+▁su
+ive
+▁v
+use
+ate
+ge
+if
+▁ex
+ess
+pp
+▁lo
+out
+▁if
+est
+ain
+ist
+and
+ea
+very
+art
+▁wor
+▁my
+ab
+ment
+▁bec
+un
+ity
+ri
+pe
+ions
+▁by
+ok
+our
+ort
+ind
+ink
+nt
+▁up
+um
+▁don
+▁get
+red
+▁out
+el
+ause
+res
+▁ma
+ich
+▁us
+rou
+▁int
+em
+os
+ies
+ie
+▁pl
+▁tr
+ven
+ous
+▁le
+▁two
+ard
+ine
+▁co
+een
+▁now
+ty
+her
+ack
+▁pe
+ame
+▁how
+▁who
+▁see
+▁tim
+ect
+ast
+▁our
+ci
+ree
+ople
+gh
+▁no
+▁had
+▁man
+▁qu
+▁en
+ide
+ure
+ud
+so
+▁his
+▁sa
+▁sp
+▁say
+ose
+ther
+▁act
+▁ta
+▁cl
+ings
+pt
+king
+▁any
+▁has
+▁un
+iv
+▁im
+▁ag
+▁te
+▁fe
+one
+per
+ong
+▁po
+▁ad
+ff
+ore
+itt
+ans
+iz
+eah
+reat
+act
+own
+hing
+enty
+age
+ber
+ice
+▁am
+ple
+are
+▁per
+und
+ite
+ix
+pl
+▁way
+▁did
+▁pr
+▁got
+ars
+▁she
+▁let
+ag
+▁ac
+int
+▁ar
+ry
+ign
+ish
+▁fir
+ace
+ble
+og
+ue
+▁ye
+ap
+iff
+▁ro
+▁her
+nder
+▁ok
+▁res
+▁gu
+ence
+▁may
+ated
+ip
+▁bo
+▁him
+way
+ac
+ical
+ass
+ase
+▁dis
+able
+ick
+▁app
+ance
+▁pre
+▁six
+▁off
+▁new
+ia
+orm
+ank
+▁lot
+ach
+▁fo
+inet
+ire
+ary
+ult
+▁tal
+▁mu
+▁bl
+ount
+sel
+vel
+▁br
+▁imp
+ep
+cess
+ord
+▁sc
+▁inc
+ound
+ang
+be
+ress
+uct
+▁ind
+▁af
+ving
+▁oh
+▁bet
+▁use
+ome
+ens
+ys
+▁bu
+co
+ory
+ater
+ild
+ght
+ial
+▁day
+ning
+na
+ile
+▁spe
+▁mar
+ody
+ough
+ade
+vers
+xt
+▁fl
+▁ke
+ian
+▁sy
+▁put
+fore
+ub
+▁ph
+fe
+▁em
+▁ser
+form
+ting
+te
+av
+ious
+▁rec
+ks
+▁gr
+ces
+wn
+ors
+▁jo
+ents
+▁des
+▁try
+▁equ
+▁z
+▁rem
+▁str
+self
+▁bit
+ph
+ved
+▁why
+▁bas
+▁hel
+▁rel
+ath
+ject
+ail
+▁la
+ual
+▁god
+▁nat
+erm
+day
+▁id
+ft
+▁wr
+▁min
+ates
+▁gen
+tain
+▁ob
+ull
+ict
+▁tra
+▁end
+▁hig
+▁fif
+oth
+tern
+▁its
+vent
+▁sm
+ons
+▁add
+iss
+▁bel
+ful
+get
+▁ele
+▁rep
+ak
+▁ho
+▁pos
+▁num
+ange
+ves
+ific
+urn
+ise
+▁cr
+▁um
+ward
+▁reg
+ady
+ower
+uc
+▁dec
+lic
+▁set
+▁gon
+▁op
+▁ear
+▁sub
+▁sl
+les
+stem
+cial
+olog
+atch
+ily
+body
+nds
+ular
+ren
+▁own
+▁too
+cent
+ible
+pect
+ered
+ways
+teen
+▁uh
+▁big
+▁mod
+▁att
+▁car
+gr
+▁acc
+ied
+mun
+ib
+▁mon
+▁sch
+▁pol
+▁dat
+▁fin
+▁sim
+▁inv
+▁def
+ked
+▁ent
+▁yes
+ows
+ics
+ited
+ute
+ism
+ps
+▁ed
+▁el
+ably
+ppen
+als
+▁ten
+ract
+ss
+▁ass
+▁met
+gan
+▁eng
+▁stu
+ween
+arch
+▁gl
+▁cor
+▁dr
+vern
+▁ty
+▁run
+hip
+cus
+cond
+▁ins
+irty
+▁pub
+lud
+llow
+▁cou
+ew
+iew
+▁sur
+ero
+ood
+ness
+▁fun
+▁eff
+cept
+▁ca
+▁exp
+duct
+▁sw
+ize
+ope
+▁par
+kes
+cy
+▁ev
+▁ref
+ell
+▁bus
+ug
+rib
+▁cur
+mo
+ock
+ures
+air
+▁war
+str
+▁med
+▁wa
+▁val
+▁sin
+blem
+▁fam
+li
+▁far
+▁cle
+▁col
+mon
+▁gra
+led
+ense
+tin
+ues
+its
+▁mem
+▁inf
+▁eas
+ideo
+▁top
+io
+pan
+▁hum
+▁old
+ead
+▁ord
+ric
+ants
+oy
+esn
+uck
+ason
+ced
+ool
+rat
+ouse
+▁lar
+▁art
+▁wee
+▁cer
+ized
+▁mat
+con
+erg
+land
+ines
+▁chr
+▁aut
+▁lea
+▁sou
+oney
+tty
+▁ple
+ulat
+oks
+▁few
+▁sol
+▁che
+chn
+ird
+▁bre
+▁dur
+▁wom
+me
+izat
+eric
+ote
+▁uni
+eren
+arn
+ross
+ices
+ten
+eral
+ever
+ieve
+lish
+ash
+▁opp
+alth
+ger
+▁sk
+▁red
+peri
+▁det
+▁ext
+ner
+ah
+▁var
+▁loc
+gram
+ists
+ives
+▁es
+▁nor
+tro
+ale
+▁iss
+▁pri
+gin
+az
+oc
+▁pop
+ern
+▁sit
+ket
+▁pa
+▁law
+ages
+br
+▁cam
+▁mom
+osed
+▁bro
+ne
+bs
+▁cre
+erat
+▁sec
+▁cap
+▁vis
+▁pat
+ield
+iet
+▁tri
+up
+▁bra
+ts
+▁mot
+▁unt
+put
+bo
+ork
+mer
+ital
+▁air
+ined
+▁beh
+▁adv
+▁ret
+imes
+▁tea
+ural
+sid
+ters
+▁pur
+▁sci
+bers
+ient
+ier
+cc
+sw
+▁av
+reen
+ode
+ont
+▁dra
+ann
+nect
+▁x
+▁eu
+ton
+inat
+ene
+ared
+els
+▁mor
+▁rat
+cri
+▁men
+▁ah
+ames
+▁arm
+eak
+▁pay
+▁hal
+ins
+ilit
+stit
+▁ra
+▁leg
+cl
+pr
+▁wal
+▁bad
+▁ge
+roup
+▁mus
+man
+▁gi
+eds
+▁aw
+po
+ark
+row
+▁dep
+ully
+ral
+lect
+pend
+▁sev
+ime
+gest
+here
+▁yet
+ted
+▁rev
+ds
+▁ask
+less
+▁di
+ets
+line
+▁aff
+ired
+▁est
+ken
+vid
+most
+ivid
+unch
+par
+med
+rop
+ased
+eone
+▁ve
+▁abs
+ergy
+ret
+▁saw
+▁ey
+▁cal
+uat
+▁mid
+vat
+ream
+vice
+ians
+rent
+ctor
+err
+ush
+ases
+▁suc
+erms
+ave
+angu
+ries
+▁wo
+arts
+▁fil
+▁fat
+▁cho
+orts
+▁fre
+ee
+ught
+eng
+ump
+▁bar
+ying
+ane
+▁tem
+anks
+ury
+iat
+mit
+trol
+▁net
+▁maj
+▁cra
+ling
+▁fig
+orn
+icat
+pany
+▁occ
+ott
+ands
+▁exc
+▁mr
+ency
+rope
+itch
+▁lit
+abil
+not
+ma
+▁typ
+▁opt
+ob
+ser
+ety
+ms
+peci
+aces
+aut
+▁hon
+cuss
+▁sal
+▁sor
+att
+▁lab
+▁har
+urch
+nded
+uce
+ids
+▁hy
+▁fut
+▁ste
+ours
+ems
+utes
+ng
+ta
+▁won
+▁fa
+▁env
+ards
+▁job
+ium
+▁dot
+▁obv
+ina
+side
+elve
+cu
+▁jes
+▁pot
+▁pie
+▁tre
+▁hey
+▁mag
+ron
+▁key
+swer
+▁win
+ucat
+work
+ides
+▁low
+▁vol
+▁oth
+atic
+lf
+ads
+inds
+com
+ths
+▁ver
+ised
+lo
+▁squ
+▁cut
+oked
+irit
+ateg
+ppy
+mitt
+come
+hn
+igin
+mand
+▁dam
+ho
+▁da
+▁fur
+iron
+ilar
+▁fac
+▁neg
+▁ago
+ged
+miss
+enth
+▁dou
+▁hit
+▁guy
+▁bi
+ove
+fess
+ples
+owed
+ured
+▁ris
+ints
+rew
+▁sum
+▁hu
+ploy
+ude
+ried
+▁cir
+▁dev
+ear
+▁tot
+▁ann
+duc
+ik
+pon
+sted
+▁ide
+▁'
+ipp
+▁eat
+▁dom
+▁
+e
+t
+o
+a
+i
+n
+s
+r
+h
+l
+d
+u
+c
+m
+y
+g
+w
+f
+p
+b
+v
+k
+'
+j
+x
+q
+z
+<EOU>
+<EOB>

voice-agent.png ADDED Viewed

Git LFS Details

SHA256: 91ba46c05bf6a211b340a361e1f0c853210b008975743942796c08203c1a16f3
Pointer size: 131 Bytes
Size of remote file: 160 kB