Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pocketsphinx_continuous is unable to convert speech to text #199

Closed
ghost opened this issue Mar 12, 2020 · 12 comments · Fixed by #303
Closed

pocketsphinx_continuous is unable to convert speech to text #199

ghost opened this issue Mar 12, 2020 · 12 comments · Fixed by #303
Milestone
5.0.0

Comments

@ghost
Copy link

ghost commented Mar 12, 2020 •
edited by nshmyrev
Loading

Hi All,
I recently compiled the pocket sphinx on MIPS architecture and then I recorded a 16 bit 16000 Hz mono audio and then I tried to run pocketsphinx_continuous using the following command
pocketsphinx_continuous -hmm en-us/ -lm TAR9897/9897.lm -dict TAR9897/9897.dic -infile input.wav

But I am unable to get any speech from the audio but when I tried to do the same in my laptop it was working. Please help

I am sharing the logs below

INFO: pocketsphinx.c(153): Parsed model-specific feature parameters from en-us//feat.params
Current configuration:
[NAME]			[DEFLT]		[VALUE]
-agc			none		none
-agcthresh		2.0		2.000000e+00
-allphone				
-allphone_ci		yes		yes
-alpha			0.97		9.700000e-01
-ascale			20.0		2.000000e+01
-aw			1		1
-backtrace		no		no
-beam			1e-48		1.000000e-48
-bestpath		yes		yes
-bestpathlw		9.5		9.500000e+00
-ceplen			13		13
-cmn			live		batch
-cmninit		40,3,-1		41.00,-5.29,-0.12,5.09,2.48,-4.07,-1.37,-1.78,-5.08,-2.05,-6.45,-1.42,1.17
-compallsen		no		no
-dict					TAR9897/9897.dic
-dictcase		no		no
-dither			no		no
-doublebw		no		no
-ds			1		1
-fdict					
-feat			1s_c_d_dd	1s_c_d_dd
-featparams				
-fillprob		1e-8		1.000000e-08
-frate			100		100
-fsg					
-fsgusealtpron		yes		yes
-fsgusefiller		yes		yes
-fwdflat		yes		yes
-fwdflatbeam		1e-64		1.000000e-64
-fwdflatefwid		4		4
-fwdflatlw		8.5		8.500000e+00
-fwdflatsfwin		25		25
-fwdflatwbeam		7e-29		7.000000e-29
-fwdtree		yes		yes
-hmm					en-us/
-input_endian		little		little
-jsgf					
-keyphrase				
-kws					
-kws_delay		10		10
-kws_plp		1e-1		1.000000e-01
-kws_threshold		1e-30		1.000000e-30
-latsize		5000		5000
-lda					
-ldadim			0		0
-lifter			0		22
-lm					TAR9897/9897.lm
-lmctl					
-lmname					
-logbase		1.0001		1.000100e+00
-logfn					
-logspec		no		no
-lowerf			133.33334	1.300000e+02
-lpbeam			1e-40		1.000000e-40
-lponlybeam		7e-29		7.000000e-29
-lw			6.5		6.500000e+00
-maxhmmpf		30000		30000
-maxwpf			-1		-1
-mdef					
-mean					
-mfclogdir				
-min_endfr		0		0
-mixw					
-mixwfloor		0.0000001	1.000000e-07
-mllr					
-mmap			yes		yes
-ncep			13		13
-nfft			512		512
-nfilt			40		25
-nwpen			1.0		1.000000e+00
-pbeam			1e-48		1.000000e-48
-pip			1.0		1.000000e+00
-pl_beam		1e-10		1.000000e-10
-pl_pbeam		1e-10		1.000000e-10
-pl_pip			1.0		1.000000e+00
-pl_weight		3.0		3.000000e+00
-pl_window		5		5
-rawlogdir				
-remove_dc		no		no
-remove_noise		yes		yes
-remove_silence		yes		yes
-round_filters		yes		yes
-samprate		16000		1.600000e+04
-seed			-1		-1
-sendump				
-senlogdir				
-senmgau				
-silprob		0.005		5.000000e-03
-smoothspec		no		no
-svspec					0-12/13-25/26-38
-tmat					
-tmatfloor		0.0001		1.000000e-04
-topn			4		4
-topn_beam		0		0
-toprule				
-transform		legacy		dct
-unit_area		yes		yes
-upperf			6855.4976	6.800000e+03
-uw			1.0		1.000000e+00
-vad_postspeech		50		50
-vad_prespeech		20		20
-vad_startspeech	10		10
-vad_threshold		3.0		3.000000e+00
-var					
-varfloor		0.0001		1.000000e-04
-varnorm		no		no
-verbose		no		no
-warp_params				
-warp_type		inverse_linear	inverse_linear
-wbeam			7e-29		7.000000e-29
-wip			0.65		6.500000e-01
-wlen			0.025625	2.562500e-02

INFO: feat.c(715): Initializing feature stream to type: '1s_c_d_dd', ceplen=13, CMN='batch', VARNORM='no', AGC='none'
INFO: acmod.c(162): Using subvector specification 0-12/13-25/26-38
INFO: mdef.c(518): Reading model definition: en-us//mdef
INFO: mdef.c(531): Found byte-order mark BMDF, assuming this is a binary mdef file
INFO: bin_mdef.c(337): Reading binary model definition: en-us//mdef
INFO: bin_mdef.c(517): 42 CI-phone, 137053 CD-phone, 3 emitstate/phone, 126 CI-sen, 5126 Sen, 29324 Sen-Seq
INFO: tmat.c(149): Reading HMM transition probability matrices: en-us//transition_matrices
INFO: acmod.c(113): Attempting to use PTM computation module
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: en-us//means
INFO: ms_gauden.c(242): 42 codebook, 3 feature, size: 
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: en-us//variances
INFO: ms_gauden.c(242): 42 codebook, 3 feature, size: 
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(304): 222 variance values floored
INFO: ptm_mgau.c(475): Loading senones from dump file en-us//sendump
INFO: ptm_mgau.c(499): BEGIN FILE FORMAT DESCRIPTION
INFO: ptm_mgau.c(562): Rows: 128, Columns: 5126
INFO: ptm_mgau.c(594): Using memory-mapped I/O for senones
INFO: ptm_mgau.c(837): Maximum top-N: 4
INFO: phone_loop_search.c(114): State beam -225 Phone exit beam -225 Insertion penalty 0
INFO: dict.c(320): Allocating 4130 * 20 bytes (80 KiB) for word entries
INFO: dict.c(333): Reading main dictionary: TAR9897/9897.dic
INFO: dict.c(213): Dictionary size 29, allocated 0 KiB for strings, 0 KiB for phones
INFO: dict.c(336): 29 words read
INFO: dict.c(358): Reading filler dictionary: en-us//noisedict
INFO: dict.c(213): Dictionary size 34, allocated 0 KiB for strings, 0 KiB for phones
INFO: dict.c(361): 5 words read
INFO: dict2pid.c(396): Building PID tables for dictionary
INFO: dict2pid.c(406): Allocating 42^3 * 2 bytes (144 KiB) for word-initial triphones
INFO: dict2pid.c(132): Allocated 21336 bytes (20 KiB) for word-final triphones
INFO: dict2pid.c(196): Allocated 21336 bytes (20 KiB) for single-phone word triphones
INFO: ngram_model_trie.c(354): Trying to read LM in trie binary format
INFO: ngram_model_trie.c(365): Header doesn't match
INFO: ngram_model_trie.c(177): Trying to read LM in arpa format
INFO: ngram_model_trie.c(193): LM of order 3
INFO: ngram_model_trie.c(195): #1-grams: 26
INFO: ngram_model_trie.c(195): #2-grams: 41
INFO: ngram_model_trie.c(195): #3-grams: 36
INFO: lm_trie.c(474): Training quantizer
INFO: lm_trie.c(482): Building LM trie
INFO: ngram_search_fwdtree.c(74): Initializing search tree
INFO: ngram_search_fwdtree.c(101): 26 unique initial diphones
INFO: ngram_search_fwdtree.c(186): Creating search channels
INFO: ngram_search_fwdtree.c(323): Max nonroot chan increased to 202
INFO: ngram_search_fwdtree.c(333): Created 26 root, 74 non-root channels, 5 single-phone words
INFO: ngram_search_fwdflat.c(157): fwdflat: min_ef_width = 4, max_sf_win = 25
INFO: continuous.c(307): pocketsphinx_continuous COMPILED ON: Mar 10 2020, AT: 16:01:07

INFO: cmn_live.c(120): Update from < 41.00 -5.29 -0.12  5.09  2.48 -4.07 -1.37 -1.78 -5.08 -2.05 -6.45 -1.42  1.17 >
INFO: cmn_live.c(138): Update to   < 43.50 12.27  5.78  3.19 -3.29  0.34 -5.26 -11.33  4.52  0.05 -2.95 11.94 -3.49 >
INFO: ngram_search_fwdtree.c(1550):      525 words recognized (6/fr)
INFO: ngram_search_fwdtree.c(1552):    27719 senones evaluated (292/fr)
INFO: ngram_search_fwdtree.c(1556):    15927 channels searched (167/fr), 2366 1st, 9410 last
INFO: ngram_search_fwdtree.c(1559):      724 words for which last channels evaluated (7/fr)
INFO: ngram_search_fwdtree.c(1561):      876 candidate words for entering last phone (9/fr)
INFO: ngram_search_fwdtree.c(1564): fwdtree 1.34 CPU 1.406 xRT
INFO: ngram_search_fwdtree.c(1567): fwdtree 1.37 wall 1.446 xRT
INFO: ngram_search_fwdflat.c(302): Utterance vocabulary contains 10 words
INFO: ngram_search_fwdflat.c(948):      547 words recognized (6/fr)
INFO: ngram_search_fwdflat.c(950):    13371 senones evaluated (141/fr)
INFO: ngram_search_fwdflat.c(952):    11350 channels searched (119/fr)
INFO: ngram_search_fwdflat.c(954):      879 words searched (9/fr)
INFO: ngram_search_fwdflat.c(957):      312 word transitions (3/fr)
INFO: ngram_search_fwdflat.c(960): fwdflat 0.45 CPU 0.474 xRT
INFO: ngram_search_fwdflat.c(963): fwdflat 0.48 wall 0.501 xRT
INFO: ngram_search.c(1250): lattice start node <s>.0 end node </s>.88
INFO: ngram_search.c(1276): Eliminated 0 nodes before end node
INFO: ngram_search.c(1381): Lattice has 211 nodes, 433 links
INFO: ps_lattice.c(1376): Bestpath score: -1763
INFO: ps_lattice.c(1380): Normalizer P(O) = alpha(</s>:88:93) = -105689
INFO: ps_lattice.c(1437): Joint P(O,S) = -110043 P(S|O) = -4354
INFO: ngram_search.c(872): bestpath 0.01 CPU 0.006 xRT
INFO: ngram_search.c(875): bestpath 0.01 wall 0.008 xRT

INFO: ngram_search_fwdtree.c(429): TOTAL fwdtree 1.34 CPU 1.421 xRT
INFO: ngram_search_fwdtree.c(432): TOTAL fwdtree 1.37 wall 1.461 xRT
INFO: ngram_search_fwdflat.c(176): TOTAL fwdflat 0.45 CPU 0.479 xRT
INFO: ngram_search_fwdflat.c(179): TOTAL fwdflat 0.48 wall 0.507 xRT
INFO: ngram_search.c(303): TOTAL bestpath 0.01 CPU 0.006 xRT
INFO: ngram_search.c(306): TOTAL bestpath 0.01 wall 0.008 xRT
@nshmyrev
Copy link
Contributor

mips big endian or little endian?

@WilliamVJacob
Copy link

WilliamVJacob commented Mar 12, 2020 •
edited by nshmyrev
Loading

Hi @nshmyrev
This is my other github account .
The mips architecture follows little endian in our device
when i tried to run another command

pocketsphinx_continuous  -hmm en-us/ -lm TAR9897/9897.lm  -dict TAR9897/9897.dic  -adcdev sysdefault  -inmic Yes


I got the following logs

INFO: pocketsphinx.c(153): Parsed model-specific feature parameters from en-us//feat.params
Current configuration:
[NAME]			[DEFLT]		[VALUE]
-agc			none		none
-agcthresh		2.0		2.000000e+00
-allphone				
-allphone_ci		yes		yes
-alpha			0.97		9.700000e-01
-ascale			20.0		2.000000e+01
-aw			1		1
-backtrace		no		no
-beam			1e-48		1.000000e-48
-bestpath		yes		yes
-bestpathlw		9.5		9.500000e+00
-ceplen			13		13
-cmn			live		batch
-cmninit		40,3,-1		41.00,-5.29,-0.12,5.09,2.48,-4.07,-1.37,-1.78,-5.08,-2.05,-6.45,-1.42,1.17
-compallsen		no		no
-dict					TAR9897/9897.dic
-dictcase		no		no
-dither			no		no
-doublebw		no		no
-ds			1		1
-fdict					
-feat			1s_c_d_dd	1s_c_d_dd
-featparams				
-fillprob		1e-8		1.000000e-08
-frate			100		100
-fsg					
-fsgusealtpron		yes		yes
-fsgusefiller		yes		yes
-fwdflat		yes		yes
-fwdflatbeam		1e-64		1.000000e-64
-fwdflatefwid		4		4
-fwdflatlw		8.5		8.500000e+00
-fwdflatsfwin		25		25
-fwdflatwbeam		7e-29		7.000000e-29
-fwdtree		yes		yes
-hmm					en-us/
-input_endian		little		little
-jsgf					
-keyphrase				
-kws					
-kws_delay		10		10
-kws_plp		1e-1		1.000000e-01
-kws_threshold		1e-30		1.000000e-30
-latsize		5000		5000
-lda					
-ldadim			0		0
-lifter			0		22
-lm					TAR9897/9897.lm
-lmctl					
-lmname					
-logbase		1.0001		1.000100e+00
-logfn					
-logspec		no		no
-lowerf			133.33334	1.300000e+02
-lpbeam			1e-40		1.000000e-40
-lponlybeam		7e-29		7.000000e-29
-lw			6.5		6.500000e+00
-maxhmmpf		30000		30000
-maxwpf			-1		-1
-mdef					
-mean					
-mfclogdir				
-min_endfr		0		0
-mixw					
-mixwfloor		0.0000001	1.000000e-07
-mllr					
-mmap			yes		yes
-ncep			13		13
-nfft			512		512
-nfilt			40		25
-nwpen			1.0		1.000000e+00
-pbeam			1e-48		1.000000e-48
-pip			1.0		1.000000e+00
-pl_beam		1e-10		1.000000e-10
-pl_pbeam		1e-10		1.000000e-10
-pl_pip			1.0		1.000000e+00
-pl_weight		3.0		3.000000e+00
-pl_window		5		5
-rawlogdir				
-remove_dc		no		no
-remove_noise		yes		yes
-remove_silence		yes		yes
-round_filters		yes		yes
-samprate		16000		1.600000e+04
-seed			-1		-1
-sendump				
-senlogdir				
-senmgau				
-silprob		0.005		5.000000e-03
-smoothspec		no		no
-svspec					0-12/13-25/26-38
-tmat					
-tmatfloor		0.0001		1.000000e-04
-topn			4		4
-topn_beam		0		0
-toprule				
-transform		legacy		dct
-unit_area		yes		yes
-upperf			6855.4976	6.800000e+03
-uw			1.0		1.000000e+00
-vad_postspeech		50		50
-vad_prespeech		20		20
-vad_startspeech	10		10
-vad_threshold		3.0		3.000000e+00
-var					
-varfloor		0.0001		1.000000e-04
-varnorm		no		no
-verbose		no		no
-warp_params				
-warp_type		inverse_linear	inverse_linear
-wbeam			7e-29		7.000000e-29
-wip			0.65		6.500000e-01
-wlen			0.025625	2.562500e-02

INFO: feat.c(715): Initializing feature stream to type: '1s_c_d_dd', ceplen=13, CMN='batch', VARNORM='no', AGC='none'
INFO: acmod.c(162): Using subvector specification 0-12/13-25/26-38
INFO: mdef.c(518): Reading model definition: en-us//mdef
INFO: mdef.c(531): Found byte-order mark BMDF, assuming this is a binary mdef file
INFO: bin_mdef.c(337): Reading binary model definition: en-us//mdef
INFO: bin_mdef.c(517): 42 CI-phone, 137053 CD-phone, 3 emitstate/phone, 126 CI-sen, 5126 Sen, 29324 Sen-Seq
INFO: tmat.c(149): Reading HMM transition probability matrices: en-us//transition_matrices
INFO: acmod.c(113): Attempting to use PTM computation module
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: en-us//means
INFO: ms_gauden.c(242): 42 codebook, 3 feature, size: 
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: en-us//variances
INFO: ms_gauden.c(242): 42 codebook, 3 feature, size: 
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(304): 222 variance values floored
INFO: ptm_mgau.c(475): Loading senones from dump file en-us//sendump
INFO: ptm_mgau.c(499): BEGIN FILE FORMAT DESCRIPTION
INFO: ptm_mgau.c(562): Rows: 128, Columns: 5126
INFO: ptm_mgau.c(594): Using memory-mapped I/O for senones
INFO: ptm_mgau.c(837): Maximum top-N: 4
INFO: phone_loop_search.c(114): State beam -225 Phone exit beam -225 Insertion penalty 0
INFO: dict.c(320): Allocating 4130 * 20 bytes (80 KiB) for word entries
INFO: dict.c(333): Reading main dictionary: TAR9897/9897.dic
INFO: dict.c(213): Dictionary size 29, allocated 0 KiB for strings, 0 KiB for phones
INFO: dict.c(336): 29 words read
INFO: dict.c(358): Reading filler dictionary: en-us//noisedict
INFO: dict.c(213): Dictionary size 34, allocated 0 KiB for strings, 0 KiB for phones
INFO: dict.c(361): 5 words read
INFO: dict2pid.c(396): Building PID tables for dictionary
INFO: dict2pid.c(406): Allocating 42^3 * 2 bytes (144 KiB) for word-initial triphones
INFO: dict2pid.c(132): Allocated 21336 bytes (20 KiB) for word-final triphones
INFO: dict2pid.c(196): Allocated 21336 bytes (20 KiB) for single-phone word triphones
INFO: ngram_model_trie.c(354): Trying to read LM in trie binary format
INFO: ngram_model_trie.c(365): Header doesn't match
INFO: ngram_model_trie.c(177): Trying to read LM in arpa format
INFO: ngram_model_trie.c(193): LM of order 3
INFO: ngram_model_trie.c(195): #1-grams: 26
INFO: ngram_model_trie.c(195): #2-grams: 41
INFO: ngram_model_trie.c(195): #3-grams: 36
INFO: lm_trie.c(474): Training quantizer
INFO: lm_trie.c(482): Building LM trie
INFO: ngram_search_fwdtree.c(74): Initializing search tree
INFO: ngram_search_fwdtree.c(101): 26 unique initial diphones
INFO: ngram_search_fwdtree.c(186): Creating search channels
INFO: ngram_search_fwdtree.c(323): Max nonroot chan increased to 202
INFO: ngram_search_fwdtree.c(333): Created 26 root, 74 non-root channels, 5 single-phone words
INFO: ngram_search_fwdflat.c(157): fwdflat: min_ef_width = 4, max_sf_win = 25
INFO: continuous.c(307): pocketsphinx_continuous COMPILED ON: Mar 10 2020, AT: 16:01:07

INFO: continuous.c(252): Ready....
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)

@nshmyrev
Copy link
Contributor

Try to recognize a file with jsgf grammar.

@nshmyrev
Copy link
Contributor

As for the second log, decoding is too slow on your device, probably it is very tiny cpu and the model needs optimization.

@WilliamVJacob
Copy link

Thanks for the prompt response
Can you give me little more details on optimizing this code

@nshmyrev
Copy link
Contributor

@WilliamVJacob sure, describe me in more details the application you want to build and the hardware you have in mind.

@WilliamVJacob
Copy link

I want to run pocketsphinx on an embedded device with 2 USB mic , 256 RAM , 128 MB Flash The processor is ALI M3627-AL single core.
I want to build an application where the user will say some commands and it will will convert it into speech, I have already prepared the dictionary and language model and tested on laptop but its not working on the embedded device

@dhdaines
Copy link
Contributor

Hi, pocketsphinx_continuous no longer exists, but, it seems we still have some issues with mipsel architecture. If you are able to try the current code (with pocketsphinx_batch) I would be interested to see the results.

@dhdaines dhdaines added this to the 5.0.0 milestone Jun 13, 2022
@zavalyshyn
Copy link

why is pocketsphinx_continuous not available anymore?

@dhdaines
Copy link
Contributor

dhdaines commented Jun 13, 2022 •
edited
Loading

No more audio library. If there's a reasonable use case for pocketsphinx_continuous we could reimplement it for specific platforms, and there will certainly be a test (there is already I think) and example code for the continuous listening API. Were people using pocketsphinx_continuous to implement actual applications?

@dhdaines
Copy link
Contributor

Also you'll note that the original reporter is using pocketsphinx_continuous to do batch mode recognition... which should be done with pocketsphinx_batch ... which will get a bit of a makeover to be more user-friendly.

@dhdaines
Copy link
Contributor

Hi, I would very much like to fix this as well as #252 but I don't have access to a mipsel machine for testing. The best I can do will be to validate it with QEMU, which is hopefully a faithful replication of the architecture.

dhdaines added a commit that referenced this issue Sep 28, 2022
@dhdaines
fix: avoid underflow in GMM computation
fbd61be 
MIPS, among others, will *not* fail gracefully here (should fix #252, #199)
@dhdaines dhdaines linked a pull request Sep 28, 2022 that will close this issue
Fix float-to-int underflows (#199, #252) #303
Merged
dhdaines added a commit that referenced this issue Sep 28, 2022
@dhdaines
Merge pull request #303 from cmusphinx/199-pocketsphinx_continuous-is…
080aa8c 
…-unable-to-convert-speech-to-text

Fix float-to-int underflows (#199, #252)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
5.0.0
Development

Successfully merging a pull request may close this issue.

Fix float-to-int underflows (#199, #252) cmusphinx/pocketsphinx
4 participants
@nshmyrev @dhdaines @zavalyshyn @WilliamVJacob