text: tion (Gong et al. 2014). To overcome this problem, we set dof(xtetm, xpto(cid:48))ra=l d1istiafn|cte−. Wt(cid:48)e| w>ilλl v, awlihdeartee λthicsohnytrpoolsthtehseisdiengtrheee (cid:79) J(θ) ≈ 1 (cid:88)N (cid:88)T R (cid:79) log π (a |h ), (9) θ n θ θ t t N Experiments section. n=1 t=1 Representativeness reward. This reward measures how where R is the reward computed at the nth episode. well the generated summary can represent the original video. n Eq. (9) is also known as the episodic REINFORCE algo- To this end, we formulate the degree of representativeness of rithm. a video summary as the k-medoids problem (Gygli, Grabner, Although the gradient in Eq. (9) is a good estimate, it may and Van Gool 2015). In particular, we want the agent to se- contain high variance which will make the network hard to lect a set of medoids such that the mean of squared errors converge. A common countermeasure is to subtract the re- between video frames and their nearest medoids is minimal. ward by a constant baseline b, so the gradient becomes Therefore, we deﬁne Rrep as 1 (cid:88)T (cid:79) 1 (cid:88)N (cid:88)T (cid:79) Rrep = exp(−T tm(cid:48)∈iYn ||xt − xt(cid:48)||2). (5) θJ(θ) ≈ N (Rn − b) θ log πθ(at|ht), (10) t=1 n=1 t=1 With this reward, the agent is encouraged to select frames where b is simply computed as the moving average of re- that are close to the cluster centers in the feature space. An wards experienced so far for computational efﬁciency. alternative formulation of Rrep can be the inverse reconstruc- Regularization tion errors achieved by the selected frames, but this formu- lation is too computationally expensive. Since selecting more frames will also increase the reward, Diversity-representativeness reward. Rdiv and Rrep we impose a regularization term on the probability distribu- complement to each other and work jointly to guide the tions p produced by DSN in order to constrain the per- 1:T learning of DSN: centage of frames selected for the summary. Inspired by (Mahasseni, Lam, and Todorovic 2017), we minimize the R(S) = Rdiv + Rrep. (6) following term during training, maDgunritiundget.raIinnifnagc,t,RitdivisanndonR-trreipviaarletsoimkieleapr iRn dtihveaonrddeRr roepf Lpercentage = ||T1 (cid:88)T pt − (cid:15)||2, (11) t=1 at the same order of magnitude during training, thus none of them would dominate in gradient computation. We give where (cid:15) determines the percentage of frames to be se- zero reward to DSN when no frames are selected, i.e., the lected. sampled actions are all zeros. In addition, we also add the (cid:96)2 regularization term on the Training with Policy Gradient weight parameters θ to avoid overﬁtting The goal of our summarization agent is to learn a policy (cid:88) function π with parameters θ by maximizing the expected Lweight = θi2,j. (12) θ rewards i,j Optimization E J(θ) = [R(S)], (7) pθ(a1:T) We optimize the policy function’s parameters θ via stochastic gradient-based method. By combing the gradients where pθ(a1:T ) denotes the probability distributions over computed from Eq. (10), Eq. (11) and Eq. (12), we update θ possible action sequences, and R(S) is computed by Eq. (6). as π is deﬁned by our DSN. θ Following the REINFORCE algorithm proposed by (cid:79) Williams (Williams 1992), we can compute the derivative θ = θ − α θ(−J + β1Lpercentage + β2Lweight), (13) of the objective function J(θ) w.r.t. the parameters θ as where α is learning rate, and β and β are hyperparame- 1 2 (cid:79) E (cid:88)T (cid:79) ters that balance the weighting. θJ(θ) = pθ(a1:T)[R(S) θ log πθ(at|ht)], (8) In practice, we use Adam (Kingma and Ba 2014) as the t=1 optimization algorithm. As a result of learning, the log- probability of actions taken by the network that have led to where a is the action taken by DSN at time t and h is t t high rewards is increased, while that of actions that have re- the hidden state from the BiRNN. sulted in low rewards is decreased. Since Eq. (8) involves the expectation over high- Extension to Supervised Learning dimensional action sequences, which is hard to compute di- rectly, we approximate the gradient by running the agent for Given the keyframe indices for a video, Y∗ = N episodes on the same video and then taking the average {y∗|i = 1, ..., |Y∗|}, we use Maximum Likelihood Esti- i gradient mation (MLE) to maximize the log-probability of selecting ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: tion

text: (Gong

text: et

text: al.

text: To

text: overcome

text: this

text: problem,

text: we

text: set

text: dof(xtetm,

text: w>ilλl

text: v,

text: awlihdeartee

text: λthicsohnytrpoolsthtehseisdiengtrheee

text: J(θ)

text: R

text: log

text: π

text: (a

text: |h

text: θ

text: n

text: θ

text: θ

text: t

text: t

text: N

text: Experiments

text: section.

text: Representativeness

text: reward.

text: This

text: reward

text: measures

text: how

text: where

text: R

text: is

text: the

text: reward

text: computed

text: at

text: the

text: nth

text: episode.

text: well

text: the

text: generated

text: summary

text: can

text: represent

text: the

text: original

text: video.

text: n

text: Eq.

text: is

text: also

text: known

text: as

text: the

text: episodic

text: REINFORCE

text: algo-

text: To

text: this

text: end,

text: we

text: formulate

text: the

text: degree

text: of

text: representativeness

text: of

text: rithm.

text: a

text: video

text: summary

text: as

text: the

text: k-medoids

text: problem

text: (Gygli,

text: Grabner,

text: Although

text: the

text: gradient

text: in

text: Eq.

text: is

text: a

text: good

text: estimate,

text: it

text: may

text: and

text: Van

text: Gool

text: In

text: particular,

text: we

text: want

text: the

text: agent

text: to

text: se-

text: contain

text: high

text: variance

text: which

text: will

text: make

text: the

text: network

text: hard

text: to

text: lect

text: a

text: set

text: of

text: medoids

text: such

text: that

text: the

text: mean

text: of

text: squared

text: errors

text: converge.

text: A

text: common

text: countermeasure

text: is

text: to

text: subtract

text: the

text: re-

text: between

text: video

text: frames

text: and

text: their

text: nearest

text: medoids

text: is

text: minimal.

text: ward

text: by

text: a

text: constant

text: baseline

text: b,

text: so

text: the

text: gradient

text: becomes

text: Therefore,

text: we

text: deﬁne

text: Rrep

text: as

text: Rrep

text: exp(−T

text: ||xt

text: θJ(θ)

text: N

text: (Rn

text: b)

text: θ

text: log

text: πθ(at|ht),

text: With

text: this

text: reward,

text: the

text: agent

text: is

text: encouraged

text: to

text: select

text: frames

text: where

text: b

text: is

text: simply

text: computed

text: as

text: the

text: moving

text: average

text: of

text: re-

text: that

text: are

text: close

text: to

text: the

text: cluster

text: centers

text: in

text: the

text: feature

text: space.

text: An

text: wards

text: experienced

text: so

text: far

text: for

text: computational

text: efﬁciency.

text: alternative

text: formulation

text: of

text: Rrep

text: can

text: be

text: the

text: inverse

text: reconstruc-

text: Regularization

text: tion

text: errors

text: achieved

text: by

text: the

text: selected

text: frames,

text: but

text: this

text: formu-

text: lation

text: is

text: too

text: computationally

text: expensive.

text: Since

text: selecting

text: more

text: frames

text: will

text: also

text: increase

text: the

text: reward,

text: Diversity-representativeness

text: reward.

text: Rdiv

text: and

text: Rrep

text: we

text: impose

text: a

text: regularization

text: term

text: on

text: the

text: probability

text: distribu-

text: complement

text: to

text: each

text: other

text: and

text: work

text: jointly

text: to

text: guide

text: the

text: tions

text: p

text: produced

text: by

text: DSN

text: in

text: order

text: to

text: constrain

text: the

text: per-

text: learning

text: of

text: DSN:

text: centage

text: of

text: frames

text: selected

text: for

text: the

text: summary.

text: Inspired

text: by

text: (Mahasseni,

text: Lam,

text: and

text: Todorovic

text: we

text: minimize

text: the

text: R(S)

text: Rdiv

text: Rrep.

text: following

text: term

text: during

text: training,

text: iRn

text: dtihveaonrddeRr

text: roepf

text: Lpercentage

text: pt

text: at

text: the

text: same

text: order

text: of

text: magnitude

text: during

text: training,

text: thus

text: of

text: them

text: would

text: dominate

text: in

text: gradient

text: computation.

text: We

text: give

text: where

text: determines

text: the

text: percentage

text: of

text: frames

text: to

text: be

text: se-

text: zero

text: reward

text: to

text: DSN

text: when

text: no

text: frames

text: are

text: selected,

text: i.e.,

text: the

text: lected.

text: sampled

text: actions

text: are

text: all

text: zeros.

text: In

text: addition,

text: we

text: also

text: add

text: the

text: regularization

text: term

text: on

text: the

text: Training

text: with

text: Policy

text: Gradient

text: weight

text: parameters

text: θ

text: to

text: avoid

text: overﬁtting

text: The

text: goal

text: of

text: our

text: summarization

text: agent

text: is

text: to

text: learn

text: a

text: policy

text: function

text: π

text: with

text: parameters

text: θ

text: by

text: maximizing

text: the

text: expected

text: Lweight

text: θ

text: rewards

text: i,j

text: Optimization

text: E

text: J(θ)

text: [R(S)],

text: We

text: optimize

text: the

text: policy

text: function’s

text: parameters

text: θ

text: via

text: stochastic

text: gradient-based

text: method.

text: By

text: combing

text: the

text: gradients

text: where

text: denotes

text: the

text: probability

text: distributions

text: over

text: computed

text: from

text: Eq.

text: Eq.

text: and

text: Eq.

text: we

text: update

text: θ

text: possible

text: action

text: sequences,

text: and

text: R(S)

text: is

text: computed

text: by

text: Eq.

text: as

text: π

text: is

text: deﬁned

text: by

text: our

text: DSN.

text: θ

text: Following

text: the

text: REINFORCE

text: algorithm

text: proposed

text: by

text: Williams

text: (Williams

text: we

text: can

text: compute

text: the

text: derivative

text: θ

text: θ

text: α

text: θ(−J

text: of

text: the

text: objective

text: function

text: J(θ)

text: w.r.t.

text: the

text: parameters

text: θ

text: as

text: where

text: α

text: is

text: learning

text: rate,

text: and

text: β

text: and

text: β

text: are

text: hyperparame-

text: E

text: ters

text: that

text: balance

text: the

text: weighting.

text: θJ(θ)

text: θ

text: log

text: πθ(at|ht)],

text: In

text: practice,

text: we

text: use

text: Adam

text: (Kingma

text: and

text: Ba

text: as

text: the

text: optimization

text: algorithm.

text: As

text: a

text: result

text: of

text: learning,

text: the

text: log-

text: probability

text: of

text: actions

text: taken

text: by

text: the

text: network

text: that

text: have

text: led

text: to

text: where

text: a

text: is

text: the

text: action

text: taken

text: by

text: DSN

text: at

text: time

text: t

text: and

text: h

text: is

text: t

text: t

text: high

text: rewards

text: is

text: increased,

text: while

text: that

text: of

text: actions

text: that

text: have

text: re-

text: the

text: hidden

text: state

text: from

text: the

text: BiRNN.

text: sulted

text: in

text: low

text: rewards

text: is

text: decreased.

text: Since

text: Eq.

text: involves

text: the

text: expectation

text: over

text: high-

text: Extension

text: to

text: Supervised

text: Learning

text: dimensional

text: action

text: sequences,

text: which

text: is

text: hard

text: to

text: compute

text: di-

text: rectly,

text: we

text: approximate

text: the

text: gradient

text: by

text: running

text: the

text: agent

text: for

text: Given

text: the

text: keyframe

text: indices

text: for

text: a

text: video,

text: Y∗

text: N

text: episodes

text: on

text: the

text: same

text: video

text: and

text: then

text: taking

text: the

text: average

text: {y∗|i

text: |Y∗|},

text: we

text: use

text: Maximum

text: Likelihood

text: Esti-

text: i

text: gradient

text: mation

text: (MLE)

text: to

text: maximize

text: the

text: log-probability

text: of

text: selecting

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: A. GALINDRO AND D. F. M. TORRES 11 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0 50 100 150 t Figure 5. Unemployment rate: real data (dashed line) versus optimal control simulation (continuous-line). 5. Conclusion In this work, we propose a simple mathematical model that describes more accurately the real data of unemployment from Portugal in the period 2004–2016. At ﬁrst, we considered the proposed model from Munoli and Gani (2016) [7], ﬁtting their initial conditions with our data. We found out that the model suggested by Munoli and Gani in 2016 does not describes at all the Portuguese unemployment environment. Because of this, we performed a few changes to the Munoli and Gani model [7], eliminating one of the differential equations, adding an unemployment/wage correlation feature, and adjusting the natural rate of unemployed/employed people in order to control and balance the inner population of the model. Simulations of our model show a much more accurate emulation of the Portuguese unemployment environment. Our results also show a slight decrease on the overall labour force (unemployed plus employed) and, since the Portuguese total population smoothly increased over the last decade, that might signals an higher percentage of inactive population, which may underpin pressure on more social protection measures. Other reading may consist on the premise that the number of unknown unemployed people is increasing and standing apart from the government ofﬁcial records. From the application of optimal control, we can state the following interesting conclusions: the indirect policies should be the predominant method of avoiding unemployment, whereas the supply of internships should be the main choice when the total level of employment offered is low. A possible explanation why to avoid internships in high unemployment periods might be correlated to the severe labour devaluation (considered when we developed the considered model) during these seasons, since the available labour force is already cheap enough and the internships supply might be offering jobs with an increased government expenditure, due to the whole administrative costs, to people that might turned out to be employed anyway. Moreover, our study also suggests that governmental policies should be performed mainly during favourable periods in order to avoid high unemployment levels in future crisis. As future work, we plan to apply different methods of optimal control to our model in order to seek even more solid ways and tools to control the unemployment issue. For that, we may consider to split the unemployment class into two: unemployed that currently receive welfare from government; and unemployed which do not receive any ﬁnancial support. These two classes present different government concerns: the ﬁrst one emphasizes ﬁnancial pressure and the second one social and well-being pressure. Another interesting study might be to include non- active population, like retired people, and study the optimal control regarding the social security ﬁnancial health. Acknowledgements The authors were supported by the Center for Research and Development in Mathematics and Applications (CIDMA) of the University of Aveiro, through Fundac¸a˜o para a Cieˆncia e a Tecnologia (FCT), within project UID/MAT/04106/2013. They are very grateful to two referees, for several comments, suggestions and questions, which helped them to improve the quality of the paper. Preprint accepted in Statistics Opt. Inform. Comput. )etaR tnemyolpmenU( )t(RU ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: A.

label: paragraph

text: GALINDRO

text: AND

text: D.

text: F.

text: M.

text: TORRES

font: default

text: t

text: Figure

text: Unemployment

label: caption

text: rate:

text: real

text: data

text: (dashed

text: line)

text: versus

text: optimal

text: control

text: simulation

text: (continuous-line).

label: section

text: Conclusion

text: In

text: this

text: work,

text: we

text: propose

text: a

text: simple

text: mathematical

text: model

text: that

text: describes

text: more

text: accurately

text: the

text: real

text: data

text: of

text: unemployment

text: from

text: Portugal

text: in

text: the

text: period

text: At

text: ﬁrst,

text: we

text: considered

text: the

text: proposed

text: model

text: from

text: Munoli

text: and

text: Gani

text: ﬁtting

text: their

text: initial

text: conditions

text: with

text: our

text: data.

text: We

text: found

text: out

text: that

text: the

text: model

text: suggested

text: by

text: Munoli

text: and

text: Gani

text: in

text: does

text: not

text: describes

text: at

text: all

text: the

text: Portuguese

text: unemployment

text: environment.

text: Because

text: of

text: this,

text: we

text: performed

text: a

text: few

text: changes

text: to

text: the

text: Munoli

text: and

text: Gani

text: model

text: eliminating

text: one

text: of

text: the

text: differential

text: equations,

text: adding

text: an

text: unemployment/wage

text: correlation

text: feature,

text: and

text: adjusting

text: the

text: natural

text: rate

text: of

text: unemployed/employed

text: people

text: in

text: order

text: to

text: control

text: and

text: balance

text: the

text: inner

text: population

text: of

text: the

text: model.

text: Simulations

text: of

text: our

text: model

text: show

text: a

text: much

text: more

text: accurate

text: emulation

text: of

text: the

text: Portuguese

text: unemployment

text: environment.

text: Our

text: results

text: also

text: show

text: a

text: slight

text: decrease

text: on

text: the

text: overall

text: labour

text: force

text: (unemployed

text: plus

text: employed)

text: and,

text: since

text: the

text: Portuguese

text: total

text: population

text: smoothly

text: increased

text: over

text: the

text: last

text: decade,

text: that

text: might

text: signals

text: an

text: higher

text: percentage

text: of

text: inactive

text: population,

text: which

text: may

text: underpin

text: pressure

text: on

text: more

text: social

text: protection

text: measures.

text: Other

text: reading

text: may

text: consist

text: on

text: the

text: premise

text: that

text: the

text: number

text: of

text: unknown

text: unemployed

text: people

text: is

text: increasing

text: and

text: standing

text: apart

text: from

text: the

text: government

text: ofﬁcial

text: records.

text: From

text: the

text: application

text: of

text: optimal

text: control,

text: we

text: can

text: state

text: the

text: following

text: interesting

text: conclusions:

text: the

text: indirect

text: policies

text: should

text: be

text: the

text: predominant

text: method

text: of

text: avoiding

text: unemployment,

text: whereas

text: the

text: supply

text: of

text: internships

text: should

text: be

text: the

text: main

text: choice

text: when

text: the

text: total

text: level

text: of

text: employment

text: offered

text: is

text: low.

text: A

text: possible

text: explanation

text: why

text: to

text: avoid

text: internships

text: in

text: high

text: unemployment

text: periods

text: might

text: be

text: correlated

text: to

text: the

text: severe

text: labour

text: devaluation

text: (considered

text: when

text: we

text: developed

text: the

text: considered

text: model)

text: during

text: these

text: seasons,

text: since

text: the

text: available

text: labour

text: force

text: is

text: already

text: cheap

text: enough

text: and

text: the

text: internships

text: supply

text: might

text: be

text: offering

text: jobs

text: with

text: an

text: increased

text: government

text: expenditure,

text: due

text: to

text: the

text: whole

text: administrative

text: costs,

text: to

text: people

text: that

text: might

text: turned

text: out

text: to

text: be

text: employed

text: anyway.

text: Moreover,

text: our

text: study

text: also

text: suggests

text: that

text: governmental

text: policies

text: should

text: be

text: performed

text: mainly

text: during

text: favourable

text: periods

text: in

text: order

text: to

text: avoid

text: high

text: unemployment

text: levels

text: in

text: future

text: crisis.

text: As

text: future

text: work,

text: we

text: plan

text: to

text: apply

text: different

text: methods

text: of

text: optimal

text: control

text: to

text: our

text: model

text: in

text: order

text: to

text: seek

text: even

text: more

text: solid

text: ways

text: and

text: tools

text: to

text: control

text: the

text: unemployment

text: issue.

text: For

text: that,

text: we

text: may

text: consider

text: to

text: split

text: the

text: unemployment

text: class

text: into

text: two:

text: unemployed

text: that

text: currently

text: receive

text: welfare

text: from

text: government;

text: and

text: unemployed

text: which

text: do

text: not

text: receive

text: any

text: ﬁnancial

text: support.

text: These

text: two

text: classes

text: present

text: different

text: government

text: concerns:

text: the

text: ﬁrst

text: one

text: emphasizes

text: ﬁnancial

text: pressure

text: and

text: the

text: second

text: one

text: social

text: and

text: well-being

text: pressure.

text: Another

text: interesting

text: study

text: might

text: be

text: to

text: include

text: non-

text: active

text: population,

text: like

text: retired

text: people,

text: and

text: study

text: the

text: optimal

text: control

text: regarding

text: the

text: social

text: security

text: ﬁnancial

text: health.

text: Acknowledgements

text: The

text: authors

text: were

text: supported

text: by

text: the

text: Center

text: for

text: Research

text: and

text: Development

text: in

text: Mathematics

text: and

text: Applications

text: (CIDMA)

text: of

text: the

text: University

text: of

text: Aveiro,

text: through

text: Fundac¸a˜o

text: para

text: a

text: Cieˆncia

text: e

text: a

text: Tecnologia

text: (FCT),

text: within

text: project

text: They

text: are

text: very

text: grateful

text: to

text: two

text: referees,

text: for

text: several

text: comments,

text: suggestions

text: and

text: questions,

text: which

text: helped

text: them

text: to

text: improve

text: the

text: quality

text: of

text: the

text: paper.

text: Preprint

text: accepted

text: in

text: Statistics

text: Opt.

text: Inform.

text: Comput.

text: )etaR

text: tnemyolpmenU(

text: )t(RU

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: A. GALINDRO AND D. F. M. TORRES 13 a0 = 1.478e+04 (1.444e+04, 1.512e+04) a1 = -1262 (-1841, -683.7) b1 = -2006 (-2469, -1543) a2 = 328.2 (-988.4, 1645) b2 = -4700 (-5169, -4231) a3 = -1992 (-2474, -1510) b3 = 2.399 (-1202, 1206) w = 0.04009 (0.03864, 0.04153) Goodness of fit: SSE: 5.995e+08 R-square: 0.8046 Adjusted R-square: 0.795 RMSE: 2055 function [fitresult, gof] = createFit(T, Vacancies) [xData, yData] = prepareCurveData( T, Vacancies ); ft = fittype( ’fourier3’ ); opts = fitoptions( ft ); opts.Display = ’Off’; opts.Lower = [-Inf -Inf -Inf -Inf -Inf -Inf -Inf -Inf]; opts.StartPoint = [0 0 0 0 0 0 0 0.0421690289072455]; opts.Upper = [Inf Inf Inf Inf Inf Inf Inf Inf]; [fitresult, gof] = fit( xData, yData, ft, opts ); figure( ’Name’, ’VacanciesFit’ ); h = plot( fitresult, xData, yData ); legend( h, ’Vacancies vs. T’, ’VacanciesFit’, ’Location’, ’NorthEast’ ); xlabel( ’T’ ); ylabel( ’Vacancies’ ); grid on corr(RCU,RCE) B. ACADO code For the numerical solution of the optimal control problem (12)–(16) described in Section , we used the ACADO Toolkit, which is a free software environment and algorithm collection for automatic control and dynamic optimization [4]: #include <acado_toolkit.hpp> #include <acado_gnuplot.hpp> int main( ){ USING_NAMESPACE_ACADO DifferentialState x1,x2,x3 ; // the differential states Control u1,u2 ; IntermediateState mu ; const double t_start = 0.0; const double t_end = 150.0; const double T = t_end - t_start; const double a0 = 1.478e+04; const double a1 = -1262; const double b1 = -2006; const double a2 = 328.2; Preprint accepted in Statistics Opt. Inform. Comput.

text: A.

text: GALINDRO

text: AND

text: D.

text: F.

text: M.

text: TORRES

font: JIIBWI+NimbusMonL-Regu

text: w

text: Goodness

text: of

text: fit:

text: SSE:

text: R-square:

text: Adjusted

text: R-square:

text: RMSE:

text: function

text: [fitresult,

text: gof]

text: createFit(T,

text: Vacancies)

text: [xData,

text: yData]

text: prepareCurveData(

text: T,

text: Vacancies

text: ft

text: fittype(

text: opts

text: fitoptions(

text: ft

text: opts.Display

text: ’Off’;

text: opts.Lower

text: [-Inf

text: -Inf

text: -Inf

text: -Inf

text: -Inf

text: -Inf

text: -Inf

text: -Inf];

text: opts.StartPoint

text: opts.Upper

text: [Inf

text: Inf

text: Inf

text: Inf

text: Inf

text: Inf

text: Inf

text: Inf];

text: [fitresult,

text: gof]

text: fit(

text: xData,

text: yData,

text: ft,

text: opts

text: figure(

text: ’Name’,

text: ’VacanciesFit’

text: h

text: plot(

text: fitresult,

text: xData,

text: yData

text: legend(

text: h,

text: ’Vacancies

text: vs.

text: T’,

text: ’VacanciesFit’,

text: ’Location’,

text: ’NorthEast’

text: xlabel(

text: ’T’

text: ylabel(

text: ’Vacancies’

text: grid

text: on

text: corr(RCU,RCE)

text: B.

text: ACADO

text: code

text: For

text: the

text: numerical

text: solution

text: of

text: the

text: optimal

text: control

text: problem

text: described

text: in

text: Section

text: we

text: used

text: the

text: ACADO

text: Toolkit,

text: which

text: is

text: a

text: free

text: software

text: environment

text: and

text: algorithm

text: collection

text: for

text: automatic

text: control

text: and

text: dynamic

text: optimization

text: #include

text: #include

text: int

text: main(

text: DifferentialState

text: the

text: differential

text: states

text: Control

text: IntermediateState

text: mu

text: const

text: double

text: const

text: double

text: const

text: double

text: T

text: const

text: double

text: const

text: double

text: const

text: double

text: const

text: double

text: Preprint

text: accepted

text: in

text: Statistics

text: Opt.

text: Inform.

text: Comput.


text: A. GALINDRO AND D. F. M. TORRES 3 calculation (1) using the variables that we actually could gather: the unemployment rate UR and the total number of unemployed people U. Munoli and Gani (2016) [7] present a model that tries to emulate an unemployment environment. Their model consists of three ordinary differential equations, considering the unemployed (U), the employed (E), and available vacancies (V ):        dU(t) − − = Λ κU(t)V (t) α U(t) + γE(t),  dt 1   dE(t) − − (4)  = κU(t)V (t) α E(t) γE(t), dt 2 dV (t) − = α E(t) + γE(t) δV (t) + φU(t). dt 2 The parameters and variables of model (4) are described on Table 1. At Munoli and Gani (2016) [7], the initial Table 1. Variables and parameters considered by Munoli and Gani (2016) [7]. Variable Meaning U(t) Number of unemployed individuals at time t E(t) Number of employed individuals at time t V (t) Number of vacancies at time t Λ Number of unemployed individuals that is increasing continuously κ Rate at which the unemployed individuals are becoming employed α Rate of migration and death of unemployed individuals 1 α Rate of retirement as well as death of employed individuals 2 γ Rate of individuals who are ﬁred from their jobs φ Represents the rate of creating new vacancies δ Denotes the diminution rate of vacancies due to lack of funds conditions were given by U(0) = 10000, E(0) = 1000 and V (0) = 100. A time line of 150 units (t = 150) was considered [7]. Values of the other parameters were fulﬁlled according with Table 2. We replaced the initial Table 2. Variables and parameters considered by Munoli and Gani (2016) [7]. Parameters Base Value Reference Λ 5000 Misra and Singh (2013) [6] κ 0.000009 Misra and Singh (2013) [6] α 0.04 Misra and Singh (2013) [6] 1 α 0.05 Misra and Singh (2013) [6] 2 γ 0.001 Assumed [7] φ 0.007 Assumed [7] δ 0.05 Assumed [7] conditions suggested by the authors of [7] by the ones given by the real data from Portugal. Precisely, we ﬁxed the initial values to the ones of Portugal at January 2004, when the number of unemployed people was 464450 (U), employed was 6450694 (E, according with (1)) and the available vacancies at the time were 4848 (V ). We can easily state that the suggested model from Munoli and Gani (2016) [7] does not replicate properly the real data from Portugal. Regarding the employment and unemployment (see Figures 1a and 1b, respectively) the model of [7] kind of implode, since the unemployed/employment values dramatically drop until the end of the time period. Munoli and Gani (2016) [7] differential system (4) also suggests an exceptional increase on the supply of job vacancies with a smoother decrease until the end of the time period, a statement that is not supported by the Portuguese real Preprint accepted in Statistics Opt. Inform. Comput. ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: A.

text: GALINDRO

text: AND

text: D.

text: F.

text: M.

text: TORRES

text: calculation

text: using

text: the

text: variables

text: that

text: we

text: actually

text: could

text: gather:

text: the

text: unemployment

text: rate

text: UR

text: and

text: the

text: total

text: number

text: of

text: unemployed

text: people

text: U.

text: Munoli

text: and

text: Gani

text: present

text: a

text: model

text: that

text: tries

text: to

text: emulate

text: an

text: unemployment

text: environment.

text: Their

text: model

text: consists

text: of

text: three

text: ordinary

text: differential

text: equations,

text: considering

text: the

text: unemployed

text: (U),

text: the

text: employed

text: (E),

text: and

text: available

text: vacancies

text: (V

label: equation

text: dU(t)

text: Λ

text: κU(t)V

text: (t)

text: α

text: U(t)

text: γE(t),

text: dt

text: dE(t)

text: κU(t)V

text: (t)

text: α

text: E(t)

text: γE(t),

text: dt

text: dV

text: (t)

text: α

text: E(t)

text: γE(t)

text: δV

text: (t)

text: φU(t).

text: dt

text: The

text: parameters

text: and

text: variables

text: of

text: model

text: are

text: described

text: on

text: Table

text: At

text: Munoli

text: and

text: Gani

text: the

text: initial

text: Table

text: Variables

text: and

text: parameters

text: considered

text: by

text: Munoli

text: and

text: Gani

text: Variable

text: Meaning

text: U(t)

text: Number

text: of

text: unemployed

text: individuals

text: at

text: time

text: t

text: E(t)

text: Number

text: of

text: employed

text: individuals

text: at

text: time

text: t

text: V

text: (t)

text: Number

text: of

text: vacancies

text: at

text: time

text: t

text: Λ

text: Number

text: of

text: unemployed

text: individuals

text: that

text: is

text: increasing

text: continuously

text: κ

text: Rate

text: at

text: which

text: the

text: unemployed

text: individuals

text: are

text: becoming

text: employed

text: α

text: Rate

text: of

text: migration

text: and

text: death

text: of

text: unemployed

text: individuals

text: α

text: Rate

text: of

text: retirement

text: as

text: well

text: as

text: death

text: of

text: employed

text: individuals

text: γ

text: Rate

text: of

text: individuals

text: who

text: are

text: ﬁred

text: from

text: their

text: jobs

text: φ

text: Represents

text: the

text: rate

text: of

text: creating

text: new

text: vacancies

text: δ

text: Denotes

text: the

text: diminution

text: rate

text: of

text: vacancies

text: due

text: to

text: lack

text: of

text: funds

text: conditions

text: were

text: given

text: by

text: and

text: V

text: A

text: time

text: line

text: of

text: units

text: (t

text: was

text: considered

text: Values

text: of

text: the

text: other

text: parameters

text: were

text: fulﬁlled

text: according

text: with

text: Table

text: We

text: replaced

text: the

text: initial

text: Table

text: Variables

text: and

text: parameters

text: considered

text: by

text: Munoli

text: and

text: Gani

text: Parameters

text: Base

text: Value

text: Reference

text: Λ

text: Misra

text: and

text: Singh

text: κ

text: Misra

text: and

text: Singh

text: α

text: Misra

text: and

text: Singh

text: α

text: Misra

text: and

text: Singh

text: γ

text: Assumed

text: φ

text: Assumed

text: δ

text: Assumed

text: conditions

text: suggested

text: by

text: the

text: authors

text: of

text: by

text: the

text: ones

text: given

text: by

text: the

text: real

text: data

text: from

text: Portugal.

text: Precisely,

text: we

text: ﬁxed

text: the

text: initial

text: values

text: to

text: the

text: ones

text: of

text: Portugal

text: at

text: January

text: when

text: the

text: number

text: of

text: unemployed

text: people

text: was

text: (U),

text: employed

text: was

text: (E,

text: according

text: with

text: and

text: the

text: available

text: vacancies

text: at

text: the

text: time

text: were

text: (V

text: We

text: can

text: easily

text: state

text: that

text: the

text: suggested

text: model

text: from

text: Munoli

text: and

text: Gani

text: does

text: not

text: replicate

text: properly

text: the

text: real

text: data

text: from

text: Portugal.

text: Regarding

text: the

text: employment

text: and

text: unemployment

text: (see

text: Figures

text: and

text: respectively)

text: the

text: model

text: of

text: kind

text: of

text: implode,

text: since

text: the

text: unemployed/employment

text: values

text: dramatically

text: drop

text: until

text: the

text: end

text: of

text: the

text: time

text: period.

text: Munoli

text: and

text: Gani

text: differential

text: system

text: also

text: suggests

text: an

text: exceptional

text: increase

text: on

text: the

text: supply

text: of

text: job

text: vacancies

text: with

text: a

text: smoother

text: decrease

text: until

text: the

text: end

text: of

text: the

text: time

text: period,

text: a

text: statement

text: that

text: is

text: not

text: supported

text: by

text: the

text: Portuguese

text: real

text: Preprint

text: accepted

text: in

text: Statistics

text: Opt.

text: Inform.

text: Comput.

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: 30 LSTM ResidualLSTM DenseLSTM(5/block) DenseLSTM(10/block) 25 20 17.5 16 15 1 5 10 15 20 25 30 Number of LSTM Layers Figure 2: Structure of a dense TDNN-LSTM acoustic model. Figure 1: WER comparison between residual and dense con- Each layer outputs 1,024 dimensional non-linear activation nection for LSTMs with the cell dimension of 128. The systems vectors. were trained with a small portion of our entire training data, i.e., 300hr Switchboard-1 Release 2 (LDC97S62), and tested Table 1: WER comparison for TDNN-LSTMs with and without against the NIST 2000 Hub5 English evaluation set. dense connection. · Acoustic Model SWBD CH ering that H ( ) is a general composite function of operations in (cid:96) the (cid:96)th layer of a given neural network, a residual connectivity TDNN-LSTM 7.3% 13.8% for the output of the (cid:96)th layer, x , can be written as Dense TDNN-LSTM 7.3% 13.0% (cid:96) x = H (x − ) + x − , (1) (cid:96) (cid:96) (cid:96) 1 (cid:96) 1 curve might be the reason why most of the residual LSTMs for while a dense connectivity can be represented as speech recognition so far have been stuck with only up to 15 layers with marginal improvement in WER. In contrast, dense x = H ([x , x , · · ·x − ]), (2) LSTMs are shown continuously beneﬁted as more layers are (cid:96) (cid:96) 1 2 (cid:96) 1 added even after the 10th layer, further pulling the lowest possi- · · · ble WER down to around 16% at the number of LSTM layers of where [x1, x2, x(cid:96)−1−] is a concatenated vector of outputs 20 (light blue curve). There are a couple of notes on the dense from the ﬁrst layer to the ((cid:96) 1)th layer. The dense connectivity LSTMs experimented. Due to the connectivity pattern in Eq. pattern accommodates more direct connections throughout lay- (2) of concatenating vectors coming from the previous layers, ers while residual connections are only made between adjacent the dimension of an input vector for the (cid:96)th LSTM layer with the layers. More direct connections from dense connectivity could − × cell dimension of d would be ((cid:96) 1) d, which keeps increasing further help information ﬂow not completely vanished during as (cid:96) goes larger. Thus we made LSTM layers into blocks where back propagation in training deep neural networks. dense connections are applied only within the same block, and We propose densely connected LSTMs in this paper, in- linked blocks by a transitional layer. This dense block concept spired by the success of dense connection for CNNs. In speech has been also exploited in the original paper for densely con- recognition, there has been a limited number of effort to ex- nected CNNs [15], but with other purposes. The green curve ploit residual connection or its variants, e.g., highway con- in Figure 1 is based on the dense LSTMs where every group of nection, to LSTMs with minor differences in implementation 5 LSTM layers belong to one block while the light blue curve [19, 20, 21, 22], but none using dense connection yet. To comes from the dense LSTMs with 10 LSTM layers per block. peek how dense LSTMs would work as layers get deeper, let Dense connection can be easily applied to existing LSTM- us take a look at Figure 1. For the experiments, we trained (uni- based neural network architectures for speech recognition, directional) LSTMs with the cell dimension of 128 using a small thanks to a simple connectivity pattern and can improve perfor- portion of our entire training data, i.e., 300hr Switchboard- mance as we add more layers since it can help alleviate layer- 1 Release 2 corpus from LDC (LDC97S62), and tested them wise vanishing gradient effect. Based on our experimental val- against the NIST 2000 Hub5 English evaluation set (Switch- idation from Figure 1, we propose two dense LSTM architec- board and CallHome combined). The red curve indicates that tures for conversational speech recognition, which are detailed normal LSTMs would not obtain any beneﬁt after the 6th layer in the next subsections. where the lowest WER of 19.1% is reached. The performance 2.1. Dense TDNN-LSTM of residual LSTMs, depicted as the orange curve, seems further improved until the 10th layer where 17.7% is marked and then The ﬁrst proposed network with dense connection is dense continues to degrade thereafter as more layers are added. This TDNN-LSTM. It has the common network skeleton with the validates that residual learning makes LSTMs perform better model conﬁguration used for the Switchboard testset in [25], with more layers as has been reported in other research works, consisting of 7-layer time delay neural networks (TDNNs) com- but we also see that there is a clear limitation. The U-shaped bined with 3-layer LSTMs. The details of the model architec- )%( etaR rorrE droW ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: LSTM

text: ResidualLSTM

text: Number

text: of

text: LSTM

text: Layers

text: Figure

text: Structure

text: of

text: a

text: dense

text: TDNN-LSTM

text: acoustic

text: model.

text: Figure

text: WER

text: comparison

text: between

text: residual

text: and

text: dense

text: con-

text: Each

text: layer

text: outputs

text: dimensional

text: non-linear

text: activation

text: nection

text: for

text: LSTMs

text: with

text: the

text: cell

text: dimension

text: of

text: The

text: systems

text: vectors.

text: were

text: trained

text: with

text: a

text: small

text: portion

text: of

text: our

text: entire

text: training

text: data,

text: i.e.,

text: Release

text: and

text: tested

text: Table

text: WER

text: comparison

text: for

text: TDNN-LSTMs

text: with

text: and

text: without

text: against

text: the

text: NIST

text: English

text: evaluation

text: set.

text: dense

text: connection.

text: Acoustic

label: table

text: Model

text: SWBD

text: CH

text: ering

text: that

text: H

text: is

text: a

text: general

text: composite

text: function

text: of

text: operations

text: in

text: the

text: layer

text: of

text: a

text: given

text: neural

text: network,

text: a

text: residual

text: connectivity

text: TDNN-LSTM

text: for

text: the

text: output

text: of

text: the

text: layer,

text: x

text: can

text: be

text: written

text: as

text: Dense

text: TDNN-LSTM

text: x

text: H

text: (x

text: x

text: curve

text: might

text: be

text: the

text: reason

text: why

text: most

text: of

text: the

text: residual

text: LSTMs

text: for

text: while

text: a

text: dense

text: connectivity

text: can

text: be

text: represented

text: as

text: speech

text: recognition

text: so

text: far

text: have

text: been

text: stuck

text: with

text: only

text: up

text: to

text: layers

text: with

text: marginal

text: improvement

text: in

text: WER.

text: In

text: contrast,

text: dense

text: x

text: H

text: ([x

text: x

text: ·x

text: LSTMs

text: are

text: shown

text: continuously

text: beneﬁted

text: as

text: more

text: layers

text: are

text: added

text: even

text: after

text: the

text: layer,

text: further

text: pulling

text: the

text: lowest

text: possi-

text: ble

text: WER

text: down

text: to

text: around

text: at

text: the

text: number

text: of

text: LSTM

text: layers

text: of

text: where

text: is

text: a

text: concatenated

text: vector

text: of

text: outputs

text: (light

text: blue

text: curve).

text: There

text: are

text: a

text: couple

text: of

text: notes

text: on

text: the

text: dense

text: from

text: the

text: ﬁrst

text: layer

text: to

text: the

text: layer.

text: The

text: dense

text: connectivity

text: LSTMs

text: experimented.

text: Due

text: to

text: the

text: connectivity

text: pattern

text: in

text: Eq.

text: pattern

text: accommodates

text: more

text: direct

text: connections

text: throughout

text: lay-

text: of

text: concatenating

text: vectors

text: coming

text: from

text: the

text: previous

text: layers,

text: ers

text: while

text: residual

text: connections

text: are

text: only

text: made

text: between

text: adjacent

text: the

text: dimension

text: of

text: an

text: input

text: vector

text: for

text: the

text: LSTM

text: layer

text: with

text: the

text: layers.

text: More

text: direct

text: connections

text: from

text: dense

text: connectivity

text: could

text: cell

text: dimension

text: of

text: d

text: would

text: be

text: d,

text: which

text: keeps

text: increasing

text: further

text: help

text: information

text: ﬂow

text: not

text: completely

text: vanished

text: during

text: as

text: goes

text: larger.

text: Thus

text: we

text: made

text: LSTM

text: layers

text: into

text: blocks

text: where

text: back

text: propagation

text: in

text: training

text: deep

text: neural

text: networks.

text: dense

text: connections

text: are

text: applied

text: only

text: within

text: the

text: same

text: block,

text: and

text: We

text: propose

text: densely

text: connected

text: LSTMs

text: in

text: this

text: paper,

text: in-

text: linked

text: blocks

text: by

text: a

text: transitional

text: layer.

text: This

text: dense

text: block

text: concept

text: spired

text: by

text: the

text: success

text: of

text: dense

text: connection

text: for

text: CNNs.

text: In

text: speech

text: has

text: been

text: also

text: exploited

text: in

text: the

text: original

text: paper

text: for

text: densely

text: con-

text: recognition,

text: there

text: has

text: been

text: a

text: limited

text: number

text: of

text: effort

text: to

text: ex-

text: nected

text: CNNs

text: but

text: with

text: other

text: purposes.

text: The

text: green

text: curve

text: ploit

text: residual

text: connection

text: or

text: its

text: variants,

text: e.g.,

text: highway

text: con-

text: in

text: Figure

text: is

text: based

text: on

text: the

text: dense

text: LSTMs

text: where

text: every

text: group

text: of

text: nection,

text: to

text: LSTMs

text: with

text: minor

text: differences

text: in

text: implementation

text: LSTM

text: layers

text: belong

text: to

text: one

text: block

text: while

text: the

text: light

text: blue

text: curve

text: but

text: using

text: dense

text: connection

text: yet.

text: To

text: comes

text: from

text: the

text: dense

text: LSTMs

text: with

text: LSTM

text: layers

text: per

text: block.

text: peek

text: how

text: dense

text: LSTMs

text: would

text: work

text: as

text: layers

text: get

text: deeper,

text: let

text: Dense

text: connection

text: can

text: be

text: easily

text: applied

text: to

text: existing

text: LSTM-

text: us

text: take

text: a

text: look

text: at

text: Figure

text: For

text: the

text: experiments,

text: we

text: trained

text: (uni-

text: based

text: neural

text: network

text: architectures

text: for

text: speech

text: recognition,

text: directional)

text: LSTMs

text: with

text: the

text: cell

text: dimension

text: of

text: using

text: a

text: small

text: thanks

text: to

text: a

text: simple

text: connectivity

text: pattern

text: and

text: can

text: improve

text: perfor-

text: portion

text: of

text: our

text: entire

text: training

text: data,

text: i.e.,

text: Switchboard-

text: mance

text: as

text: we

text: add

text: more

text: layers

text: since

text: it

text: can

text: help

text: alleviate

text: layer-

text: Release

text: corpus

text: from

text: LDC

text: and

text: tested

text: them

text: wise

text: vanishing

text: gradient

text: effect.

text: Based

text: on

text: our

text: experimental

text: val-

text: against

text: the

text: NIST

text: English

text: evaluation

text: set

text: (Switch-

text: idation

text: from

text: Figure

text: we

text: propose

text: two

text: dense

text: LSTM

text: architec-

text: board

text: and

text: CallHome

text: combined).

text: The

text: red

text: curve

text: indicates

text: that

text: tures

text: for

text: conversational

text: speech

text: recognition,

text: which

text: are

text: detailed

text: normal

text: LSTMs

text: would

text: not

text: obtain

text: any

text: beneﬁt

text: after

text: the

text: layer

text: in

text: the

text: next

text: subsections.

text: where

text: the

text: lowest

text: WER

text: of

text: is

text: reached.

text: The

text: performance

text: Dense

text: TDNN-LSTM

text: of

text: residual

text: LSTMs,

text: depicted

text: as

text: the

text: orange

text: curve,

text: seems

text: further

text: improved

text: until

text: the

text: layer

text: where

text: is

text: marked

text: and

text: then

text: The

text: ﬁrst

text: proposed

text: network

text: with

text: dense

text: connection

text: is

text: dense

text: continues

text: to

text: degrade

text: thereafter

text: as

text: more

text: layers

text: are

text: added.

text: This

text: TDNN-LSTM.

text: It

text: has

text: the

text: common

text: network

text: skeleton

text: with

text: the

text: validates

text: that

text: residual

text: learning

text: makes

text: LSTMs

text: perform

text: better

text: model

text: conﬁguration

text: used

text: for

text: the

text: Switchboard

text: testset

text: in

text: with

text: more

text: layers

text: as

text: has

text: been

text: reported

text: in

text: other

text: research

text: works,

text: consisting

text: of

text: time

text: delay

text: neural

text: networks

text: (TDNNs)

text: com-

text: but

text: we

text: also

text: see

text: that

text: there

text: is

text: a

text: clear

text: limitation.

text: The

text: U-shaped

text: bined

text: with

text: LSTMs.

text: The

text: details

text: of

text: the

text: model

text: architec-

text: etaR

text: rorrE

text: droW

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: Table 5: Experimental evaluation in WER for the 8 individual systems and their combinations. The conﬁguration indexes (a), (b) and (c) for the dense CNN-bLSTMs are inherited from Table 2. d: LSTM cell dimension. SWBD CH Acoustic Model d Phoneset HES N-gram RNN N-gram RNN CNN-bLSTM 1,024 CMU 11 6.8% 5.9% 11.5% 10.7% CNN-bLSTM 1,024 MSU 11 6.8% 5.9% 11.3% 10.5% CNN-bLSTM 1,024 PronLex 11 6.7% 5.8% 11.6% 10.7% CNN-bLSTM 1,024 PronLex 2 6.4% 5.6% 11.4% 10.7% Dense CNN-bLSTM-(a) 512 PronLex 2 6.9% 6.0% 12.0% 11.5% Dense CNN-bLSTM-(b) 256 PronLex 2 7.1% 6.1% 11.9% 11.1% Dense CNN-bLSTM-(c) 512,256,128 PronLex 2 7.2% 6.1% 12.1% 11.2% Dense TDNN-LSTM 1,024 PronLex 2 7.2% 6.1% 12.1% 11.0% 4 CNN-bLSTMs Combined - - - - 5.2% - 9.5% 4 Dense LSTMs Combined - - - - 5.1% - 9.6% System Combination - - - - 5.0% - 9.1% hesitation modeling approaches in 11 hesitations versus 2 hesi- with the cell dimension of 1,024 has lower WER than any dense tations, which resulted in differently normalized transcripts for model with lesser cell dimension. As for the dense TDNN- model training. These RNN LMs are also expected to provide a LSTM model, its lower WER than those of the dense CNN- different level of diversity when combining the systems. bLSTMs (although the gap is not huge) can also be explained 4.3. System Combination by the larger cell dimension of 1,024 as compared to a max- imum of 512 in the dense CNN-bLSTMs, even though it has In order to combine the total 8 systems, we applied a lattice uni-directional LSTMs, not bi-directional. combination which conducts a union of lattices from individual In a system combination perspective, the proposed dense systems and searches the best path from the extended lattices models achieved a similar performance for SWBD and CH with using minimum Bayes risk decoding [37]. Due to the different the combined CNN-bLSTMs (5.1% vs 5.2% for SWBD & 9.6% mix ups across the systems in terms of hesitation modeling and vs 9.5% for CH), signiﬁcantly contributed our combined system phoneset, we had to relabel the word list of each individual sys- to reach the state-of-the-art performance of 5.0% and 9.1% in tem before the lattice combination. The combination weights WER for SWBD and CH, respectively. We achieved the same were found through a hyper-parameter optimization algorithm, results from the combination of the 5 systems as well, from the called a tree-structured Parzen estimator [38], using a held-out 3 CNN-bLSTMs with CMU, MSU and PronLex (with 2 hesita- development set. tion phones), dense CNN-bLSTM-(b) and dense TDNN-LSTM. 5. Experimental Results To date these are the best reported numbers on these two test- We evaluated the performance of the total 8 individual sys- sets. tems in 4 CNN-bLSTMs, 3 dense CNN-bLSTMs and 1 dense 6. Conclusions TDNN-LSTM across the three different phonesets against the Switchboard and CallHome testset of the NIST 2000 Hub5 En- In this paper we have proposed several densely connected glish evaluation set. The performance in terms of WER, before LSTM architectures, bringing the dense connectivity that was and after RNN LM rescoring, is shown in Table 5. successful for CNNs in image classiﬁcation tasks to the LSTM Among the acoustic models, the PronLex-based CNN- framework for conversational speech recognition. This allowed bLSTM with 2 hesitation phones outperformed the other mod- LSTMs to have more direct connections between layers such els, marking 5.6% for SWBD, while the MSU-based CNN- that layer-wise vanishing gradient effect is alleviated even as bLSTM with 11 hesitation phones reached the lowest 10.5% for more layers are stacked in a deep neural network model. Also CH. These two numbers are the best reported WERs achieved we have introduced parameter averaging for acoustic model by any single system so far in the industry. The CNN-bLSTM adaptation that averages the parameters of a seed neural net- models obtained WERs 0.1%-0.5% (absolute) for SWBD and work acoustic model and its adapted one, in order to balance 0.4%-1.0% (absolute) for CH lower than the dense acoustic out between domain adaptation and generalization. models. Having 2 hesitations when using the PronLex phone- We note that in a comaprison of the reported numbers of set appears to be a better choice to improve the robustness of 5.1% and 9.9% from [5] our combined system has made a sig- the CNN-bLSTM model, but we didn’t have the same pattern niﬁcant improvement on the CallHome portion of the NIST for the other phonesets in CMU and MSU (WERs not reported 2000 Hub5 English evaluation set, mainly due to the acoustic in the table). We observed that RNN LM rescoring provides model adaptation using the CallHome training data of approxi- consistent improvement in all the cases with absolute improve- mately 13 hours of speech. This shows that more domain spe- ments between 0.5% and 1.1%, and a maximum reduction in ciﬁc data which has similar acoustics and lexical information WER of up to 8% relative in the case of the dense TDNN-LSTM would have direct impact on performance improvement. As dis- model on SWBD (from 7.2% to 6.1%). cussed in [3], even the best conversational speech recognition As brieﬂy mentioned in Section 2.2, the cell dimension of system could suffer from higher error rates when it is tested LSTMs turns out to be a dominating factor to decide how the against real-world data with a number of unseen dynamics in models perform. It is noticeable that any CNN-bLSTM model data characteristics. To have systems more robust to unseen ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: Table

text: Experimental

text: evaluation

text: in

text: WER

text: for

text: the

text: individual

text: systems

text: and

text: their

text: combinations.

text: The

text: conﬁguration

text: indexes

text: (a),

text: (b)

text: and

text: (c)

text: for

text: the

text: dense

text: CNN-bLSTMs

text: are

text: inherited

text: from

text: Table

text: d:

text: LSTM

text: cell

text: dimension.

text: SWBD

text: CH

text: Acoustic

text: Model

text: d

text: Phoneset

text: HES

text: N-gram

text: RNN

text: N-gram

text: RNN

text: CNN-bLSTM

text: CMU

text: CNN-bLSTM

text: MSU

text: CNN-bLSTM

text: PronLex

text: CNN-bLSTM

text: PronLex

text: Dense

text: CNN-bLSTM-(a)

text: PronLex

text: Dense

text: CNN-bLSTM-(b)

text: PronLex

text: Dense

text: CNN-bLSTM-(c)

text: PronLex

text: Dense

text: TDNN-LSTM

text: PronLex

text: CNN-bLSTMs

text: Combined

text: Dense

text: LSTMs

text: Combined

text: System

text: Combination

text: hesitation

text: modeling

text: approaches

text: in

text: hesitations

text: versus

text: hesi-

text: with

text: the

text: cell

text: dimension

text: of

text: has

text: lower

text: WER

text: than

text: any

text: dense

text: tations,

text: which

text: resulted

text: in

text: differently

text: normalized

text: transcripts

text: for

text: model

text: with

text: lesser

text: cell

text: dimension.

text: As

text: for

text: the

text: dense

text: TDNN-

text: model

text: training.

text: These

text: RNN

text: LMs

text: are

text: also

text: expected

text: to

text: provide

text: a

text: LSTM

text: model,

text: its

text: lower

text: WER

text: than

text: those

text: of

text: the

text: dense

text: CNN-

text: different

text: level

text: of

text: diversity

text: when

text: combining

text: the

text: systems.

text: bLSTMs

text: (although

text: the

text: gap

text: is

text: not

text: huge)

text: can

text: also

text: be

text: explained

text: System

text: Combination

text: by

text: the

text: larger

text: cell

text: dimension

text: of

text: as

text: compared

text: to

text: a

text: max-

text: imum

text: of

text: in

text: the

text: dense

text: CNN-bLSTMs,

text: even

text: though

text: it

text: has

text: In

text: order

text: to

text: combine

text: the

text: total

text: systems,

text: we

text: applied

text: a

text: lattice

text: uni-directional

text: LSTMs,

text: not

text: bi-directional.

text: combination

text: which

text: conducts

text: a

text: union

text: of

text: lattices

text: from

text: individual

text: In

text: a

text: system

text: combination

text: perspective,

text: the

text: proposed

text: dense

text: systems

text: and

text: searches

text: the

text: best

text: path

text: from

text: the

text: extended

text: lattices

text: models

text: achieved

text: a

text: similar

text: performance

text: for

text: SWBD

text: and

text: CH

text: with

text: using

text: minimum

text: Bayes

text: risk

text: decoding

text: Due

text: to

text: the

text: different

text: the

text: combined

text: CNN-bLSTMs

text: vs

text: for

text: SWBD

text: mix

text: ups

text: across

text: the

text: systems

text: in

text: terms

text: of

text: hesitation

text: modeling

text: and

text: vs

text: for

text: CH),

text: signiﬁcantly

text: contributed

text: our

text: combined

text: system

text: phoneset,

text: we

text: had

text: to

text: relabel

text: the

text: word

text: list

text: of

text: each

text: individual

text: sys-

text: to

text: reach

text: the

text: state-of-the-art

text: performance

text: of

text: and

text: in

text: tem

text: before

text: the

text: lattice

text: combination.

text: The

text: combination

text: weights

text: WER

text: for

text: SWBD

text: and

text: CH,

text: respectively.

text: We

text: achieved

text: the

text: same

text: were

text: found

text: through

text: a

text: hyper-parameter

text: optimization

text: algorithm,

text: results

text: from

text: the

text: combination

text: of

text: the

text: systems

text: as

text: well,

text: from

text: the

text: called

text: a

text: tree-structured

text: Parzen

text: estimator

text: using

text: a

text: held-out

text: CNN-bLSTMs

text: with

text: CMU,

text: MSU

text: and

text: PronLex

text: (with

text: hesita-

text: development

text: set.

text: tion

text: phones),

text: dense

text: CNN-bLSTM-(b)

text: and

text: dense

text: TDNN-LSTM.

text: Experimental

text: Results

text: To

text: date

text: these

text: are

text: the

text: best

text: reported

text: numbers

text: on

text: these

text: two

text: test-

text: We

text: evaluated

text: the

text: performance

text: of

text: the

text: total

text: individual

text: sys-

text: sets.

text: tems

text: in

text: CNN-bLSTMs,

text: dense

text: CNN-bLSTMs

text: and

text: dense

text: Conclusions

text: TDNN-LSTM

text: across

text: the

text: three

text: different

text: phonesets

text: against

text: the

text: Switchboard

text: and

text: CallHome

text: testset

text: of

text: the

text: NIST

text: En-

text: In

text: this

text: paper

text: we

text: have

text: proposed

text: several

text: densely

text: connected

text: glish

text: evaluation

text: set.

text: The

text: performance

text: in

text: terms

text: of

text: WER,

text: before

text: LSTM

text: architectures,

text: bringing

text: the

text: dense

text: connectivity

text: that

text: was

text: and

text: after

text: RNN

text: LM

text: rescoring,

text: is

text: shown

text: in

text: Table

text: successful

text: for

text: CNNs

text: in

text: image

text: classiﬁcation

text: tasks

text: to

text: the

text: LSTM

text: Among

text: the

text: acoustic

text: models,

text: the

text: PronLex-based

text: CNN-

text: framework

text: for

text: conversational

text: speech

text: recognition.

text: This

text: allowed

text: bLSTM

text: with

text: hesitation

text: phones

text: outperformed

text: the

text: other

text: mod-

text: LSTMs

text: to

text: have

text: more

text: direct

text: connections

text: between

text: layers

text: such

text: els,

text: marking

text: for

text: SWBD,

text: while

text: the

text: MSU-based

text: CNN-

text: that

text: layer-wise

text: vanishing

text: gradient

text: effect

text: is

text: alleviated

text: even

text: as

text: bLSTM

text: with

text: hesitation

text: phones

text: reached

text: the

text: lowest

text: for

text: more

text: layers

text: are

text: stacked

text: in

text: a

text: deep

text: neural

text: network

text: model.

text: Also

text: CH.

text: These

text: two

text: numbers

text: are

text: the

text: best

text: reported

text: WERs

text: achieved

text: we

text: have

text: introduced

text: parameter

text: averaging

text: for

text: acoustic

text: model

text: by

text: any

text: single

text: system

text: so

text: far

text: in

text: the

text: industry.

text: The

text: CNN-bLSTM

text: adaptation

text: that

text: averages

text: the

text: parameters

text: of

text: a

text: seed

text: neural

text: net-

text: models

text: obtained

text: WERs

text: (absolute)

text: for

text: SWBD

text: and

text: work

text: acoustic

text: model

text: and

text: its

text: adapted

text: one,

text: in

text: order

text: to

text: balance

text: (absolute)

text: for

text: CH

text: lower

text: than

text: the

text: dense

text: acoustic

text: out

text: between

text: domain

text: adaptation

text: and

text: generalization.

text: models.

text: Having

text: hesitations

text: when

text: using

text: the

text: PronLex

text: phone-

text: We

text: note

text: that

text: in

text: a

text: comaprison

text: of

text: the

text: reported

text: numbers

text: of

text: set

text: appears

text: to

text: be

text: a

text: better

text: choice

text: to

text: improve

text: the

text: robustness

text: of

text: and

text: from

text: our

text: combined

text: system

text: has

text: made

text: a

text: sig-

text: the

text: CNN-bLSTM

text: model,

text: but

text: we

text: didn’t

text: have

text: the

text: same

text: pattern

text: niﬁcant

text: improvement

text: on

text: the

text: CallHome

text: portion

text: of

text: the

text: NIST

text: for

text: the

text: other

text: phonesets

text: in

text: CMU

text: and

text: MSU

text: (WERs

text: not

text: reported

text: English

text: evaluation

text: set,

text: mainly

text: due

text: to

text: the

text: acoustic

text: in

text: the

text: table).

text: We

text: observed

text: that

text: RNN

text: LM

text: rescoring

text: provides

text: model

text: adaptation

text: using

text: the

text: CallHome

text: training

text: data

text: of

text: approxi-

text: consistent

text: improvement

text: in

text: all

text: the

text: cases

text: with

text: absolute

text: improve-

text: mately

text: hours

text: of

text: speech.

text: This

text: shows

text: that

text: more

text: domain

text: spe-

text: ments

text: between

text: and

text: and

text: a

text: maximum

text: reduction

text: in

text: ciﬁc

text: data

text: which

text: has

text: similar

text: acoustics

text: and

text: lexical

text: information

text: WER

text: of

text: up

text: to

text: relative

text: in

text: the

text: case

text: of

text: the

text: dense

text: TDNN-LSTM

text: would

text: have

text: direct

text: impact

text: on

text: performance

text: improvement.

text: As

text: dis-

text: model

text: on

text: SWBD

text: (from

text: to

text: cussed

text: in

text: even

text: the

text: best

text: conversational

text: speech

text: recognition

text: As

text: brieﬂy

text: mentioned

text: in

text: Section

text: the

text: cell

text: dimension

text: of

text: system

text: could

text: suffer

text: from

text: higher

text: error

text: rates

text: when

text: it

text: is

text: tested

text: LSTMs

text: turns

text: out

text: to

text: be

text: a

text: dominating

text: factor

text: to

text: decide

text: how

text: the

text: against

text: real-world

text: data

text: with

text: a

text: number

text: of

text: unseen

text: dynamics

text: in

text: models

text: perform.

text: It

text: is

text: noticeable

text: that

text: any

text: CNN-bLSTM

text: model

text: data

text: characteristics.

text: To

text: have

text: systems

text: more

text: robust

text: to

text: unseen

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: testing conditions, given limited resources of audio data and [18] ——, “Highway networks,” in Proc of ICML 2015, 2015. the corresponding reference transcripts, unsupervised learning [19] Y. Zhang, G. Chen, D. Yu, K. Yao, S. Khudanpur, and J. Glass, that can continuously improve its recognition coverage would “Highway long short-term memory RNNs for distant speech be required. In addition, various testing materials beyond the recognition,” in Proc of ICASSP 2016, 2016, pp. 5755–5759. NIST 2000 Hub5 English evaluation set would be able to pro- [20] G. Pundak and T. Sainath, “Highway-LSTM and recurrent high- vide deeper insights on how systems could generalize against way networks for speech recognition,” in Proc of INTERSPEECH real-world data. 2017, 2017. 7. References [21] J. Kim, M. El-Khamy, and J. Lee, “Residual LSTM: Design of a deep recurrent architecture for distant speech recognition,” in [1] W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, and A. Stol- Proc of INTERSPEECH 2017, 2017. cke, “Achieving human parity in conversational speech recogni- [22] L. Huang, J. Sun, J. Xu, and Y. Yang, “An improved residual tion,” in MSR-TR-2016-71, 2016. LSTM architecture for acoustic modeling,” in Proc of ICCCS [2] G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dim- 2017, 2017. itriadis, X. Cui, B. Ramabhadran, M. Picheny, L. Lim, B. Roomi, [23] A. Krizhevsky and G. Hinton, “Learning multiple layers of fea- and P. Hall, “English conversational telephone speech recognition tures from tiny images,” in Tech Report, 2009. by humans and machines,” in Proc of INTERSPEECH 2017, 2017, pp. 132–136. [24] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in arXiv:1605.07146, 2009. [3] K. J. Han, S. Hahm, B. Kim, J. Kim, and I. Lane, “Deep learning- based telephony speech recognition in the wild,” in Proc of IN- [25] G. Cheng, V. Peddinti, D. Povey, V. Manohar, S. Khudanpur, and TERSPEECH 2017, 2017, pp. 1323–1327. Y. Yan, “An exploration of dropout with LSTMs,” in Proc of IN- TERSPEECH 2017, 2017. [4] W. Xiong, L. Wu, F. Alleva, J. Droppo, X. Huang, and A. Stolcke, “The Microsoft 2017 conversational speech recognition system,” [26] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, in MSR-TR-2017-39, 2017. N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, [5] G. Kurata, B. Ramabhadran, G. Saon, and A. Sethy, “Language J. Silovsky, G. Stemmer, and K. Vesely, “The Kaldi speech recog- modeling with highway LSTM,” in Proc of ASRU 2017, 2017. nition toolkit,” in Proc of ASRU 2011, 2011. [6] A. Stolcke and J. Droppo, “Comparing human and machine er- [27] D. Povey, V. Peddinti, G. Galvez, P. Ghahrmani, V. Manohar, rors in conversational speech transcription,” in Proc of INTER- X. Na, Y. Wang, and S. Khudanpur, “Purely sequence-trained neu- SPEECH 2017, 2017, pp. 137–141. ral networks for asr based on lattice-free MMI,” in Proc of INTER- SPEECH 2016, 2016. [7] B. Li and K. C. Sim, “Comparison of discriminative input and output transformations for speaker adaptation in the hybrid [28] M. J. F. Gales, “Maximum likelihood linear transformations NN/HMM systems,” in Proc of INTERSPEECH 2010, 2010. for HMM-based speech recognition,” Comp. Speech and Lang., vol. 12, pp. 75–98, 1997. [8] H. Liao, “Speaker adaptation of context dependent deep neural networks,” in Acoustics, Speech and Signal Processing (ICASSP), [29] D. Povey, X. Zhang, and K. Sanjeev, “Parallel training of DNNs 2013 IEEE International Conference on. IEEE, 2013, pp. 7947– with natural gradient and paramter averaging,” in Proc of ICLR 7951. 2015, 2015. [9] I. Himawan, P. Motlicek, M. F. Font, and S. Madikeri, “Towards [30] J. F. William, W. M. Fisher, A. F. Martin, M. A. Przybocki, and utterance-based neural network adaptation in acoustic modeling,” D. S. Pallett, “NIST evaluation of conversational speech recogni- in Proc of ASRU 2015. IEEE, 2015, pp. 289–295. tion over the telephone: English and mandarin performance re- sults,” in NIST, 2000. [10] L. Lu, “Sequence training and adaptation of highway deep neu- ral networks,” in Spoken Language Technology Workshop (SLT), [31] A. Stolcke, “SRILM – An extensible language modeling toolkit,” 2016 IEEE. IEEE, 2016, pp. 461–466. in Proc of ICSLP 2002, 2002, pp. 901–904. [11] N. Dehak, P. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, [32] A. Rousseau, P. Dele´glise, and Y. Esteve, “TED-LIUM: an auto- “Front-end factor analysis for speaker veriﬁcation,” IEEE Trans. matic speech recognition dedicated corpus.” in LREC, 2012, pp. Audio, Speech and Lang. Process., vol. 19, no. 4, pp. 788–797, 125–129. 2011. [33] P. Thompson and H. Nesi, “Research in progress, the British Aca- [12] G. Saon, H. Soltau, D. Namahoo, and M. Picheny, “Speaker adap- demic Spoken English (base) corpus project.” Language Teaching tation of neural network acoustic models using i-vectors,” in Proc Research, vol. 5, no. 3, 2001. of ASRU 2013, 2013, pp. 55–59. [34] R. C. Simpson, S. L. Briggs, J. Ovens, and J. M. Swales, “The [13] V. Gupta, P. Kenny, P. Ouellet, and T. Stafylakis, “I-vector-based Michigan Corpus of Academic Spoken English.” Ann Arbor, MI: speaker adaptation of deep neural networks for French broadcast The Regents of the University of Michigan, 2002. audio transcription,” in Proc of ICASSP 2014, 2014, pp. 6334– [35] X. Chen, X. Liu, Y. Qian, M. J. F. Gales, and P. C. Woodland, 6338. “CUED-RNNLM - An open-source toolkit for efﬁcient training [14] Y. Miao, L. Jiang, H. Zhang, and F. Metze, “Towards speaker and evaluation of recurrent neural network language models,” in adaptive training of deep neural network acoustic models,” in Proc Proc of ICASSP 2016, 2016, pp. 6000–6004. of INTERSPEECH 2014, 2014. [36] Y. Shi, W. Zhang, M. Cai, and J. Liu, “Variance regularization of [15] G. Huang, Z. Liu, K. Q. Weinberger, and L. van der RNNLM for speech recognition,” in Proc of ICASSP 2014, 2014. Maaten, “Densely connected convolutional networks,” in [37] H. Xu, D. Povey, L. Mangu, and J. Zhu, “Minimum bayes risk arXiv:1608.06993, 2016. decoding and system combination based on a recursion for edit [16] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for distance,” Comp. Speech and Lang., vol. 4, pp. 802–828, 2011. image recognition,” in Proc of CVPR 2016, 2016, pp. 770–778. [38] J. Bergstra, R. Bardenet, Y. Bengio, and B. Ke´gl, “Algorithms for [17] R. K. Srivastava, K. Greff, and J. Schmidhuber, “Training very hyper-parameter optimization,” in Proc of NIPS 2011, 2011. deep networks,” in Proc of NIPS 2015, 2015, pp. 2377–2385.

text: testing

text: conditions,

text: given

text: limited

text: resources

text: of

text: audio

text: data

text: and

label: reference

text: “Highway

text: networks,”

text: in

text: Proc

text: of

text: ICML

text: the

text: corresponding

text: reference

text: transcripts,

text: unsupervised

text: learning

text: Y.

text: Zhang,

text: G.

text: Chen,

text: D.

text: Yu,

text: K.

text: Yao,

text: S.

text: Khudanpur,

text: and

text: J.

text: Glass,

text: that

text: can

text: continuously

text: improve

text: its

text: recognition

text: coverage

text: would

text: “Highway

text: long

text: short-term

text: memory

text: RNNs

text: for

text: distant

text: speech

text: be

text: required.

text: In

text: addition,

text: various

text: testing

text: materials

text: beyond

text: the

text: recognition,”

text: in

text: Proc

text: of

text: ICASSP

text: pp.

text: NIST

text: English

text: evaluation

text: set

text: would

text: be

text: able

text: to

text: pro-

text: G.

text: Pundak

text: and

text: T.

text: Sainath,

text: “Highway-LSTM

text: and

text: recurrent

text: high-

text: vide

text: deeper

text: insights

text: on

text: how

text: systems

text: could

text: generalize

text: against

text: way

text: networks

text: for

text: speech

text: recognition,”

text: in

text: Proc

text: of

text: INTERSPEECH

text: real-world

text: data.

text: References

text: J.

text: Kim,

text: M.

text: El-Khamy,

text: and

text: J.

text: Lee,

text: “Residual

text: LSTM:

text: Design

text: of

text: a

text: deep

text: recurrent

text: architecture

text: for

text: distant

text: speech

text: recognition,”

text: in

text: W.

text: Xiong,

text: J.

text: Droppo,

text: X.

text: Huang,

text: F.

text: Seide,

text: M.

text: Seltzer,

text: and

text: A.

text: Stol-

text: Proc

text: of

text: INTERSPEECH

text: cke,

text: “Achieving

text: human

text: parity

text: in

text: conversational

text: speech

text: recogni-

text: L.

text: Huang,

text: J.

text: Sun,

text: J.

text: Xu,

text: and

text: Y.

text: Yang,

text: “An

text: improved

text: residual

text: tion,”

text: in

text: LSTM

text: architecture

text: for

text: acoustic

text: modeling,”

text: in

text: Proc

text: of

text: ICCCS

text: G.

text: Saon,

text: G.

text: Kurata,

text: T.

text: Sercu,

text: K.

text: Audhkhasi,

text: S.

text: Thomas,

text: D.

text: Dim-

text: itriadis,

text: X.

text: Cui,

text: B.

text: Ramabhadran,

text: M.

text: Picheny,

text: L.

text: Lim,

text: B.

text: Roomi,

text: A.

text: Krizhevsky

text: and

text: G.

text: Hinton,

text: “Learning

text: multiple

text: layers

text: of

text: fea-

text: and

text: P.

text: Hall,

text: “English

text: conversational

text: telephone

text: speech

text: recognition

text: tures

text: from

text: tiny

text: images,”

text: in

text: Tech

text: Report,

text: by

text: humans

text: and

text: machines,”

text: in

text: Proc

text: of

text: INTERSPEECH

text: pp.

text: S.

text: Zagoruyko

text: and

text: N.

text: Komodakis,

text: “Wide

text: residual

text: networks,”

text: in

text: K.

text: J.

text: Han,

text: S.

text: Hahm,

text: B.

text: Kim,

text: J.

text: Kim,

text: and

text: I.

text: Lane,

text: “Deep

text: learning-

text: based

text: telephony

text: speech

text: recognition

text: in

text: the

text: wild,”

text: in

text: Proc

text: of

text: IN-

text: G.

text: Cheng,

text: V.

text: Peddinti,

text: D.

text: Povey,

text: V.

text: Manohar,

text: S.

text: Khudanpur,

text: and

text: TERSPEECH

text: pp.

text: Y.

text: Yan,

text: “An

text: exploration

text: of

text: dropout

text: with

text: LSTMs,”

text: in

text: Proc

text: of

text: IN-

text: TERSPEECH

text: W.

text: Xiong,

text: L.

text: Wu,

text: F.

text: Alleva,

text: J.

text: Droppo,

text: X.

text: Huang,

text: and

text: A.

text: Stolcke,

text: “The

text: Microsoft

text: conversational

text: speech

text: recognition

text: system,”

text: D.

text: Povey,

text: A.

text: Ghoshal,

text: G.

text: Boulianne,

text: L.

text: Burget,

text: O.

text: Glembek,

text: in

text: N.

text: Goel,

text: M.

text: Hannemann,

text: P.

text: Motlicek,

text: Y.

text: Qian,

text: P.

text: Schwarz,

text: G.

text: Kurata,

text: B.

text: Ramabhadran,

text: G.

text: Saon,

text: and

text: A.

text: Sethy,

text: “Language

text: J.

text: Silovsky,

text: G.

text: Stemmer,

text: and

text: K.

text: Vesely,

text: “The

text: Kaldi

text: speech

text: recog-

text: modeling

text: with

text: highway

text: LSTM,”

text: in

text: Proc

text: of

text: ASRU

text: nition

text: toolkit,”

text: in

text: Proc

text: of

text: ASRU

text: A.

text: Stolcke

text: and

text: J.

text: Droppo,

text: “Comparing

text: human

text: and

text: machine

text: er-

text: D.

text: Povey,

text: V.

text: Peddinti,

text: G.

text: Galvez,

text: P.

text: Ghahrmani,

text: V.

text: Manohar,

text: rors

text: in

text: conversational

text: speech

text: transcription,”

text: in

text: Proc

text: of

text: INTER-

text: X.

text: Na,

text: Y.

text: Wang,

text: and

text: S.

text: Khudanpur,

text: “Purely

text: sequence-trained

text: neu-

text: SPEECH

text: pp.

text: ral

text: networks

text: for

text: asr

text: based

text: on

text: lattice-free

text: MMI,”

text: in

text: Proc

text: of

text: INTER-

text: SPEECH

text: B.

text: Li

text: and

text: K.

text: C.

text: Sim,

text: “Comparison

text: of

text: discriminative

text: input

text: and

text: output

text: transformations

text: for

text: speaker

text: adaptation

text: in

text: the

text: hybrid

text: M.

text: J.

text: F.

text: Gales,

text: “Maximum

text: likelihood

text: linear

text: transformations

text: NN/HMM

text: systems,”

text: in

text: Proc

text: of

text: INTERSPEECH

text: for

text: HMM-based

text: speech

text: recognition,”

text: Comp.

text: Speech

text: and

text: Lang.,

text: vol.

text: pp.

text: H.

text: Liao,

text: “Speaker

text: adaptation

text: of

text: context

text: dependent

text: deep

text: neural

text: networks,”

text: in

text: Acoustics,

text: Speech

text: and

text: Signal

text: Processing

text: (ICASSP),

text: D.

text: Povey,

text: X.

text: Zhang,

text: and

text: K.

text: Sanjeev,

text: “Parallel

text: training

text: of

text: DNNs

text: IEEE

text: International

text: Conference

text: on.

text: IEEE,

text: pp.

text: with

text: natural

text: gradient

text: and

text: paramter

text: averaging,”

text: in

text: Proc

text: of

text: ICLR

text: I.

text: Himawan,

text: P.

text: Motlicek,

text: M.

text: F.

text: Font,

text: and

text: S.

text: Madikeri,

text: “Towards

text: J.

text: F.

text: William,

text: W.

text: M.

text: Fisher,

text: A.

text: F.

text: Martin,

text: M.

text: A.

text: Przybocki,

text: and

text: utterance-based

text: neural

text: network

text: adaptation

text: in

text: acoustic

text: modeling,”

text: D.

text: S.

text: Pallett,

text: “NIST

text: evaluation

text: of

text: conversational

text: speech

text: recogni-

text: in

text: Proc

text: of

text: ASRU

text: IEEE,

text: pp.

text: tion

text: over

text: the

text: telephone:

text: English

text: and

text: mandarin

text: performance

text: re-

text: sults,”

text: in

text: NIST,

text: L.

text: Lu,

text: “Sequence

text: training

text: and

text: adaptation

text: of

text: highway

text: deep

text: neu-

text: ral

text: networks,”

text: in

text: Spoken

text: Language

text: Technology

text: Workshop

text: (SLT),

text: A.

text: Stolcke,

text: “SRILM

text: An

text: extensible

text: language

text: modeling

text: toolkit,”

text: IEEE.

text: IEEE,

text: pp.

text: in

text: Proc

text: of

text: ICSLP

text: pp.

text: N.

text: Dehak,

text: P.

text: Kenny,

text: R.

text: Dehak,

text: P.

text: Dumouchel,

text: and

text: P.

text: Ouellet,

text: A.

text: Rousseau,

text: P.

text: Dele´glise,

text: and

text: Y.

text: Esteve,

text: “TED-LIUM:

text: an

text: auto-

text: “Front-end

text: factor

text: analysis

text: for

text: speaker

text: veriﬁcation,”

text: IEEE

text: Trans.

text: matic

text: speech

text: recognition

text: dedicated

text: corpus.”

text: in

text: LREC,

text: pp.

text: Audio,

text: Speech

text: and

text: Lang.

text: Process.,

text: vol.

text: no.

text: pp.

text: P.

text: Thompson

text: and

text: H.

text: Nesi,

text: “Research

text: in

text: progress,

text: the

text: British

text: Aca-

text: G.

text: Saon,

text: H.

text: Soltau,

text: D.

text: Namahoo,

text: and

text: M.

text: Picheny,

text: “Speaker

text: adap-

text: demic

text: Spoken

text: English

text: (base)

text: corpus

text: project.”

text: Language

text: Teaching

text: tation

text: of

text: neural

text: network

text: acoustic

text: models

text: using

text: i-vectors,”

text: in

text: Proc

text: Research,

text: vol.

text: no.

text: of

text: ASRU

text: pp.

text: R.

text: C.

text: Simpson,

text: S.

text: L.

text: Briggs,

text: J.

text: Ovens,

text: and

text: J.

text: M.

text: Swales,

text: “The

text: V.

text: Gupta,

text: P.

text: Kenny,

text: P.

text: Ouellet,

text: and

text: T.

text: Stafylakis,

text: “I-vector-based

text: Michigan

text: Corpus

text: of

text: Academic

text: Spoken

text: English.”

text: Ann

text: Arbor,

text: MI:

text: speaker

text: adaptation

text: of

text: deep

text: neural

text: networks

text: for

text: French

text: broadcast

text: The

text: Regents

text: of

text: the

text: University

text: of

text: Michigan,

text: audio

text: transcription,”

text: in

text: Proc

text: of

text: ICASSP

text: pp.

text: X.

text: Chen,

text: X.

text: Liu,

text: Y.

text: Qian,

text: M.

text: J.

text: F.

text: Gales,

text: and

text: P.

text: C.

text: Woodland,

text: “CUED-RNNLM

text: An

text: open-source

text: toolkit

text: for

text: efﬁcient

text: training

text: Y.

text: Miao,

text: L.

text: Jiang,

text: H.

text: Zhang,

text: and

text: F.

text: Metze,

text: “Towards

text: speaker

text: and

text: evaluation

text: of

text: recurrent

text: neural

text: network

text: language

text: models,”

text: in

text: adaptive

text: training

text: of

text: deep

text: neural

text: network

text: acoustic

text: models,”

text: in

text: Proc

text: Proc

text: of

text: ICASSP

text: pp.

text: of

text: INTERSPEECH

text: Y.

text: Shi,

text: W.

text: Zhang,

text: M.

text: Cai,

text: and

text: J.

text: Liu,

text: “Variance

text: regularization

text: of

text: G.

text: Huang,

text: Z.

text: Liu,

text: K.

text: Q.

text: Weinberger,

text: and

text: L.

text: van

text: der

text: RNNLM

text: for

text: speech

text: recognition,”

text: in

text: Proc

text: of

text: ICASSP

text: Maaten,

text: “Densely

text: connected

text: convolutional

text: networks,”

text: in

text: H.

text: Xu,

text: D.

text: Povey,

text: L.

text: Mangu,

text: and

text: J.

text: Zhu,

text: “Minimum

text: bayes

text: risk

text: decoding

text: and

text: system

text: combination

text: based

text: on

text: a

text: recursion

text: for

text: edit

text: K.

text: He,

text: X.

text: Zhang,

text: S.

text: Ren,

text: and

text: J.

text: Sun,

text: “Deep

text: residual

text: learning

text: for

text: distance,”

text: Comp.

text: Speech

text: and

text: Lang.,

text: vol.

text: pp.

text: image

text: recognition,”

text: in

text: Proc

text: of

text: CVPR

text: pp.

text: J.

text: Bergstra,

text: R.

text: Bardenet,

text: Y.

text: Bengio,

text: and

text: B.

text: Ke´gl,

text: “Algorithms

text: for

text: R.

text: K.

text: Srivastava,

text: K.

text: Greff,

text: and

text: J.

text: Schmidhuber,

text: “Training

text: very

text: hyper-parameter

text: optimization,”

text: in

text: Proc

text: of

text: NIPS

text: deep

text: networks,”

text: in

text: Proc

text: of

text: NIPS

text: pp.


text: Figure 1: Regions in the (t , t )-plane occurring in the extension of a function χ on P(t )2 1 2 \ × 0 to P(t + dt). R corresponds to P(t )2, R to (P(t + dt) P(t )) P(t ), R to ×0 0 \ 0 1 \ 0 0 0 2 P(t ) (P(t + dt) P(t )) and R to (P(t + dt) P(t ))2. 0 0 0 3 0 0 ≤ ≤ ∈ (E3). For 0 dt , dt dt, i.e., (t , t ) R : 1 2 1 2 3 − ψ(t + dt , x , t + dt , x ) = χ(t , x , t , x ) i[(Hfree + L )]χ(t , x , t , x )dt 0 1 1 0 2 2 0 1 0 2 1 1 0 1 0 2 1 − i[(Hfree + L )]χ(t , x , t , x )dt . (56) 2 2 0 1 0 2 2 Note that a solution of (50) clearly satisﬁes (E1)-(E3), so these prescriptions are necessary. We now show that a function ψ constructed via (E1)-(E3) actually yields a conditional solution on P(t + dt)2. To this end, we decompose P(t + dt)2 in four parts (three 0 0 new ones besides P(t )2; see also Fig. 1) and show for each of them that (some of) 0 the multi-time equations are satisﬁed (statements (S1)-(S3)). In addition, we extract the condition under which the following holds (statements (S4), (S5)). If the given function χ is a solution on P(t )2, then ψ also solves (50) on P(t + dt)2. 0 0 \ × (S1). On (P(t + dt) P(t )) P(t ), the ﬁrst equation of (50) holds. 0 0 0 × \ (S2). On P(t ) (P(t + dt) P(t )), the second equation of (50) holds. 0 0 0 \ (S3). On (P(t + dt) (cid:0)P(t ))2, bot(cid:1)h multi-time equations (50) are satisﬁed. 0 0 (S4). If − i∂ Hfree χ(t , x , t , x ) = (L χ)(t , x , t , x ) (57) (cid:0) t2 (cid:1) 2 0 1 2 2 2 0 1 2 2 holds for t < t , then we also have 2 0 − i∂ Hfree ψ(t + dt , x , t , x ) = (L ψ)(t + dt , x , t , x ). (58) t2 2 0 1 1 2 2 2 0 1 1 2 2 That is, (E1) is compatible with the second equation of (50). (cid:0) (cid:1) (S5). Analogously, if − i∂ Hfree χ(t , x , t + dt , x ) = (L χ)(t , x , t + dt , x ) (59) (cid:0) t1 1 (cid:1) 1 1 0 2 2 1 1 1 0 2 2 for t < t , then also 1 0 − i∂ Hfree ψ(t , x , t + dt , x ) = (L ψ)(t , x , t + dt , x ), (60) t1 1 1 1 0 2 2 1 1 1 0 2 2 i.e., (E2) is compatible with the ﬁrst equation of (50). 17 ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: Figure

text: Regions

text: in

text: the

text: (t

text: t

text: )-plane

text: occurring

text: in

text: the

text: extension

text: of

text: a

text: function

text: χ

text: on

text: P(t

text: to

text: P(t

text: dt).

text: R

text: corresponds

text: to

text: P(t

text: R

text: to

text: (P(t

text: dt)

text: P(t

text: P(t

text: R

text: to

text: ×0

text: P(t

text: (P(t

text: dt)

text: P(t

text: and

text: R

text: to

text: (P(t

text: dt)

text: P(t

label: list

text: For

text: dt

text: dt

text: dt,

text: i.e.,

text: (t

text: t

text: R

text: ψ(t

text: dt

text: x

text: t

text: dt

text: x

text: χ(t

text: x

text: t

text: x

text: i[(Hfree

text: L

text: )]χ(t

text: x

text: t

text: x

text: )dt

text: i[(Hfree

text: L

text: )]χ(t

text: x

text: t

text: x

text: )dt

text: Note

text: that

text: a

text: solution

text: of

text: clearly

text: satisﬁes

text: so

text: these

text: prescriptions

text: are

text: necessary.

text: We

text: now

text: show

text: that

text: a

text: function

text: ψ

text: constructed

text: via

text: actually

text: yields

text: a

text: conditional

text: solution

text: on

text: P(t

text: To

text: this

text: end,

text: we

text: decompose

text: P(t

text: in

text: four

text: parts

text: (three

text: new

text: ones

text: besides

text: P(t

text: see

text: also

text: Fig.

text: and

text: show

text: for

text: each

text: of

text: them

text: that

text: (some

text: of)

text: the

text: multi-time

text: equations

text: are

text: satisﬁed

text: (statements

text: In

text: addition,

text: we

text: extract

text: the

text: condition

text: under

text: which

text: the

text: following

text: holds

text: (statements

text: If

text: the

text: given

text: function

text: χ

text: is

text: a

text: solution

text: on

text: P(t

text: then

text: ψ

text: also

text: solves

text: on

text: P(t

text: On

text: (P(t

text: dt)

text: P(t

text: P(t

text: the

text: ﬁrst

text: equation

text: of

text: holds.

text: On

text: P(t

text: (P(t

text: dt)

text: P(t

text: the

text: second

text: equation

text: of

text: holds.

text: On

text: (P(t

text: dt)

text: multi-time

text: equations

text: are

text: satisﬁed.

text: If

text: i∂

text: Hfree

text: χ(t

text: x

text: t

text: x

text: (L

text: χ)(t

text: x

text: t

text: x

text: holds

text: for

text: t

text: t

text: then

text: we

text: also

text: have

text: i∂

text: Hfree

text: ψ(t

text: dt

text: x

text: t

text: x

text: (L

text: ψ)(t

text: dt

text: x

text: t

text: x

text: That

text: is,

text: is

text: compatible

text: with

text: the

text: second

text: equation

text: of

text: Analogously,

text: if

text: i∂

text: Hfree

text: χ(t

text: x

text: t

text: dt

text: x

text: (L

text: χ)(t

text: x

text: t

text: dt

text: x

text: for

text: t

text: t

text: then

text: also

text: i∂

text: Hfree

text: ψ(t

text: x

text: t

text: dt

text: x

text: (L

text: ψ)(t

text: x

text: t

text: dt

text: x

text: i.e.,

text: is

text: compatible

text: with

text: the

text: ﬁrst

text: equation

text: of

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##