text: approach in data analysis is that it enables one to understand the eﬀects of systematic uncertainties in the measurement more clearly. The eﬀects from systematic uncertainty are expressed explicitly as the second term in Eq. (3) (χ2 ) while the ﬁrst term represents sys a measurement of diﬀerence between observations and theoretical predictions (χ2 ) obs The parameters in Eq. (3) include parameters needed for theoretical predictions which are of physics interest and pull factors. The number of pull factors introduced in Eq. (3) is normally more than the number of parameters needed in theoretical calculations. Al- though all pull factors are determined in principle when all uncertainties are known, in practice, the minimization of χ2 is carried out in the parameter space including param- pull eter set for the theory and the pull factors. When the number of data points is less than the number of parameters, as in the cases for the Daya Bay and RENO experiments, the variational minimization usually will result in multiple sets of parameters that all give similar value for the χ2, i.e. having same level of goodness-of-ﬁt. One needs to rely on other information implied in the design of χ2 to distinguish these ﬁts. pull Apart from goodness-of-ﬁt test, there are other requirements for a (an) good/acceptable ﬁt when using the pull method. 1. All pull factors are small since the pull factors are Gaussian random variables with (cid:104) (cid:105) (cid:104) (cid:105) the expectations ξ = 0 and ξ2 = 1. k k 2. χ2 << χ2 . A large value of χ2 means one or more systematic uncertainties sys obs sys dominates in achieving an agreement between the data and theoretical calculations. A violation of either condition usually indicates a great tension between experimen- tal measurements and theoretical predictions. We consider a ﬁt that might pass the ∼ goodness-of-ﬁt test (χ d.o.f) but violates one or both above requirements is not ac- ceptable. Even after applying the goodness-of-ﬁt test and the above two criteria, there are still multiple acceptable ﬁts depending on the minimization algorithm and/or initial guessed values for the parameters. This dependence should be insigniﬁcant for the quantities of interest. To evaluate uncertainties associated with the existence of multiple acceptable 6

text: approach

text: in

text: data

text: analysis

text: is

text: that

text: it

text: enables

text: one

text: to

text: understand

text: the

text: eﬀects

text: of

text: systematic

text: uncertainties

text: in

text: the

text: measurement

text: more

text: clearly.

text: The

text: eﬀects

text: from

text: systematic

text: uncertainty

text: are

text: expressed

text: explicitly

text: as

text: the

text: second

text: term

text: in

text: Eq.

text: while

text: the

text: ﬁrst

text: term

text: represents

text: sys

text: a

text: measurement

text: of

text: diﬀerence

text: between

text: observations

text: and

text: theoretical

text: predictions

text: obs

text: The

text: parameters

text: in

text: Eq.

text: include

text: parameters

text: needed

text: for

text: theoretical

text: predictions

text: which

text: are

text: of

text: physics

text: interest

text: and

text: pull

text: factors.

text: The

text: number

text: of

text: pull

text: factors

text: introduced

text: in

text: Eq.

text: is

text: normally

text: more

text: than

text: the

text: number

text: of

text: parameters

text: needed

text: in

text: theoretical

text: calculations.

text: Al-

text: though

text: all

text: pull

text: factors

text: are

text: determined

text: in

text: principle

text: when

text: all

text: uncertainties

text: are

text: known,

text: in

text: practice,

text: the

text: minimization

text: of

text: χ2

text: is

text: carried

text: out

text: in

text: the

text: parameter

text: space

text: including

text: param-

text: pull

text: eter

text: set

text: for

text: the

text: theory

text: and

text: the

text: pull

text: factors.

text: When

text: the

text: number

text: of

text: data

text: points

text: is

text: less

text: than

text: the

text: number

text: of

text: parameters,

text: as

text: in

text: the

text: cases

text: for

text: the

text: Daya

text: Bay

text: and

text: RENO

text: experiments,

text: the

text: variational

text: minimization

text: usually

text: will

text: result

text: in

text: multiple

text: sets

text: of

text: parameters

text: that

text: all

text: give

text: similar

text: value

text: for

text: the

text: i.e.

text: having

text: same

text: level

text: of

text: goodness-of-ﬁt.

text: One

text: needs

text: to

text: rely

text: on

text: other

text: information

text: implied

text: in

text: the

text: design

text: of

text: χ2

text: to

text: distinguish

text: these

text: ﬁts.

text: pull

text: Apart

text: from

text: goodness-of-ﬁt

text: test,

text: there

text: are

text: other

text: requirements

text: for

text: a

text: (an)

text: good/acceptable

text: ﬁt

text: when

text: using

text: the

text: pull

text: method.

text: All

text: pull

text: factors

text: are

text: small

text: since

text: the

text: pull

text: factors

text: are

text: Gaussian

text: random

text: variables

text: with

text: the

text: expectations

text: ξ

text: and

text: ξ2

text: k

text: k

text: χ2

text: χ2

text: A

text: large

text: value

text: of

text: χ2

text: means

text: one

text: or

text: more

text: systematic

text: uncertainties

text: sys

text: obs

text: sys

text: dominates

text: in

text: achieving

text: an

text: agreement

text: between

text: the

text: data

text: and

text: theoretical

text: calculations.

text: A

text: violation

text: of

text: either

text: condition

text: usually

text: indicates

text: a

text: great

text: tension

text: between

text: experimen-

text: tal

text: measurements

text: and

text: theoretical

text: predictions.

text: We

text: consider

text: a

text: ﬁt

text: that

text: might

text: pass

text: the

text: goodness-of-ﬁt

text: test

text: (χ

text: d.o.f)

text: but

text: violates

text: one

text: or

text: both

text: above

text: requirements

text: is

text: not

text: ac-

text: ceptable.

text: Even

text: after

text: applying

text: the

text: goodness-of-ﬁt

text: test

text: and

text: the

text: above

text: two

text: criteria,

text: there

text: are

text: still

text: multiple

text: acceptable

text: ﬁts

text: depending

text: on

text: the

text: minimization

text: algorithm

text: and/or

text: initial

text: guessed

text: values

text: for

text: the

text: parameters.

text: This

text: dependence

text: should

text: be

text: insigniﬁcant

text: for

text: the

text: quantities

text: of

text: interest.

text: To

text: evaluate

text: uncertainties

text: associated

text: with

text: the

text: existence

text: of

text: multiple

text: acceptable


text: TABLE I: Results of random input ﬁts with Eq. (4) × − Parameter χ2 sin2(2θ ) (cid:15)( 10 4) 13 − Mean 3.872 0.08844 2.3 90%CI 0.073 0.00025 2.7 Parameter ρ ρ ρ ρ ρ ρ 1 2 3 4 5 6 × − − − − Mean ( 10 4) 4.90 6.45 1.36 4.573 0.24 4.335 × − 90%CI ( 10 4) 0.35 0.07 0.42 0.039 0.14 0.031 Parameter α α α α α α 1 2 3 4 5 6 × − − − Mean ( 10 4) 6.7 7.96 7.8 4.01 8.4 4.72 × − 90%CI ( 10 4) 4.9 0.54 4.6 0.88 5.5 0.73 Parameter η η η η η η 1 2 3 4 5 6 × − − Mean ( 10 1) 2.0 1.2 1.8 4.2 1.5 4.5 × − 90%CI ( 10 1) 1.5 2.4 2.6 1.6 3.8 2.1 2 2 Contour minimization of χ on the sin (2θ ), ǫ plane 13 0.01 0.005 0 -0.005 -0.01 0.065 0.07 0.075 0.08 0.085 0.09 0.095 0.1 0.105 0.11 0.115 2 sin (2θ ) 13 FIG. 1: (Color on line) Contour plot of the sin2(2θ )-(cid:15) plane for Eq. (4). Each contour line 13 represents 1σ. The red x is the minimum. 10 ǫ ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: TABLE

label: paragraph

text: I:

text: Results

label: caption

text: of

text: random

text: input

text: ﬁts

text: with

text: Eq.

label: table

text: Parameter

text: χ2

text: Mean

text: Parameter

text: ρ

text: ρ

text: ρ

text: ρ

text: ρ

text: ρ

text: Mean

text: Parameter

text: α

text: α

text: α

text: α

text: α

text: α

text: Mean

text: Parameter

text: η

text: η

text: η

text: η

text: η

text: η

text: Mean

text: Contour

text: minimization

text: of

text: χ

text: on

text: the

text: sin

text: ǫ

text: plane

text: sin

text: FIG.

text: (Color

text: on

text: line)

text: Contour

text: plot

text: of

text: the

text: plane

text: for

text: Eq.

text: Each

text: contour

text: line

text: represents

text: The

text: red

text: x

text: is

text: the

text: minimum.

text: ǫ

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: SPEECH DEREVERBERATION BASED ON INTEGRATED DEEP AND ENSEMBLE LEARNING ALGORITHM Wei-Jen Lee1, Syu-Siang Wang1, Fei Chen2, Xugang Lu3, Shao-Yi Chien4 and Yu Tsao1 1Research Center for Information Technology Innovation, Academia Sinica, Taiwan 2Department of Electrical and Electronic Engineering, Southern University of Science and Technology, China 3National Institute of Information and Communications Technology, Japan 4Department of Electrical Engineering, National Taiwan University, Taiwan ABSTRACT model-based method, which estimates the clean signal by employ- ing the priori knowledge about time–frequency speech structures Reverberation, which is generally caused by sound reﬂections from [10–13]. The second category is the homomorphic ﬁltering tech- walls, ceilings, and ﬂoors, can result in severe performance degrada- nique, which adopts a homomorphic transformation to decompose tion of acoustic applications. Due to a complicated combination of the reverberant signal from the time domain to the cepstral domain, attenuation and time-delay effects, the reverberation property is dif- and thus separates the reverberation from the input cepstral coefﬁ- ﬁcult to characterize, and it remains a challenging task to effectively cients with a simple subtraction operation [14]. Channel-inversion retrieve the anechoic speech signals from reverberation ones. In the methods belong to the third category, which considers the reverber- present study, we proposed a novel integrated deep and ensemble ation as a convolution of the original sound with the room impulse learning algorithm (IDEA) for speech dereverberation. The IDEA response (RIR) and thereby performs an inverse ﬁltering to decon- consists of ofﬂine and online phases. In the ofﬂine phase, we train volve the captured signal [15–20]. Even though the above three cat- multiple dereverberation models, each aiming to precisely dereverb egories of approaches have been shown to provide satisfactory per- speech signals in a particular acoustic environment; then a uniﬁed formance, they usually require an accurate estimation of time-varied fusion function is estimated that aims to integrate the information RIR, which may not always be accessible in practice [21]. of multiple dereverberation models. In the online phase, an input Recently, deep neural network (DNN) models, which show utterance is ﬁrst processed by each of the dereverberation models. strong regression capabilities, have been used to address the speech The outputs of all models are integrated accordingly to generate the dereverberation issue [21, 22]. The main concept here is to use a ﬁnal anechoic signal. We evaluated the IDEA on designed acoustic DNN model to characterize the non-linear spectral mapping from environments, including both matched and mismatched conditions reverberant to anechoic speech in the training stage. In the testing of the training and testing data. Experimental results conﬁrm that stage, the trained DNN model is used to generate dereverbed ut- the proposed IDEA outperforms single deep-neural-network-based terances given the input reverberant signals. The same concept has dereverberation model with the same model architecture and train- been applied to perform denoising and dereverberation simultane- ing data. ously [6]. Despite providing notable improvements over traditional Index Terms— Deep neural networks, Speech dereverberation, algorithms, DNN-based dereverberation methods achieve the opti- Ensemble learning, Convolutional neural networks, Deep denoising mal performance only in matched training and testing reverberant autoencoder conditions. To further improve the performance, an environment- aware DNN-based dereverberation system has been proposed, which 1. INTRODUCTION selects the optimal DNN models online to perform dereverbera- tion [23]. In realistic environments, the perceived speech signal may com- Contrary to the idea used in [23], the present study extends the prise of the original speech and multiple copies of the attenuated and previous work on the deep denoise autoencoder (DDAE) in speech time-delayed signals [1]. The combination of these signals can cause enhancement [24,25] and proposes a novel integrated deep and en- serious performance degradation of speech-related applications. For semble learning algorithm (IDEA) for speech dereverberation. The example, distant-talking speech signiﬁcantly degrades the perfor- IDEA consists of ofﬂine and online phases. In the ofﬂine phase, mul- mance of automatic speech recognition (ASR) [2, 3] and speaker tiple DDAE-based dereverberation models are prepared, with each identiﬁcation [4,5]. Meanwhile, the adverse effects of reverberation aiming to precisely dereverb speech signals in a particular acoustic will lower sound quality and intelligibility for both hearing-impaired environment. Then, a uniﬁed fusion model is estimated to integrate and normal-hearing listeners [6–8]. In the past, various speech dere- the information of the multiple dereverberation models with the aim verberation methods have been developed. The goal of these meth- to estimate clean speech. In the online phase, an input reverber- ods is to extract anechoic speech signals from reverberant ones to ant speech is ﬁrst processed by all dereverberation models simul- enhance the performance of speech-related applications and to im- taneously, and the outputs are integrated to ultimately generate the prove sound quality and intelligibility simultaneously for listeners in anechoic signals. The ensemble learning strategy, which has been reverberant environments. proven to be able to improve system performance in speech enhance- Traditional speech dereverberation methods can be roughly di- ment [25] and ASR [26, 27], is adopted in the task to increase the vided into three categories [9]. The ﬁrst category is the source- generalization ability of DDAEs. As will be introduced in the re-

text: SPEECH

text: DEREVERBERATION

text: BASED

text: ON

text: INTEGRATED

text: DEEP

text: AND

text: ENSEMBLE

text: LEARNING

text: ALGORITHM

text: Wei-Jen

text: Syu-Siang

text: Fei

text: Xugang

text: Shao-Yi

text: and

text: Yu

label: author

text: Center

text: for

text: Information

text: Technology

text: Innovation,

text: Academia

text: Sinica,

text: Taiwan

text: of

text: Electrical

text: and

text: Electronic

text: Engineering,

text: Southern

text: University

text: of

text: Science

text: and

text: Technology,

text: China

text: Institute

text: of

text: Information

text: and

text: Communications

text: Technology,

text: Japan

text: of

text: Electrical

text: Engineering,

text: National

text: Taiwan

text: University,

text: Taiwan

text: ABSTRACT

text: model-based

text: method,

text: which

text: estimates

text: the

text: clean

text: signal

text: by

text: employ-

text: ing

text: the

text: priori

text: knowledge

text: about

text: time–frequency

text: speech

text: structures

text: Reverberation,

label: abstract

text: which

text: is

text: generally

text: caused

text: by

text: sound

text: reﬂections

text: from

text: The

text: second

text: category

text: is

text: the

text: homomorphic

text: ﬁltering

text: tech-

text: walls,

text: ceilings,

text: and

text: ﬂoors,

text: can

text: result

text: in

text: severe

text: performance

text: degrada-

text: nique,

text: which

text: adopts

text: a

text: homomorphic

text: transformation

text: to

text: decompose

text: tion

text: of

text: acoustic

text: applications.

text: Due

text: to

text: a

text: complicated

text: combination

text: of

text: the

text: reverberant

text: signal

text: from

text: the

text: time

text: domain

text: to

text: the

text: cepstral

text: domain,

text: attenuation

text: and

text: time-delay

text: effects,

text: the

text: reverberation

text: property

text: is

text: dif-

text: and

text: thus

text: separates

text: the

text: reverberation

text: from

text: the

text: input

text: cepstral

text: coefﬁ-

text: ﬁcult

text: to

text: characterize,

text: and

text: it

text: remains

text: a

text: challenging

text: task

text: to

text: effectively

text: cients

text: with

text: a

text: simple

text: subtraction

text: operation

text: Channel-inversion

text: retrieve

text: the

text: anechoic

text: speech

text: signals

text: from

text: reverberation

text: ones.

text: In

text: the

text: methods

text: belong

text: to

text: the

text: third

text: category,

text: which

text: considers

text: the

text: reverber-

text: present

text: study,

text: we

text: proposed

text: a

text: novel

text: integrated

text: deep

text: and

text: ensemble

text: ation

text: as

text: a

text: convolution

text: of

text: the

text: original

text: sound

text: with

text: the

text: room

text: impulse

text: learning

text: algorithm

text: (IDEA)

text: for

text: speech

text: dereverberation.

text: The

text: IDEA

text: response

text: (RIR)

text: and

text: thereby

text: performs

text: an

text: inverse

text: ﬁltering

text: to

text: decon-

text: consists

text: of

text: ofﬂine

text: and

text: online

text: phases.

text: In

text: the

text: ofﬂine

text: phase,

text: we

text: train

text: volve

text: the

text: captured

text: signal

text: Even

text: though

text: the

text: above

text: three

text: cat-

text: multiple

text: dereverberation

text: models,

text: each

text: aiming

text: to

text: precisely

text: dereverb

text: egories

text: of

text: approaches

text: have

text: been

text: shown

text: to

text: provide

text: satisfactory

text: per-

text: speech

text: signals

text: in

text: a

text: particular

text: acoustic

text: environment;

text: then

text: a

text: uniﬁed

text: formance,

text: they

text: usually

text: require

text: an

text: accurate

text: estimation

text: of

text: time-varied

text: fusion

text: function

text: is

text: estimated

text: that

text: aims

text: to

text: integrate

text: the

text: information

text: RIR,

text: which

text: may

text: not

text: always

text: be

text: accessible

text: in

text: practice

text: of

text: multiple

text: dereverberation

text: models.

text: In

text: the

text: online

text: phase,

text: an

text: input

text: Recently,

text: deep

text: neural

text: network

text: (DNN)

text: models,

text: which

text: show

text: utterance

text: is

text: ﬁrst

text: processed

text: by

text: each

text: of

text: the

text: dereverberation

text: models.

text: strong

text: regression

text: capabilities,

text: have

text: been

text: used

text: to

text: address

text: the

text: speech

text: The

text: outputs

text: of

text: all

text: models

text: are

text: integrated

text: accordingly

text: to

text: generate

text: the

text: dereverberation

text: issue

text: The

text: main

text: concept

text: here

text: is

text: to

text: use

text: a

text: ﬁnal

text: anechoic

text: signal.

text: We

text: evaluated

text: the

text: IDEA

text: on

text: designed

text: acoustic

text: DNN

text: model

text: to

text: characterize

text: the

text: non-linear

text: spectral

text: mapping

text: from

text: environments,

text: including

text: both

text: matched

text: and

text: mismatched

text: conditions

text: reverberant

text: to

text: anechoic

text: speech

text: in

text: the

text: training

text: stage.

text: In

text: the

text: testing

text: of

text: the

text: training

text: and

text: testing

text: data.

text: Experimental

text: results

text: conﬁrm

text: that

text: stage,

text: the

text: trained

text: DNN

text: model

text: is

text: used

text: to

text: generate

text: dereverbed

text: ut-

text: the

text: proposed

text: IDEA

text: outperforms

text: single

text: deep-neural-network-based

text: terances

text: given

text: the

text: input

text: reverberant

text: signals.

text: The

text: same

text: concept

text: has

text: dereverberation

text: model

text: with

text: the

text: same

text: model

text: architecture

text: and

text: train-

text: been

text: applied

text: to

text: perform

text: denoising

text: and

text: dereverberation

text: simultane-

text: ing

text: data.

text: ously

text: Despite

text: providing

text: notable

text: improvements

text: over

text: traditional

text: Index

text: Terms—

text: Deep

text: neural

text: networks,

text: Speech

text: dereverberation,

text: algorithms,

text: DNN-based

text: dereverberation

text: methods

text: achieve

text: the

text: opti-

text: Ensemble

text: learning,

text: Convolutional

text: neural

text: networks,

text: Deep

text: denoising

text: mal

text: performance

text: only

text: in

text: matched

text: training

text: and

text: testing

text: reverberant

text: autoencoder

text: conditions.

text: To

text: further

text: improve

text: the

text: performance,

text: an

text: environment-

text: aware

text: DNN-based

text: dereverberation

text: system

text: has

text: been

text: proposed,

text: which

text: INTRODUCTION

text: selects

text: the

text: optimal

text: DNN

text: models

text: online

text: to

text: perform

text: dereverbera-

text: tion

text: In

text: realistic

text: environments,

text: the

text: perceived

text: speech

text: signal

text: may

text: com-

text: Contrary

text: to

text: the

text: idea

text: used

text: in

text: the

text: present

text: study

text: extends

text: the

text: prise

text: of

text: the

text: original

text: speech

text: and

text: multiple

text: copies

text: of

text: the

text: attenuated

text: and

text: previous

text: work

text: on

text: the

text: deep

text: denoise

text: autoencoder

text: (DDAE)

text: in

text: speech

text: time-delayed

text: signals

text: The

text: combination

text: of

text: these

text: signals

text: can

text: cause

text: enhancement

text: and

text: proposes

text: a

text: novel

text: integrated

text: deep

text: and

text: en-

text: serious

text: performance

text: degradation

text: of

text: speech-related

text: applications.

text: For

text: semble

text: learning

text: algorithm

text: (IDEA)

text: for

text: speech

text: dereverberation.

text: The

text: example,

text: distant-talking

text: speech

text: signiﬁcantly

text: degrades

text: the

text: perfor-

text: IDEA

text: consists

text: of

text: ofﬂine

text: and

text: online

text: phases.

text: In

text: the

text: ofﬂine

text: phase,

text: mul-

text: mance

text: of

text: automatic

text: speech

text: recognition

text: (ASR)

text: and

text: speaker

text: tiple

text: DDAE-based

text: dereverberation

text: models

text: are

text: prepared,

text: with

text: each

text: identiﬁcation

text: Meanwhile,

text: the

text: adverse

text: effects

text: of

text: reverberation

text: aiming

text: to

text: precisely

text: dereverb

text: speech

text: signals

text: in

text: a

text: particular

text: acoustic

text: will

text: lower

text: sound

text: quality

text: and

text: intelligibility

text: for

text: both

text: hearing-impaired

text: environment.

text: Then,

text: a

text: uniﬁed

text: fusion

text: model

text: is

text: estimated

text: to

text: integrate

text: and

text: normal-hearing

text: listeners

text: In

text: the

text: past,

text: various

text: speech

text: dere-

text: the

text: information

text: of

text: the

text: multiple

text: dereverberation

text: models

text: with

text: the

text: aim

text: verberation

text: methods

text: have

text: been

text: developed.

text: The

text: goal

text: of

text: these

text: meth-

text: to

text: estimate

text: clean

text: speech.

text: In

text: the

text: online

text: phase,

text: an

text: input

text: reverber-

text: ods

text: is

text: to

text: extract

text: anechoic

text: speech

text: signals

text: from

text: reverberant

text: ones

text: to

text: ant

text: speech

text: is

text: ﬁrst

text: processed

text: by

text: all

text: dereverberation

text: models

text: simul-

text: enhance

text: the

text: performance

text: of

text: speech-related

text: applications

text: and

text: to

text: im-

text: taneously,

text: and

text: the

text: outputs

text: are

text: integrated

text: to

text: ultimately

text: generate

text: the

text: prove

text: sound

text: quality

text: and

text: intelligibility

text: simultaneously

text: for

text: listeners

text: in

text: anechoic

text: signals.

text: The

text: ensemble

text: learning

text: strategy,

text: which

text: has

text: been

text: reverberant

text: environments.

text: proven

text: to

text: be

text: able

text: to

text: improve

text: system

text: performance

text: in

text: speech

text: enhance-

text: Traditional

text: speech

text: dereverberation

text: methods

text: can

text: be

text: roughly

text: di-

text: ment

text: and

text: ASR

text: is

text: adopted

text: in

text: the

text: task

text: to

text: increase

text: the

text: vided

text: into

text: three

text: categories

text: The

text: ﬁrst

text: category

text: is

text: the

text: source-

text: generalization

text: ability

text: of

text: DDAEs.

text: As

text: will

text: be

text: introduced

text: in

text: the

text: re-


text: 6 [20] I. Lee, X. Liu, C. Zhou, and B. Kosko, “Noise-enhanced detection of subthreshold signals with carbon nanotubes,” IEEE Transactions on Nanotechnology, vol. 5, no. 6, pp. 613–627, 2006. [21] B. Kosko, Noise. Viking, 2006. [22] M. McDonnell, N. Stocks, C. Pearce, and D. Abbott, Stochastic resonance: from suprathreshold stochastic resonance to stochastic signal quantization. Cambridge University Press, 2008. [23] A. Patel and B. Kosko, “Optimal Mean-Square Noise Beneﬁts in Quantizer-Array Linear Estimation,” IEEE Signal Processing Letters, vol. 17, no. 12, pp. 1005 –1009, 12 2010. [24] ——, “Noise Beneﬁts in Quantizer-Array Correlation Detection and Watermark Decoding,” IEEE Transactions on Signal Processing, vol. 59, no. 2, pp. 488 –505, 2 2011. [25] H. Chen, L. R. Varshney, and P. K. Varshney, “Noise-enhanced informa- tion systems,” Proceedings of the IEEE, vol. 102, no. 10, pp. 1607–1621, 2014. [26] S. Mitaim and B. Kosko, “Noise-beneﬁt forbidden-interval theorems for threshold signal detectors based on cross correlations,” Physical Review E, vol. 90, no. 5, p. 052124, 2014. [27] B. Franzke and B. Kosko, “Noise can speed convergence in markov chains,” Physical Review E, vol. 84, no. 4, p. 041112, 2011. [28] ——, “Using noise to speed up markov chain monte carlo estimation,” Procedia Computer Science, vol. 53, pp. 113–120, 2015. [29] S. Kirkpatrick, C. Gelatt Jr, and M. Vecchi, “Optimization by simulated annealing,” Science, vol. 220, no. 4598, pp. 671–680, 1983. [30] V. Cˇerny`, “Thermodynamical approach to the Traveling Salesman Problem: An efﬁcient simulation algorithm,” Journal of Optimization Theory and Applications, vol. 45, no. 1, pp. 41–51, 1985. [31] S. Geman and C. Hwang, “Diffusions for global optimization,” SIAM Journal on Control and Optimization, vol. 24, no. 5, pp. 1031–1043, 1986. [32] B. Hajek, “Cooling schedules for optimal annealing,” Mathematics of operations research, vol. 13, no. 2, pp. 311–329, 1988. [33] B. Kosko, Neural Networks and Fuzzy Systems: A Dynamical Systems Approach to Machine Intelligence. Prentice Hall, 1991. [34] R. A. Redner and H. F. Walker, “Mixture Densities, Maximum Likelihood and the EM algorithm,” SIAM Review, vol. 26, no. 2, pp. 195–239, 1984. [35] G. J. McLachlan and D. Peel, Finite Mixture Models, ser. Wiley series in probability and statistics: Applied probability and statistics. Wiley, 2004. [36] R. V. Hogg and E. A. Tanis, Probability and Statistical Inference, 7th ed. Prentice Hall, 2006. [37] N. A. Gershenfeld, The Nature of Mathematical Modeling. Cambridge University Press, 1999.

label: reference

text: I.

text: Lee,

text: X.

text: Liu,

text: C.

text: Zhou,

text: and

text: B.

text: Kosko,

text: “Noise-enhanced

text: detection

text: of

text: subthreshold

text: signals

text: with

text: carbon

text: nanotubes,”

text: IEEE

text: Transactions

text: on

text: Nanotechnology,

text: vol.

text: no.

text: pp.

text: B.

text: Kosko,

text: Noise.

text: Viking,

text: M.

text: McDonnell,

text: N.

text: Stocks,

text: C.

text: Pearce,

text: and

text: D.

text: Abbott,

text: Stochastic

text: resonance:

text: from

text: suprathreshold

text: stochastic

text: resonance

text: to

text: stochastic

text: signal

text: quantization.

text: Cambridge

text: University

text: Press,

text: A.

text: Patel

text: and

text: B.

text: Kosko,

text: “Optimal

text: Mean-Square

text: Noise

text: Beneﬁts

text: in

text: Quantizer-Array

text: Linear

text: Estimation,”

text: IEEE

text: Signal

text: Processing

text: Letters,

text: vol.

text: no.

text: pp.

text: “Noise

text: Beneﬁts

text: in

text: Quantizer-Array

text: Correlation

text: Detection

text: and

text: Watermark

text: Decoding,”

text: IEEE

text: Transactions

text: on

text: Signal

text: Processing,

text: vol.

text: no.

text: pp.

text: H.

text: Chen,

text: L.

text: R.

text: Varshney,

text: and

text: P.

text: K.

text: Varshney,

text: “Noise-enhanced

text: informa-

text: tion

text: systems,”

text: Proceedings

text: of

text: the

text: IEEE,

text: vol.

text: no.

text: pp.

text: S.

text: Mitaim

text: and

text: B.

text: Kosko,

text: “Noise-beneﬁt

text: forbidden-interval

text: theorems

text: for

text: threshold

text: signal

text: detectors

text: based

text: on

text: cross

text: correlations,”

text: Physical

text: Review

text: E,

text: vol.

text: no.

text: p.

text: B.

text: Franzke

text: and

text: B.

text: Kosko,

text: “Noise

text: can

text: speed

text: convergence

text: in

text: markov

text: chains,”

text: Physical

text: Review

text: E,

text: vol.

text: no.

text: p.

text: “Using

text: noise

text: to

text: speed

text: up

text: markov

text: chain

text: monte

text: carlo

text: estimation,”

text: Procedia

text: Computer

text: Science,

text: vol.

text: pp.

text: S.

text: Kirkpatrick,

text: C.

text: Gelatt

text: Jr,

text: and

text: M.

text: Vecchi,

text: “Optimization

text: by

text: simulated

text: annealing,”

text: Science,

text: vol.

text: no.

text: pp.

text: V.

text: Cˇerny`,

text: “Thermodynamical

text: approach

text: to

text: the

text: Traveling

text: Salesman

text: Problem:

text: An

text: efﬁcient

text: simulation

text: algorithm,”

text: Journal

text: of

text: Optimization

text: Theory

text: and

text: Applications,

text: vol.

text: no.

text: pp.

text: S.

text: Geman

text: and

text: C.

text: Hwang,

text: “Diffusions

text: for

text: global

text: optimization,”

text: SIAM

text: Journal

text: on

text: Control

text: and

text: Optimization,

text: vol.

text: no.

text: pp.

text: B.

text: Hajek,

text: “Cooling

text: schedules

text: for

text: optimal

text: annealing,”

text: Mathematics

text: of

text: operations

text: research,

text: vol.

text: no.

text: pp.

text: B.

text: Kosko,

text: Neural

text: Networks

text: and

text: Fuzzy

text: Systems:

text: A

text: Dynamical

text: Systems

text: Approach

text: to

text: Machine

text: Intelligence.

text: Prentice

text: Hall,

text: R.

text: A.

text: Redner

text: and

text: H.

text: F.

text: Walker,

text: “Mixture

text: Densities,

text: Maximum

text: Likelihood

text: and

text: the

text: EM

text: algorithm,”

text: SIAM

text: Review,

text: vol.

text: no.

text: pp.

text: G.

text: J.

text: McLachlan

text: and

text: D.

text: Peel,

text: Finite

text: Mixture

text: Models,

text: ser.

text: Wiley

text: series

text: in

text: probability

text: and

text: statistics:

text: Applied

text: probability

text: and

text: statistics.

text: Wiley,

text: R.

text: V.

text: Hogg

text: and

text: E.

text: A.

text: Tanis,

text: Probability

text: and

text: Statistical

text: Inference,

text: ed.

text: Prentice

text: Hall,

text: N.

text: A.

text: Gershenfeld,

text: The

text: Nature

text: of

text: Mathematical

text: Modeling.

text: Cambridge

text: University

text: Press,


text: 13 therefore numerically solved the equations using low relative and absolute tolerances in order to ensure the accuracy of the resulting time-series. S2. SYSTEM PARAMETERS The numerical values for the reaction rates are obtained from Ref. [13], and are listed in Table S2. TABLE S2: Reaction Rates Rate constant Value Units − k 1002 (µM.min) 1 1 − k− 150 min 1 1 − k 150 min 1 2 − kp 1002 (µM.min) 1 1 − kp− 150 min 1 1 − kp 150 min 1 2 − k 1002 (µM.min) 1 3 − k− 30 min 1 3 − k 30 min 1 4 − kp 1002 (µM.min) 1 3 − kp− 150 min 1 3 − kp 150 min 1 4 − k 1002 (µM.min) 1 5 − k− 30 min 1 5 − k 30 min 1 6 − kp 1002 (µM.min) 1 5 − kp− 150 min 1 5 − kp 150 min 1 6 − k 1002 (µM.min) 1 7 − k− 30 min 1 7 − k 30 min 1 8 − kp 1002 (µM.min) 1 7 − kp− 150 min 1 7 − kp 150 min 1 8 − k 1002 (µM.min) 1 9 − k− 150 min 1 9 − k 150 min 1 10 − kp 1002 (µM.min) 1 9 − kp− 150 min 1 9 − kp 150 min 1 10 ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: therefore

text: numerically

text: solved

text: the

text: equations

text: using

text: low

text: relative

text: and

text: absolute

text: tolerances

text: in

text: order

text: to

text: ensure

text: the

text: accuracy

text: of

text: the

text: resulting

text: time-series.

label: section

text: SYSTEM

text: PARAMETERS

text: The

text: numerical

text: values

text: for

text: the

text: reaction

text: rates

text: are

text: obtained

text: from

text: Ref.

text: and

text: are

text: listed

text: in

text: Table

text: TABLE

text: Reaction

text: Rates

text: Rate

text: constant

text: Value

text: Units

text: k

text: (µM.min)

text: k−

text: min

text: k

text: min

text: kp

text: (µM.min)

text: kp−

text: min

text: kp

text: min

text: k

text: (µM.min)

text: k−

text: min

text: k

text: min

text: kp

text: (µM.min)

text: kp−

text: min

text: kp

text: min

text: k

text: (µM.min)

text: k−

text: min

text: k

text: min

text: kp

text: (µM.min)

text: kp−

text: min

text: kp

text: min

text: k

text: (µM.min)

text: k−

text: min

text: k

text: min

text: kp

text: (µM.min)

text: kp−

text: min

text: kp

text: min

text: k

text: (µM.min)

text: k−

text: min

text: k

text: min

text: kp

text: (µM.min)

text: kp−

text: min

text: kp

text: min

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: 19 FIG. S6: Protein complexes in the MAPK cascade exhibit relaxation behavior occurring over a broad range of time-scales. × − Decay of activity is shown after withdrawing an applied stimulus of amplitude S = 1.2 10 6µM. The relaxation times τ of the x diﬀerent molecular species, viz., (a) the protein complexes between non-phosphorylated and singly phosphorylated (non-active) kinase proteins and the doubly phosphorylated (active) kinase protein of the preceding layer, and (b) the protein complexes between the phosphorylated (singly- or doubly-) kinase proteins and the phosphatase that carries out dephosphorylation in the corresponding layer of the MAPK cascade, vary with the total concentration of MAP2K. The nature of this dependence is distinct for lower and higher values of [2K] . For both panels, [K] = 0.8µM and [3K] = 0.0020µM. The total tot tot tot concentrations of the phosphatases are provided in Table S4. ##LTFigure## ##LTFigure## ##LTFigure##

text: FIG.

text: Protein

text: complexes

text: in

text: the

text: MAPK

text: cascade

text: exhibit

text: relaxation

text: behavior

text: occurring

text: over

text: a

text: broad

text: range

text: of

text: time-scales.

text: Decay

text: of

text: activity

text: is

text: shown

text: after

text: withdrawing

text: an

text: applied

text: stimulus

text: of

text: amplitude

text: S

text: The

text: relaxation

text: times

text: τ

text: of

text: the

text: x

text: diﬀerent

text: molecular

text: species,

text: viz.,

text: (a)

text: the

text: protein

text: complexes

text: between

text: non-phosphorylated

text: and

text: singly

text: phosphorylated

text: (non-active)

text: kinase

text: proteins

text: and

text: the

text: doubly

text: phosphorylated

text: (active)

text: kinase

text: protein

text: of

text: the

text: preceding

text: layer,

text: and

text: (b)

text: the

text: protein

text: complexes

text: between

text: the

text: phosphorylated

text: (singly-

text: or

text: doubly-)

text: kinase

text: proteins

text: and

text: the

text: phosphatase

text: that

text: carries

text: out

text: dephosphorylation

text: in

text: the

text: corresponding

text: layer

text: of

text: the

text: MAPK

text: cascade,

text: vary

text: with

text: the

text: total

text: concentration

text: of

text: The

text: nature

text: of

text: this

text: dependence

text: is

text: distinct

text: for

text: lower

text: and

text: higher

text: values

text: of

text: For

text: both

text: panels,

text: [K]

text: and

text: The

text: total

text: tot

text: tot

text: tot

text: concentrations

text: of

text: the

text: phosphatases

text: are

text: provided

text: in

text: Table

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##


text: 12 BOSCHINI ET AL. 103 PAMELA e- Jan-Jun 2009 101 10 1 10 3 10 5 10 7 100 101 102 103 0.4 0.2 0.0 0.2 0.4 100 101 102 103 Kinetic Energy [GeV] Figure 9. Differential intensity of CR electrons for PAMELA datasets integrated from January to June 2009. Points represent experimental data, the black dashed line is the GALPROP LIS, and the solid line is the computed modulated spectrum. The bottom panel shows the relative difference between the numerical solution and experimental data. ]1 )VG rs s 2m([ ytisnetnI laitnereffiD ecnereffid evitaleR ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: BOSCHINI

text: ET

text: AL.

font: default

text: PAMELA

text: e-

text: Jan-Jun

text: Kinetic

text: Energy

text: [GeV]

text: Figure

text: Differential

text: intensity

text: of

text: CR

text: electrons

text: for

text: PAMELA

text: datasets

text: integrated

text: from

text: January

text: to

text: June

text: Points

text: represent

text: experimental

text: data,

text: the

text: black

text: dashed

text: line

text: is

text: the

text: GALPROP

text: LIS,

text: and

text: the

text: solid

text: line

text: is

text: the

text: computed

text: modulated

text: spectrum.

text: The

text: bottom

text: panel

text: shows

text: the

text: relative

text: difference

text: between

text: the

text: numerical

text: solution

text: and

text: experimental

text: data.

text: )VG

text: rs

text: s

text: ytisnetnI

text: laitnereffiD

text: ecnereffid

text: evitaleR

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: LOCAL INTERSTELLAR SPECTRUM OF COSMIC-RAY ELECTRONS 13 Table 4 Electron LIS Kinetic energy, Differential Kinetic energy, Differential Kinetic energy, Differential Kinetic energy, Differential GeV intensitya GeV intensitya GeV intensitya GeV intensitya 1.000e-03 3.481e+06 3.927e-02 3.225e+04 1.542e+00 6.057e+01 6.482e+01 5.236e-04 1.070e-03 3.710e+06 4.203e-02 2.935e+04 1.651e+00 5.145e+01 6.938e+01 4.167e-04 1.146e-03 3.523e+06 4.499e-02 2.671e+04 1.767e+00 4.370e+01 7.426e+01 3.318e-04 1.226e-03 3.288e+06 4.815e-02 2.431e+04 1.891e+00 3.713e+01 7.949e+01 2.644e-04 1.312e-03 3.056e+06 5.154e-02 2.212e+04 2.024e+00 3.155e+01 8.508e+01 2.108e-04 1.405e-03 2.835e+06 5.517e-02 2.014e+04 2.166e+00 2.681e+01 9.106e+01 1.681e-04 1.504e-03 2.627e+06 5.905e-02 1.834e+04 2.319e+00 2.278e+01 9.747e+01 1.342e-04 1.609e-03 2.431e+06 6.320e-02 1.670e+04 2.482e+00 1.935e+01 1.043e+02 1.071e-04 1.722e-03 2.246e+06 6.764e-02 1.521e+04 2.656e+00 1.642e+01 1.117e+02 8.560e-05 1.844e-03 2.074e+06 7.240e-02 1.386e+04 2.843e+00 1.392e+01 1.195e+02 6.841e-05 1.973e-03 1.912e+06 7.749e-02 1.262e+04 3.043e+00 1.179e+01 1.279e+02 5.470e-05 2.112e-03 1.761e+06 8.295e-02 1.149e+04 3.257e+00 9.962e+00 1.369e+02 4.374e-05 2.261e-03 1.621e+06 8.878e-02 1.046e+04 3.486e+00 8.397e+00 1.465e+02 3.499e-05 2.420e-03 1.490e+06 9.502e-02 9.525e+03 3.732e+00 7.054e+00 1.569e+02 2.800e-05 2.590e-03 1.368e+06 1.017e-01 8.669e+03 3.994e+00 5.904e+00 1.679e+02 2.240e-05 2.772e-03 1.256e+06 1.089e-01 7.887e+03 4.275e+00 4.919e+00 1.797e+02 1.793e-05 2.967e-03 1.151e+06 1.165e-01 7.173e+03 4.576e+00 4.076e+00 1.923e+02 1.435e-05 3.176e-03 1.055e+06 1.247e-01 6.521e+03 4.898e+00 3.358e+00 2.059e+02 1.149e-05 3.399e-03 9.658e+05 1.335e-01 5.926e+03 5.242e+00 2.748e+00 2.203e+02 9.193e-06 3.638e-03 8.835e+05 1.429e-01 5.382e+03 5.611e+00 2.235e+00 2.358e+02 7.358e-06 3.894e-03 8.077e+05 1.529e-01 4.886e+03 6.005e+00 1.806e+00 2.524e+02 5.889e-06 4.168e-03 7.380e+05 1.637e-01 4.433e+03 6.428e+00 1.451e+00 2.702e+02 4.713e-06 4.461e-03 6.739e+05 1.752e-01 4.019e+03 6.880e+00 1.160e+00 2.892e+02 3.772e-06 4.775e-03 6.150e+05 1.875e-01 3.640e+03 7.364e+00 9.239e-01 3.095e+02 3.018e-06 5.111e-03 5.609e+05 2.007e-01 3.295e+03 7.882e+00 7.335e-01 3.313e+02 2.415e-06 5.470e-03 5.114e+05 2.148e-01 2.979e+03 8.436e+00 5.811e-01 3.546e+02 1.932e-06 5.855e-03 4.660e+05 2.299e-01 2.690e+03 9.029e+00 4.598e-01 3.795e+02 1.545e-06 6.267e-03 4.244e+05 2.461e-01 2.426e+03 9.665e+00 3.634e-01 4.062e+02 1.236e-06 6.707e-03 3.864e+05 2.634e-01 2.185e+03 1.034e+01 2.871e-01 4.348e+02 9.887e-07 7.179e-03 3.517e+05 2.819e-01 1.964e+03 1.107e+01 2.268e-01 4.654e+02 7.907e-07 7.684e-03 3.200e+05 3.018e-01 1.762e+03 1.185e+01 1.791e-01 4.981e+02 6.323e-07 8.225e-03 2.910e+05 3.230e-01 1.577e+03 1.268e+01 1.415e-01 5.332e+02 5.056e-07 8.803e-03 2.646e+05 3.457e-01 1.409e+03 1.358e+01 1.118e-01 5.707e+02 4.042e-07 9.422e-03 2.406e+05 3.700e-01 1.255e+03 1.453e+01 8.829e-02 6.108e+02 3.231e-07 1.008e-02 2.186e+05 3.960e-01 1.115e+03 1.555e+01 6.976e-02 6.538e+02 2.583e-07 1.079e-02 1.987e+05 4.239e-01 9.886e+02 1.665e+01 5.512e-02 6.997e+02 2.064e-07 1.155e-02 1.805e+05 4.537e-01 8.739e+02 1.782e+01 4.356e-02 7.489e+02 1.650e-07 1.237e-02 1.640e+05 4.856e-01 7.703e+02 1.907e+01 3.444e-02 8.016e+02 1.318e-07 1.324e-02 1.489e+05 5.198e-01 6.771e+02 2.041e+01 2.722e-02 8.580e+02 1.053e-07 1.417e-02 1.353e+05 5.563e-01 5.934e+02 2.185e+01 2.152e-02 9.184e+02 8.416e-08 1.516e-02 1.229e+05 5.955e-01 5.185e+02 2.339e+01 1.703e-02 9.830e+02 6.724e-08 1.623e-02 1.116e+05 6.374e-01 4.517e+02 2.503e+01 1.347e-02 1.052e+03 5.372e-08 1.737e-02 1.014e+05 6.822e-01 3.923e+02 2.679e+01 1.066e-02 1.126e+03 4.291e-08 1.859e-02 9.206e+04 7.302e-01 3.398e+02 2.867e+01 8.439e-03 1.205e+03 3.427e-08 1.990e-02 8.363e+04 7.815e-01 2.934e+02 3.069e+01 6.680e-03 1.290e+03 2.737e-08 2.130e-02 7.597e+04 8.365e-01 2.527e+02 3.285e+01 5.304e-03 1.381e+03 2.186e-08 2.280e-02 6.903e+04 8.953e-01 2.171e+02 3.516e+01 4.192e-03 1.478e+03 1.746e-08 2.440e-02 6.273e+04 9.583e-01 1.861e+02 3.763e+01 3.316e-03 1.582e+03 1.394e-08 2.612e-02 5.701e+04 1.026e+00 1.592e+02 4.028e+01 2.633e-03 1.693e+03 1.113e-08 2.796e-02 5.182e+04 1.098e+00 1.359e+02 4.311e+01 2.077e-03 1.812e+03 8.888e-09 2.992e-02 4.712e+04 1.175e+00 1.158e+02 4.615e+01 1.649e-03 1.940e+03 7.096e-09 3.203e-02 4.285e+04 1.258e+00 9.861e+01 4.939e+01 1.310e-03 2.076e+03 5.665e-09 3.428e-02 3.897e+04 1.346e+00 8.387e+01 5.287e+01 1.042e-03 2.222e+03 4.522e-09 3.669e-02 3.545e+04 1.441e+00 7.129e+01 5.658e+01 8.282e-04 2.378e+03 3.610e-09 − a Differential intensity units: (m2 s sr GV) 1. ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: LOCAL

text: INTERSTELLAR

text: SPECTRUM

text: OF

text: COSMIC-RAY

text: ELECTRONS

text: Table

text: Electron

text: LIS

text: Kinetic

text: energy,

text: Differential

text: Kinetic

text: energy,

text: Differential

text: Kinetic

text: energy,

text: Differential

text: Kinetic

text: energy,

text: Differential

text: GeV

text: intensitya

text: GeV

text: intensitya

text: GeV

text: intensitya

text: GeV

text: intensitya

text: a

text: Differential

text: intensity

text: units:

text: s

text: sr

text: GV)

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: 10 BOSCHINI ET AL. Torii, S. et al. 2001, ApJ, 559, 973 Vladimirov, A. E. et al. 2011, Comp. Phys. Comm., 182, 1156

text: BOSCHINI

text: ET

text: AL.

text: Torii,

text: S.

text: et

text: al.

text: ApJ,

text: Vladimirov,

text: A.

text: E.

text: et

text: al.

text: Comp.

text: Phys.

text: Comm.,


text: Mutual Information Neural Estimation Discriminator Layer Number of outputs Kernel size Stride Activation function ∗ ∗ Input x 28 28 3 ∗ Convolution 14*14*64 5 5 2 ReLU ∗ Convolution 7*7*128 5 5 2 ReLU ∗ Convolution 4*4*256 5 5 2 ReLU ∗ Convolution 2*2*512 5 5 2 ReLU Fully-connected 1 1 Valid Sigmoid Table 5. Discriminator network for Stacked-MNIST experiment. Statistics Network Layer number of outputs kernel size stride activation function Input x, z ∗ Convolution 14*14*16 5 5 2 ELU ∗ Convolution 7*7*32 5 5 2 ELU ∗ Convolution 4*4*64 5 5 2 ELU Flatten - - - - Fully-Connected 1024 1 Valid None Fully-Connected 1 1 Valid None Table 6. Statistics network for Stacked-MNIST experiment. 8.1.4. ALI+MINE: MNIST AND CELEBA In this section we state the details of experimental setup and the network architectures used for the task of improving reconstructions and representations in bidirectional adversarial models with MINE. The generator and discriminator network architectures along with the hyper parameter setup used in these tasks are similar to the ones used in DCGAN (Radford et al., 2015). For conditioning the latent code in statistics network, we used spatial replication and concatenation at the ﬁrst layer of the network. We used Adam as the optimizer with a learning rate of 0.0001. We trained the model for a total of 35, 000 iterations on CelebA and 50, 000 iterations on MNIST, both with a mini batch-size of 100. Encoder Layer Number of outputs Kernel size Stride Activation function Input [x, (cid:15)] 28*28*129 ∗ Convolution 14*14*64 5 5 2 ReLU ∗ Convolution 7*7*128 5 5 2 ReLU ∗ Convolution 4*4*256 5 5 2 ReLU ∗ Convolution 256 4 4 Valid ReLU Fully-connected 128 - - None ∼ N Table 7. Encoder network for bi-directional models on MNIST. (cid:15) (0, I). 128 ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: Mutual

text: Information

text: Neural

text: Estimation

text: Discriminator

text: Layer

text: Number

text: of

text: outputs

text: Kernel

text: size

text: Stride

text: Activation

text: function

text: Input

text: x

text: Convolution

text: ReLU

text: Convolution

text: ReLU

text: Convolution

text: ReLU

text: Convolution

text: ReLU

text: Fully-connected

text: Valid

text: Sigmoid

text: Table

text: Discriminator

text: network

text: for

text: Stacked-MNIST

text: experiment.

text: Statistics

text: Network

text: Layer

text: number

text: of

text: outputs

text: kernel

text: size

text: stride

text: activation

text: function

text: Input

text: x,

text: z

text: Convolution

text: ELU

text: Convolution

text: ELU

text: Convolution

text: ELU

text: Flatten

text: Fully-Connected

text: Valid

text: Fully-Connected

text: Valid

text: Table

text: Statistics

text: network

text: for

text: Stacked-MNIST

text: experiment.

text: ALI+MINE:

text: MNIST

text: AND

text: CELEBA

text: In

text: this

text: section

text: we

text: state

text: the

text: details

text: of

text: experimental

text: setup

text: and

text: the

text: network

text: architectures

text: used

text: for

text: the

text: task

text: of

text: improving

text: reconstructions

text: and

text: representations

text: in

text: bidirectional

text: adversarial

text: models

text: with

text: MINE.

text: The

text: generator

text: and

text: discriminator

text: network

text: architectures

text: along

text: with

text: the

text: hyper

text: parameter

text: setup

text: used

text: in

text: these

text: tasks

text: are

text: similar

text: to

text: the

text: ones

text: used

text: in

text: DCGAN

text: (Radford

text: et

text: al.,

text: For

text: conditioning

text: the

text: latent

text: code

text: in

text: statistics

text: network,

text: we

text: used

text: spatial

text: replication

text: and

text: concatenation

text: at

text: the

text: ﬁrst

text: layer

text: of

text: the

text: network.

text: We

text: used

text: Adam

text: as

text: the

text: optimizer

text: with

text: a

text: learning

text: rate

text: of

text: We

text: trained

text: the

text: model

text: for

text: a

text: total

text: of

text: iterations

text: on

text: CelebA

text: and

text: iterations

text: on

text: MNIST,

text: both

text: with

text: a

text: mini

text: batch-size

text: of

text: Encoder

text: Layer

text: Number

text: of

text: outputs

text: Kernel

text: size

text: Stride

text: Activation

text: function

text: Input

text: [x,

text: Convolution

text: ReLU

text: Convolution

text: ReLU

text: Convolution

text: ReLU

text: Convolution

text: Valid

text: ReLU

text: Fully-connected

text: N

text: Table

text: Encoder

text: network

text: for

text: bi-directional

text: models

text: on

text: MNIST.

text: I).

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: Mutual Information Neural Estimation where I (X, Z) = sup EP [T ] − log(EP ⊗P [eTθ]) (29) Θ ∈ XZ θ X Z θ Θ ∗ P ∗ Proof. Let T = log dQ. By construction, T satisﬁes: d EP[T∗] = I(X, Z), EQ[eT∗] = 1 (30) − For a function T, the (positive) gap I(X, Z) Iˆ(T) can be written as I(X, Z) − Iˆ(T) = EP[T∗ − T] + log EQ[eT] ≤ EP[T∗ − T] + EQ[eT − eT∗] (31) ≤ − where we used the inequality log x x 1. ∗ Fix η > 0. We ﬁrst consider the case where T is bounded from above by a constant M. By the universal approximation theorem (see ≤ corollary 2.2 of Hornik (1989)9), we may choose a feedforward network function T M such that θˆ EP|T∗ − T | ≤ η and EQ|T∗ − T | ≤ η e−M (32) θˆ 2 θˆ 2 −∞ Since exp is Lipschitz continuous with constant eM on ( , M], we have EQ|eT∗ − eTθˆ| ≤ eM EQ|T∗ − Tθˆ| ≤ η2 (33) From Equ 31 and the triangular inequality, we then obtain: |I(X, Z) − Iˆ(Tθˆ)| ≤ EP|T∗ − Tθˆ| + EQ|eT∗ − eTθˆ| ≤ η2 + η2 ≤ η (34) { ∗ } { ∗ ≤ } In the general case, the idea is to partition Ω in two subset T > M and T M for a suitably chosen large value of M. For a ⊂ ∈ ∗ gtoivPen10,suabnsdeteTS∗ isΩin,twegerawbillel dweintohterebsype1cSt tiotsQc,hsaorabctyertihseticdofumnicntaiotend, 1coSn(vωe)rg=en1ceiftωheoreSma,nwde0motahyercwhoisoes.eTMissointtheagtrathbeleewxpitehcrteastipoencst E ∗ E ∗ P[T 1T∗>M] and Q[eT 1T∗>M] are lower than η/4. Just like above, we then use the universal approximation theorem to ﬁnd a feed forward network function T , which we can assume without loss of generality to be upper-bounded by M, such that θˆ EP|T∗ − Tθˆ| ≤ η2 and EQ|T∗ − Tθˆ|1T∗≤M ≤ η4e−M (35) We then write EQ[eT∗ − eTθˆ] = EQ[(eT∗ − eTθˆ)1T∗≤M] + EQ[(eT∗ − eTθˆ)1T∗>M] ≤ eMEQ[|T∗ − Tθˆ|1T∗≤M] + EQ[eT∗1T∗>M] ≤ η η + (36) 4 4 ≤ η (37) 2 where the inequality in the second line arises from the convexity and positivity of exp. Eqns. 35 and 36, together with the triangular inequality, lead to Eqn. 34, which proves the Lemma. F Le⊂mmRa 4 (Lemma 2 rest∈ateNd). Let η > 0. Given a family(cid:16) of neural network functions(cid:17)Tθ with parameters θ in some compact domain Θ k, there exists N such that ∀ ≥ | (cid:92) − | ≤ n N, Pr I(X; Z) IF(X, Z) η = 1 (38) n 9Speciﬁcally, the argument relies on the density of feedforward network functions in the space L1(Ω, µ) of integrable functions with P Q respect the measure µ = + . (cid:12) (cid:12) 10This can be seen from the identity (Gyo¨rﬁ(cid:12) & van d(cid:12)er Meulen, 1987) (cid:112) (cid:12) (cid:12) (cid:12) (cid:12) P E d ≤ P || Q P || Q P log dQ DKL( ) + 4 DKL( ) ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: Mutual

text: Information

text: Neural

text: Estimation

text: where

text: I

text: (X,

text: Z)

text: sup

text: EP

text: [T

text: log(EP

text: ⊗P

text: [eTθ])

text: Θ

text: XZ

text: θ

text: X

text: Z

text: θ

text: Θ

text: P

text: Proof.

text: Let

text: T

text: log

text: dQ.

text: By

text: construction,

text: T

text: satisﬁes:

text: d

text: EP[T∗]

text: I(X,

text: Z),

text: EQ[eT∗]

text: For

text: a

text: function

text: T,

text: the

text: (positive)

text: gap

text: I(X,

text: Z)

text: Iˆ(T)

text: can

text: be

text: written

text: as

text: I(X,

text: Z)

text: Iˆ(T)

text: EP[T∗

text: T]

text: log

text: EQ[eT]

text: EP[T∗

text: T]

text: EQ[eT

text: eT∗]

text: where

text: we

text: used

text: the

text: inequality

text: log

text: x

text: x

text: Fix

text: η

text: We

text: ﬁrst

text: consider

text: the

text: case

text: where

text: T

text: is

text: bounded

text: from

text: above

text: by

text: a

text: constant

text: M.

text: By

text: the

text: universal

text: approximation

text: theorem

text: (see

text: corollary

text: of

text: Hornik

text: we

text: may

text: choose

text: a

text: feedforward

text: network

text: function

text: T

text: M

text: such

text: that

text: θˆ

text: EP|T∗

text: T

text: η

text: and

text: EQ|T∗

text: T

text: η

text: e−M

text: θˆ

text: θˆ

text: Since

text: exp

text: is

text: Lipschitz

text: continuous

text: with

text: constant

text: eM

text: on

text: M],

text: we

text: have

text: EQ|eT∗

text: eTθˆ|

text: eM

text: EQ|T∗

text: Tθˆ|

text: η2

text: From

text: Equ

text: and

text: the

text: triangular

text: inequality,

text: we

text: then

text: obtain:

text: |I(X,

text: Z)

text: Iˆ(Tθˆ)|

text: EP|T∗

text: Tθˆ|

text: EQ|eT∗

text: eTθˆ|

text: η2

text: η2

text: η

text: In

text: the

text: general

text: case,

text: the

text: idea

text: is

text: to

text: partition

text: Ω

text: in

text: two

text: subset

text: T

text: M

text: and

text: T

text: M

text: for

text: a

text: suitably

text: chosen

text: large

text: value

text: of

text: M.

text: For

text: a

text: isΩin,twegerawbillel

text: E

text: E

text: P[T

text: and

text: Q[eT

text: are

text: lower

text: than

text: Just

text: like

text: above,

text: we

text: then

text: use

text: the

text: universal

text: approximation

text: theorem

text: to

text: ﬁnd

text: a

text: feed

text: forward

text: network

text: function

text: T

text: which

text: we

text: can

text: assume

text: without

text: loss

text: of

text: generality

text: to

text: be

text: upper-bounded

text: by

text: M,

text: such

text: that

text: θˆ

text: EP|T∗

text: Tθˆ|

text: η2

text: and

text: EQ|T∗

text: We

text: then

text: write

text: EQ[eT∗

text: eTθˆ]

text: EQ[(eT∗

text: EQ[(eT∗

text: eMEQ[|T∗

text: η

text: η

text: η

text: where

text: the

text: inequality

text: in

text: the

text: second

text: line

text: arises

text: from

text: the

text: convexity

text: and

text: positivity

text: of

text: exp.

text: Eqns.

text: and

text: together

text: with

text: the

text: triangular

text: inequality,

text: lead

text: to

text: Eqn.

text: which

text: proves

text: the

text: Lemma.

text: F

text: Le⊂mmRa

text: (Lemma

text: rest∈ateNd).

text: Let

text: η

text: Given

text: a

text: of

text: neural

text: network

text: with

text: parameters

text: θ

text: in

text: some

text: compact

text: domain

text: Θ

text: k,

text: there

text: exists

text: N

text: such

text: that

text: n

text: N,

text: Pr

text: I(X;

text: Z)

text: IF(X,

text: Z)

text: η

text: n

text: the

text: argument

text: relies

text: on

text: the

text: density

text: of

text: feedforward

text: network

text: functions

text: in

text: the

text: space

text: µ)

text: of

text: integrable

text: functions

text: with

text: P

text: Q

text: respect

text: the

text: measure

text: µ

text: can

text: be

text: seen

text: from

text: the

text: identity

text: van

text: Meulen,

text: P

text: E

text: d

text: P

text: Q

text: P

text: Q

text: P

text: log

text: dQ

text: DKL(

text: DKL(

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: Mutual Information Neural Estimation (a) Training set (b) DCGAN (c) DCGAN+MINE Figure 5. Samples from the Stacked MNIST dataset along with generated samples from DCGAN and DCGAN with MINE. While DCGAN only shows a very limited number of modes, the inclusion of MINE generates a much better representative set of samples. considering the reconstruction and the corresponding encod- Reconstruction. Reconstruction. MS-SSIM Error Acc.(%) ings as an additional fake pair. Belghazi et al. (2018) shows that Markovian hierarchical generator in a bi-directional MNIST adversarial model provide a hierarchy of reconstructions ALI 14.24 45.95 0.97 ALICE(l ) 3.20 99.03 0.97 with increasing levels of ﬁdelity (increasing reconstruction 2 ALICE(Adv.) 5.20 98.17 0.98 quality). Li et al. (2017) shows that the expected reconstruc- MINE 9.73 96.10 0.99 tion error can be diminished by minimizing the conditional CelebA entropy of the observables given the latent representations. ALI 53.75 57.49 0.81 The conditional entropy being intractable for general pos- ALICE(l ) 8.01 32.22 0.93 2 terior, Li et al. (2017) proposes to augment the generator’s ALICE(Adv.) 92.56 48.95 0.51 loss with an adversarial cycle consistency loss (Zhu et al., MINE 36.11 76.08 0.99 2017) between the observables and their reconstructions. Table 2. Comparison of MINE with other bi-directional adversar- ial models in terms of euclidean reconstruction error, reconstruc- tion accuracy, and MS-SSIM on the MNIST and CelebA datasets. Experiment: ALI+MINE In this section we compare MINE does a good job compared to ALI in terms of reconstruc- tions. Though the explicit reconstruction based baselines (ALICE) MINE to existing bi-directional adversarial models. As the can sometimes do better than MINE in terms of reconstructions decoder’s density is generally intractable, we use three dif- related tasks, they consistently lag behind in MS-SSIM scores and ferent metrics to measure the ﬁdelity of the reconstructions reconstruction accuracy on CelebA. with respect to the samples; (i) the euclidean reconstruction error, (ii) reconstruction accuracy, which is the proportion of labels preserved by the reconstruction as identiﬁed by a 5.3. Information Bottleneck pre-trained classiﬁer; (iii) the Multi-scale structural sim- The Information Bottleneck (IB, Tishby et al., 2000) is an ilarity metric (MS-SSIM, Wang et al., 2004) between the information theoretic method for extracting relevant infor- observables and their reconstructions. ∈ X mation, or yielding a representation, that an input X ∈ Y We train MINE on datasets of increasing order of complex- contains about an output Y . An optimal representation ity: a toy dataset composed of 25-Gaussians, MNIST, and of X would capture the relevant factors and compress X the CelebA dataset. Fig. 6 shows the reconstruction ability by diminishing the irrelevant parts which do not contribute of MINE compared to ALI. Although ALICE does perfect to the prediction of Y . IB was recently covered in the con- reconstruction (which is in its explicit formulation), we ob- text of deep learning (Tishby & Zaslavsky, 2015), and as serve signiﬁcant mode-dropping in the sample space. MINE such can be seen as a process to construct an approxima- does a balanced job of reconstructing along with capturing tion of the minimally sufﬁcient statistics of the data. IB | all the modes of the underlying data distribution. seeks an encoder, q(Z X), that induces the Markovian → → structure X Z Y . This is done by minimizing the IB Next, measure the ﬁdelity of ﬁdelity of the reconstructions Lagrangian, over ALI, ALICE, and MINE. Tbl. 2 compares MINE to the existing baselines in terms of euclidean reconstruction er- rors, reconstruction accuracy, and MS-SSIM metric. MINE L | | does a better job than ALI in terms of reconstruction errors [q(Z X)] = H(Y Z) + βI(X, Z), (20) by a good margin and is competitive to ALICE with respect to reconstruction accuracy and MS-SSIM. Our results show that MINE’s effect on reconstructions is even more dramatic which appears as a standard cross-entropy loss augmented when compared to ALI and ALICE. with a regularizer promoting minimality of the representa- ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: Mutual

text: Information

text: Neural

text: Estimation

text: (a)

text: Training

text: set

text: (b)

text: DCGAN

text: (c)

text: DCGAN+MINE

text: Figure

text: Samples

text: from

text: the

text: Stacked

text: MNIST

text: dataset

text: along

text: with

text: generated

text: samples

text: from

text: DCGAN

text: and

text: DCGAN

text: with

text: MINE.

text: While

text: DCGAN

text: only

text: shows

text: a

text: very

text: limited

text: number

text: of

text: modes,

text: the

text: inclusion

text: of

text: MINE

text: generates

text: a

text: much

text: better

text: representative

text: set

text: of

text: samples.

text: considering

text: the

text: reconstruction

text: and

text: the

text: corresponding

text: encod-

text: Reconstruction.

text: Reconstruction.

text: MS-SSIM

text: Error

text: Acc.(%)

text: ings

text: as

text: an

text: additional

text: fake

text: pair.

text: Belghazi

text: et

text: al.

text: shows

text: that

text: Markovian

text: hierarchical

text: generator

text: in

text: a

text: bi-directional

text: MNIST

text: adversarial

text: model

text: provide

text: a

text: hierarchy

text: of

text: reconstructions

text: ALI

text: ALICE(l

text: with

text: increasing

text: levels

text: of

text: ﬁdelity

text: (increasing

text: reconstruction

text: ALICE(Adv.)

text: quality).

text: Li

text: et

text: al.

text: shows

text: that

text: the

text: expected

text: reconstruc-

text: MINE

text: tion

text: error

text: can

text: be

text: diminished

text: by

text: minimizing

text: the

text: conditional

text: CelebA

text: entropy

text: of

text: the

text: observables

text: given

text: the

text: latent

text: representations.

text: ALI

text: The

text: conditional

text: entropy

text: being

text: intractable

text: for

text: general

text: pos-

text: ALICE(l

text: terior,

text: Li

text: et

text: al.

text: proposes

text: to

text: augment

text: the

text: generator’s

text: ALICE(Adv.)

text: loss

text: with

text: an

text: adversarial

text: cycle

text: consistency

text: loss

text: (Zhu

text: et

text: al.,

text: MINE

text: between

text: the

text: observables

text: and

text: their

text: reconstructions.

text: Table

text: Comparison

text: of

text: MINE

text: with

text: other

text: bi-directional

text: adversar-

text: ial

text: models

text: in

text: terms

text: of

text: euclidean

text: reconstruction

text: error,

text: reconstruc-

text: tion

text: accuracy,

text: and

text: MS-SSIM

text: on

text: the

text: MNIST

text: and

text: CelebA

text: datasets.

text: Experiment:

text: ALI+MINE

text: In

text: this

text: section

text: we

text: compare

text: MINE

text: does

text: a

text: good

text: job

text: compared

text: to

text: ALI

text: in

text: terms

text: of

text: reconstruc-

text: tions.

text: Though

text: the

text: explicit

text: reconstruction

text: based

text: baselines

text: (ALICE)

text: MINE

text: to

text: existing

text: bi-directional

text: adversarial

text: models.

text: As

text: the

text: can

text: sometimes

text: do

text: better

text: than

text: MINE

text: in

text: terms

text: of

text: reconstructions

text: decoder’s

text: density

text: is

text: generally

text: intractable,

text: we

text: use

text: three

text: dif-

text: related

text: tasks,

text: they

text: consistently

text: lag

text: behind

text: in

text: MS-SSIM

text: scores

text: and

text: ferent

text: metrics

text: to

text: measure

text: the

text: ﬁdelity

text: of

text: the

text: reconstructions

text: reconstruction

text: accuracy

text: on

text: CelebA.

text: with

text: respect

text: to

text: the

text: samples;

text: (i)

text: the

text: euclidean

text: reconstruction

text: error,

text: (ii)

text: reconstruction

text: accuracy,

text: which

text: is

text: the

text: proportion

text: of

text: labels

text: preserved

text: by

text: the

text: reconstruction

text: as

text: identiﬁed

text: by

text: a

text: Information

text: Bottleneck

text: pre-trained

text: classiﬁer;

text: (iii)

text: the

text: Multi-scale

text: structural

text: sim-

text: The

text: Information

text: Bottleneck

text: (IB,

text: Tishby

text: et

text: al.,

text: is

text: an

text: ilarity

text: metric

text: (MS-SSIM,

text: Wang

text: et

text: al.,

text: between

text: the

text: information

text: theoretic

text: method

text: for

text: extracting

text: relevant

text: infor-

text: observables

text: and

text: their

text: reconstructions.

text: X

text: mation,

text: or

text: yielding

text: a

text: representation,

text: that

text: an

text: input

text: X

text: Y

text: We

text: train

text: MINE

text: on

text: datasets

text: of

text: increasing

text: order

text: of

text: complex-

text: contains

text: about

text: an

text: output

text: Y

text: An

text: optimal

text: representation

text: ity:

text: a

text: toy

text: dataset

text: composed

text: of

text: MNIST,

text: and

text: of

text: X

text: would

text: capture

text: the

text: relevant

text: factors

text: and

text: compress

text: X

text: the

text: CelebA

text: dataset.

text: Fig.

text: shows

text: the

text: reconstruction

text: ability

text: by

text: diminishing

text: the

text: irrelevant

text: parts

text: which

text: do

text: not

text: contribute

text: of

text: MINE

text: compared

text: to

text: ALI.

text: Although

text: ALICE

text: does

text: perfect

text: to

text: the

text: prediction

text: of

text: Y

text: IB

text: was

text: recently

text: covered

text: in

text: the

text: con-

text: reconstruction

text: (which

text: is

text: in

text: its

text: explicit

text: formulation),

text: we

text: ob-

text: text

text: of

text: deep

text: learning

text: (Tishby

text: Zaslavsky,

text: and

text: as

text: serve

text: signiﬁcant

text: mode-dropping

text: in

text: the

text: sample

text: space.

text: MINE

text: such

text: can

text: be

text: seen

text: as

text: a

text: process

text: to

text: construct

text: an

text: approxima-

text: does

text: a

text: balanced

text: job

text: of

text: reconstructing

text: along

text: with

text: capturing

text: tion

text: of

text: the

text: minimally

text: sufﬁcient

text: statistics

text: of

text: the

text: data.

text: IB

text: all

text: the

text: modes

text: of

text: the

text: underlying

text: data

text: distribution.

text: seeks

text: an

text: encoder,

text: q(Z

text: X),

text: that

text: induces

text: the

text: Markovian

text: structure

text: X

text: Z

text: Y

text: This

text: is

text: done

text: by

text: minimizing

text: the

text: IB

text: Next,

text: measure

text: the

text: ﬁdelity

text: of

text: ﬁdelity

text: of

text: the

text: reconstructions

text: Lagrangian,

text: over

text: ALI,

text: ALICE,

text: and

text: MINE.

text: Tbl.

text: compares

text: MINE

text: to

text: the

text: existing

text: baselines

text: in

text: terms

text: of

text: euclidean

text: reconstruction

text: er-

text: rors,

text: reconstruction

text: accuracy,

text: and

text: MS-SSIM

text: metric.

text: MINE

text: L

text: does

text: a

text: better

text: job

text: than

text: ALI

text: in

text: terms

text: of

text: reconstruction

text: errors

text: [q(Z

text: X)]

text: H(Y

text: Z)

text: βI(X,

text: Z),

text: by

text: a

text: good

text: margin

text: and

text: is

text: competitive

text: to

text: ALICE

text: with

text: respect

text: to

text: reconstruction

text: accuracy

text: and

text: MS-SSIM.

text: Our

text: results

text: show

text: that

text: MINE’s

text: effect

text: on

text: reconstructions

text: is

text: even

text: more

text: dramatic

text: which

text: appears

text: as

text: a

text: standard

text: cross-entropy

text: loss

text: augmented

text: when

text: compared

text: to

text: ALI

text: and

text: ALICE.

text: with

text: a

text: regularizer

text: promoting

text: minimality

text: of

text: the

text: representa-

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##