text: ≤ ≤ − which is bounded by a constant independent of x since T (q) is ﬁnite for all 0 i m 1 (cid:48) (cid:89) (cid:88) Ui such that p > 0. Also, the strong law of large numbers yields i − − n − − m 1 (cid:48) (cid:80) (cid:0) nl→im∞ n 1 log m TUxk (q) =(cid:1) piTUi(q) < 0 k=1 (cid:101) i=0 for η-almost every x. Consequently, the estimate (B.3) shows that for η-almost every x (cid:101) (cid:101) we have ≥ E(|X (x| )−E(Z)X (x| )|q) 1/q < ∞, which implies that X (x| ) n 1 U n+1 U n+1 U n+1 P E E (cid:3) converges -almost surely to the same limit as (Z)X (x| ), that is (Z)X (x). U n+1 U Appendix C. Lemma C.1. Let (Z ) ≥ be a sequence of non-negative random variables on a probability n n 1 P space (Ω, ). Then we have almost surely E log Z ≤ log (Z ) n n lim sup lim sup . →∞ n →∞ n n n E Proof. Let b > a > lim sup →∞ log (Zn). Then by Markov’s inequality, n n (cid:80) P { ≥ } ≤ E − ≤ − ( (1/n) log z b ) (Z )e bn en(a b) n n ∞ P{ ≥ } ∞ when n is large enough. Hence (1/n) log z b < . The Borel-Cantelli lemma ≤ n=1 n E implies that lim sup →∞ log Zn b almost surely. Letting b tend to lim sup →∞ log (Zn) n n n n (cid:3) yields the desired result. Appendix D. The following proposition explains how to transfer our result on the multifractal analysis × × { } of projections to Σ of Mandelbrot measures on Σ Σ to projections on [0, 1] 0 of Mandelbrot measures on [0, 1]2. Indeed, the natural projection form Σ to [0, 1] does not map cylinders to centered balls, so we need some additional argument. (cid:101) ∈ (cid:48) Proposition D.1. Let ρ be a positive and ﬁnite Borel measure on Σ. Let α [0, τ (0+)] (cid:101) ρ (cid:101) and suppose that there exists a positive and ﬁnite Borel measure ρ on Σ such that (cid:101) ≥ ∗ α (cid:101)∗ ρ (E(ρ, α)) > 0 and dim (ρ ) τ (α) > 0. Then dim E(ρ, α) = τ (α), where ρ α H α ρ H ρ and ρ respectively stand for the natural projections of ρ and ρ onto [0, 1]. α α 90 ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: which

text: is

text: bounded

text: by

text: a

text: constant

text: independent

text: of

text: x

text: since

text: T

text: (q)

text: is

text: ﬁnite

text: for

text: all

text: i

text: m

text: Ui

text: such

text: that

text: p

text: Also,

text: the

text: strong

text: law

text: of

text: large

text: numbers

text: yields

text: i

text: n

text: m

text: nl→im∞

text: n

text: log

text: m

text: TUxk

text: (q)

text: piTUi(q)

text: for

text: η-almost

text: every

text: x.

text: Consequently,

text: the

text: estimate

text: shows

text: that

text: for

text: η-almost

text: every

text: x

text: we

text: have

text: E(|X

text: (x|

text: )−E(Z)X

text: (x|

text: )|q)

text: which

text: implies

text: that

text: X

text: (x|

text: n

text: U

text: U

text: U

text: P

text: E

text: E

text: converges

text: -almost

text: surely

text: to

text: the

text: same

text: limit

text: as

text: (Z)X

text: (x|

text: that

text: is

text: (Z)X

text: (x).

text: U

text: U

text: Appendix

text: C.

text: Lemma

text: Let

text: (Z

text: be

text: a

text: sequence

text: of

text: non-negative

text: random

text: variables

text: on

text: a

text: probability

text: n

text: n

text: P

text: space

text: (Ω,

text: Then

text: we

text: have

text: almost

text: surely

text: E

text: log

text: Z

text: log

text: (Z

text: n

text: n

text: lim

text: sup

text: lim

text: sup

text: n

text: n

text: n

text: n

text: E

text: Proof.

text: Let

text: b

text: a

text: lim

text: sup

text: log

text: (Zn).

text: Then

text: by

text: Markov’s

text: inequality,

text: n

text: n

text: P

text: E

text: log

text: z

text: b

text: (Z

text: )e

text: bn

text: en(a

text: b)

text: n

text: n

text: P{

text: when

text: n

text: is

text: large

text: enough.

text: Hence

text: log

text: z

text: b

text: The

text: Borel-Cantelli

text: lemma

text: n

text: E

text: implies

text: that

text: lim

text: sup

text: log

text: Zn

text: b

text: almost

text: surely.

text: Letting

text: b

text: tend

text: to

text: lim

text: sup

text: log

text: (Zn)

text: n

text: n

text: n

text: n

text: yields

text: the

text: desired

text: result.

text: Appendix

text: D.

text: The

text: following

text: proposition

text: explains

text: how

text: to

text: transfer

text: our

text: result

text: on

text: the

text: multifractal

text: analysis

text: of

text: projections

text: to

text: Σ

text: of

text: Mandelbrot

text: measures

text: on

text: Σ

text: Σ

text: to

text: projections

text: on

text: of

text: Mandelbrot

text: measures

text: on

text: Indeed,

text: the

text: natural

text: projection

text: form

text: Σ

text: to

text: does

text: not

text: map

text: cylinders

text: to

text: centered

text: balls,

text: so

text: we

text: need

text: some

text: additional

text: argument.

text: Proposition

text: Let

text: ρ

text: be

text: a

text: positive

text: and

text: ﬁnite

text: Borel

text: measure

text: on

text: Σ.

text: Let

text: α

text: τ

text: ρ

text: and

text: suppose

text: that

text: there

text: exists

text: a

text: positive

text: and

text: ﬁnite

text: Borel

text: measure

text: ρ

text: on

text: Σ

text: such

text: that

text: α

text: ρ

text: (E(ρ,

text: α))

text: and

text: dim

text: (ρ

text: τ

text: (α)

text: Then

text: dim

text: E(ρ,

text: α)

text: τ

text: (α),

text: where

text: ρ

text: α

text: H

text: α

text: ρ

text: H

text: ρ

text: and

text: ρ

text: respectively

text: stand

text: for

text: the

text: natural

text: projections

text: of

text: ρ

text: and

text: ρ

text: onto

text: α

text: α

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: [2] F. Bahroun, I. Bhouri, Multifractals and projections, Extr. Math., 21 (2006), 83-91. [3] J. Barral, Moments, continuit´e, et analyse multifractale des martingales de Mandelbrot, Probab. Theory Related Fields, 113 (1999), 535–569. [4] J. Barral, Continuity of the multifractal spectrum of a random statistically self-similar measure, J. Theoret. Probab., 13 (2000), 1027–1060. [5] J. Barral, I. Bhouri, How projections aﬀect the validity of the multifractal formalism, Ergodic Theory Dynam. Systems, 31 (2011), 673–701. [6] J. Barral, D. J. Feng, Multifractal formalism for almost all self-aﬃne measures, Commun. Math. Phys., 318 (2013), 473-504. [7] F. Ben Nasr, Dimension de Hausdorﬀ de certains fractals alatoires, J. Th. Nomb. Bordeaux, 4 (1992), 129–140. [8] J. D. Biggins, A. Kyprianou, Measure change in multitype branching, Adv. Appl. Probab., 36 (2004), 544–581. [9] G. Brown, G. Michon, J. Peyri`ere, On the multifractal analysis of measures, J. Stat. Phys. 66 (1992), 775–790. [10] P. Collet, F. Koukiou, Large deviations for multiplicative chaos. Commun. Math. Phys., 147 (1992), 329–342. [11] R. Durrett, T. Liggett, Fixed points of the smoothing transformation, Z. Wahrsch. Verw. Gebiete, 64 (1983), 275-301. [12] J. T. Chayes, L. Chayes, R. Durrett, Connectivity properties of Mandelbrot’s percolation process, Probab. Theory Related Fields, 77 (1988), 307–324. [13] J. R. Chazottes, E. Ugalde, On the preservation of Gibbsianness under symbol amalgamation. In: Entropy of hidden Markov processes and connections to dynamical systems, 72-97, LMS Lecture Note No. 385, Cambridge Univ. Press, 2011. [14] P. Collet, J. L. Lebowitz, A. Porzio, The dimension spectrum of some dynamical systems, J. Stat. Phys., 47 (1987), 609–644. [15] C. D. Cutler, The Hausdorﬀ dimension distribution of ﬁnite measures in Euclidean spaces, Canad. J. Math., 88 (1986), 1459–1484. [16] C. D. Cutler, Measure disintegrations with respect to σ-stable monotone indices and pointwise rep- resentation of packing dimension. In: Proceedings of the 1990 Measure Theory Conference at Ober- walfach, Supplemento Ai Rendiconti del Circolo Matematico di Palerma, Ser. II, Vol. 28, 319–340, 1992. [17] D. Dacunha-Castelle, M. Duﬂo, Probabilit´es et statistiques 2. Probl`emes a` temps mobile, Masson, Paris, 1983. [18] F. M. Dekking, G. R. Grimmett, Superbranching processes and projections of random Cantor sets, Probab. Theory Related Fields, 78 (1988), 335-355. [19] F. M. Dekking, R. W. J. Meester, On the structure of Mandelbrot’s percolation process ans other random Cantor sets, J. Stat. Phys., 58 (1990), 1109–1126. [20] F. M. Dekking, K. Simon, On the size of the algebraic diﬀerence of two random cantor sets, Random Structures Algorithms, 32 (2008), 205-222. [21] A. Dembo, O. Zeitouni, Large deviations techniques and applications, vol. 38 of Applications of Mathematics. Springer-Verlag, New York, 2nd ed., 1998. [22] K. J. Falconer, Projections of random Cantor sets, J. Theoret. Probab., 2 (1989), 65–70. [23] K. J. Falconer, The multifractal spectrum of statistically self-similar measures, J. Theoret. Probab., 7 (1994), 681–702. [24] K. J. Falconer, Generalized dimensions of measures on self-aﬃne sets, Nonlinearity, 12 (1999), 877– 891. [25] K. J. Falconer, G. R. Grimmett, On the geometry of random Cantor sets and fractal percolation, J. Theoret. Probab., 5 (1992), 465–485. [26] K. J. Falconer, X. Jin, Exact dimensionality and projections of random self-similar measures and sets, J. London Math. Soc., 90 (2014), 388–412. [27] A. H. Fan, J. P. Kahane, Decomposition principle and random cascades. In: Third International Congress of Chinese Mathematicians. Part 1, 2, 447–456, AMS/IP Stud. Adv. Math., 42, pt. 1, 2, Amer. Math. Soc., Providence, RI, 2008. [28] D. J. Feng, Multifractal analysis of Bernoulli convolutions associated with Salem numbers. Adv. Math., 229 (2012), 3052–3077. 92

label: reference

text: F.

text: Bahroun,

text: I.

text: Bhouri,

text: Multifractals

text: and

text: projections,

text: Extr.

text: Math.,

text: J.

text: Barral,

text: Moments,

text: continuit´e,

text: et

text: analyse

text: multifractale

text: des

text: martingales

text: de

text: Mandelbrot,

text: Probab.

text: Theory

text: Related

text: Fields,

text: J.

text: Barral,

text: Continuity

text: of

text: the

text: multifractal

text: spectrum

text: of

text: a

text: random

text: statistically

text: self-similar

text: measure,

text: J.

text: Theoret.

text: Probab.,

text: J.

text: Barral,

text: I.

text: Bhouri,

text: How

text: projections

text: aﬀect

text: the

text: validity

text: of

text: the

text: multifractal

text: formalism,

text: Ergodic

text: Theory

text: Dynam.

text: Systems,

text: J.

text: Barral,

text: D.

text: J.

text: Feng,

text: Multifractal

text: formalism

text: for

text: almost

text: all

text: self-aﬃne

text: measures,

text: Commun.

text: Math.

text: Phys.,

text: F.

text: Ben

text: Nasr,

text: Dimension

text: de

text: Hausdorﬀ

text: de

text: certains

text: fractals

text: alatoires,

text: J.

text: Th.

text: Nomb.

text: Bordeaux,

text: J.

text: D.

text: Biggins,

text: A.

text: Kyprianou,

text: Measure

text: change

text: in

text: multitype

text: branching,

text: Adv.

text: Appl.

text: Probab.,

text: G.

text: Brown,

text: G.

text: Michon,

text: J.

text: Peyri`ere,

text: On

text: the

text: multifractal

text: analysis

text: of

text: measures,

text: J.

text: Stat.

text: Phys.

text: P.

text: Collet,

text: F.

text: Koukiou,

text: Large

text: deviations

text: for

text: multiplicative

text: chaos.

text: Commun.

text: Math.

text: Phys.,

text: R.

text: Durrett,

text: T.

text: Liggett,

text: Fixed

text: points

text: of

text: the

text: smoothing

text: transformation,

text: Z.

text: Wahrsch.

text: Verw.

text: Gebiete,

text: J.

text: T.

text: Chayes,

text: L.

text: Chayes,

text: R.

text: Durrett,

text: Connectivity

text: properties

text: of

text: Mandelbrot’s

text: percolation

text: process,

text: Probab.

text: Theory

text: Related

text: Fields,

text: J.

text: R.

text: Chazottes,

text: E.

text: Ugalde,

text: On

text: the

text: preservation

text: of

text: Gibbsianness

text: under

text: symbol

text: amalgamation.

text: In:

text: Entropy

text: of

text: hidden

text: Markov

text: processes

text: and

text: connections

text: to

text: dynamical

text: systems,

text: LMS

text: Lecture

text: Note

text: No.

text: Cambridge

text: Univ.

text: Press,

text: P.

text: Collet,

text: J.

text: L.

text: Lebowitz,

text: A.

text: Porzio,

text: The

text: dimension

text: spectrum

text: of

text: some

text: dynamical

text: systems,

text: J.

text: Stat.

text: Phys.,

text: C.

text: D.

text: Cutler,

text: The

text: Hausdorﬀ

text: dimension

text: distribution

text: of

text: ﬁnite

text: measures

text: in

text: Euclidean

text: spaces,

text: Canad.

text: J.

text: Math.,

text: C.

text: D.

text: Cutler,

text: Measure

text: disintegrations

text: with

text: respect

text: to

text: σ-stable

text: monotone

text: indices

text: and

text: pointwise

text: rep-

text: resentation

text: of

text: packing

text: dimension.

text: In:

text: Proceedings

text: of

text: the

text: Measure

text: Theory

text: Conference

text: at

text: Ober-

text: walfach,

text: Supplemento

text: Ai

text: Rendiconti

text: del

text: Circolo

text: Matematico

text: di

text: Palerma,

text: Ser.

text: II,

text: Vol.

text: D.

text: Dacunha-Castelle,

text: M.

text: Duﬂo,

text: Probabilit´es

text: et

text: statistiques

text: Probl`emes

text: a`

text: temps

text: mobile,

text: Masson,

text: Paris,

text: F.

text: M.

text: Dekking,

text: G.

text: R.

text: Grimmett,

text: Superbranching

text: processes

text: and

text: projections

text: of

text: random

text: Cantor

text: sets,

text: Probab.

text: Theory

text: Related

text: Fields,

text: F.

text: M.

text: Dekking,

text: R.

text: W.

text: J.

text: Meester,

text: On

text: the

text: structure

text: of

text: Mandelbrot’s

text: percolation

text: process

text: ans

text: other

text: random

text: Cantor

text: sets,

text: J.

text: Stat.

text: Phys.,

text: F.

text: M.

text: Dekking,

text: K.

text: Simon,

text: On

text: the

text: size

text: of

text: the

text: algebraic

text: diﬀerence

text: of

text: two

text: random

text: cantor

text: sets,

text: Random

text: Structures

text: Algorithms,

text: A.

text: Dembo,

text: O.

text: Zeitouni,

text: Large

text: deviations

text: techniques

text: and

text: applications,

text: vol.

text: of

text: Applications

text: of

text: Mathematics.

text: Springer-Verlag,

text: New

text: York,

text: ed.,

text: K.

text: J.

text: Falconer,

text: Projections

text: of

text: random

text: Cantor

text: sets,

text: J.

text: Theoret.

text: Probab.,

text: K.

text: J.

text: Falconer,

text: The

text: multifractal

text: spectrum

text: of

text: statistically

text: self-similar

text: measures,

text: J.

text: Theoret.

text: Probab.,

text: K.

text: J.

text: Falconer,

text: Generalized

text: dimensions

text: of

text: measures

text: on

text: self-aﬃne

text: sets,

text: Nonlinearity,

text: K.

text: J.

text: Falconer,

text: G.

text: R.

text: Grimmett,

text: On

text: the

text: geometry

text: of

text: random

text: Cantor

text: sets

text: and

text: fractal

text: percolation,

text: J.

text: Theoret.

text: Probab.,

text: K.

text: J.

text: Falconer,

text: X.

text: Jin,

text: Exact

text: dimensionality

text: and

text: projections

text: of

text: random

text: self-similar

text: measures

text: and

text: sets,

text: J.

text: London

text: Math.

text: Soc.,

text: A.

text: H.

text: Fan,

text: J.

text: P.

text: Kahane,

text: Decomposition

text: principle

text: and

text: random

text: cascades.

text: In:

text: Third

text: International

text: Congress

text: of

text: Chinese

text: Mathematicians.

text: Part

text: AMS/IP

text: Stud.

text: Adv.

text: Math.,

text: pt.

text: Amer.

text: Math.

text: Soc.,

text: Providence,

text: RI,

text: D.

text: J.

text: Feng,

text: Multifractal

text: analysis

text: of

text: Bernoulli

text: convolutions

text: associated

text: with

text: Salem

text: numbers.

text: Adv.

text: Math.,


text: Stochastic Function Norm Regularization of Deep Networks ∗ Amal Rannen Triki Matthew B. Blaschko Dept. of Computational Science and Engineering Center for Processing Speech and Images Yonsei University Departement Elektrotechniek Seoul, South Korea KU Leuven, Belgium amal.rannen@yonsei.ac.kr matthew.blaschko@esat.kuleuven.be Abstract Deep neural networks have had an enormous impact on image analysis. State-of- the-art training methods, based on weight decay and DropOut, result in impressive performance when a very large training set is available. However, they tend to have large problems overﬁtting to small data sets. Indeed, the available regularization methods deal with the complexity of the network function only indirectly. In this paper, we study the feasibility of directly using the L function norm for 2 regularization. Two methods to integrate this new regularization in the stochastic backpropagation are proposed. Moreover, the convergence of these new algorithms is studied. We ﬁnally show that they outperform the state-of-the-art methods in the low sample regime on benchmark datasets (MNIST and CIFAR10). The obtained results demonstrate very clear improvement, especially in the context of small sample regimes with data laying in a low dimensional manifold. Source code of the method can be found at https://github.com/AmalRT/DNN_Reg. 1 Introduction Deep Neural Networks (DNNs) have been shown to be a powerful tool in learning from large datasets in many domains. However, they require a large number of training samples in order to have reasonable generalization power. Indeed, this fact tends to limit their application to the type of images where it is easy to construct a large labeled database, such as natural images where collection can be done from image sharing websites and labels obtained at low cost via crowdsourcing. When only few labeled samples are available, such as in medical imaging, DNNs tend to overﬁt to the data and may not be competitive with methods for which known tractable methods for stronger regularization are a(cid:82)vailable, e.g. kernel methods. R Generally in learning theory, one wants to minimize the statistical risk (W) = (cid:80) (cid:96)(f (x), y)dP(x, y), where (cid:96) is a loss a function and P(x, y) is the data distribution of in- W puts and outputs, respectively. However, as P is generally unknown, only the empirical risk R (W) = 1 N (cid:96)(f (x ), y ) is accessible. Statistical learning theory [23] gives conditions for N N i=1 W i i the empirical risk to be a good approximation of the statistical risk, by giving a bound of the form: √ R ≤ R (W) (W) + g(h)/ N, (1) N where g(h) is an increasing function of h, the Vapnik–Chervonenkis dimension (VC dimension). This measure depends on the function set. When g(h) is small, the minimizer of the empirical risk gives a small statistical risk even when N is small. Through their activation function and their weights, DNNs ∗ ART is also afﬁliated with KU Leuven. ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: Stochastic

label: title

text: Function

text: Norm

text: Regularization

text: of

text: Deep

text: Networks

label: author

text: Amal

text: Rannen

text: Triki

text: Matthew

label: paragraph

text: B.

text: Blaschko

text: Dept.

text: of

text: Computational

text: Science

text: and

text: Engineering

text: Center

text: for

text: Processing

text: Speech

text: and

text: Images

text: Yonsei

text: University

text: Departement

text: Elektrotechniek

text: Seoul,

text: South

text: Korea

text: KU

text: Leuven,

text: Belgium

text: amal.rannen@yonsei.ac.kr

text: matthew.blaschko@esat.kuleuven.be

text: Abstract

text: Deep

label: abstract

text: neural

text: networks

text: have

text: had

text: an

text: enormous

text: impact

text: on

text: image

text: analysis.

text: State-of-

text: the-art

text: training

text: methods,

text: based

text: on

text: weight

text: decay

text: and

text: DropOut,

text: result

text: in

text: impressive

text: performance

text: when

text: a

text: very

text: large

text: training

text: set

text: is

text: available.

text: However,

text: they

text: tend

text: to

text: have

text: large

text: problems

text: overﬁtting

text: to

text: small

text: data

text: sets.

text: Indeed,

text: the

text: available

text: regularization

text: methods

text: deal

text: with

text: the

text: complexity

text: of

text: the

text: network

text: function

text: only

text: indirectly.

text: In

text: this

text: paper,

text: we

text: study

text: the

text: feasibility

text: of

text: directly

text: using

text: the

text: L

text: function

text: norm

text: for

text: regularization.

text: Two

text: methods

text: to

text: integrate

text: this

text: new

text: regularization

text: in

text: the

text: stochastic

text: backpropagation

text: are

text: proposed.

text: Moreover,

text: the

text: convergence

text: of

text: these

text: new

text: algorithms

text: is

text: studied.

text: We

text: ﬁnally

text: show

text: that

text: they

text: outperform

text: the

text: state-of-the-art

text: methods

text: in

text: the

text: low

text: sample

text: regime

text: on

text: benchmark

text: datasets

text: (MNIST

text: and

text: The

text: obtained

text: results

text: demonstrate

text: very

text: clear

text: improvement,

text: especially

text: in

text: the

text: context

text: of

text: small

text: sample

text: regimes

text: with

text: data

text: laying

text: in

text: a

text: low

text: dimensional

text: manifold.

text: Source

text: code

text: of

text: the

text: method

text: can

text: be

text: found

text: at

text: Introduction

text: Deep

text: Neural

text: Networks

text: (DNNs)

text: have

text: been

text: shown

text: to

text: be

text: a

text: powerful

text: tool

text: in

text: learning

text: from

text: large

text: datasets

text: in

text: many

text: domains.

text: However,

text: they

text: require

text: a

text: large

text: number

text: of

text: training

text: samples

text: in

text: order

text: to

text: have

text: reasonable

text: generalization

text: power.

text: Indeed,

text: this

text: fact

text: tends

text: to

text: limit

text: their

text: application

text: to

text: the

text: type

text: of

text: images

text: where

text: it

text: is

text: easy

text: to

text: construct

text: a

text: large

text: labeled

text: database,

text: such

text: as

text: natural

text: images

text: where

text: collection

text: can

text: be

text: done

text: from

text: image

text: sharing

text: websites

text: and

text: labels

text: obtained

text: at

text: low

text: cost

text: via

text: crowdsourcing.

text: When

text: only

text: few

text: labeled

text: samples

text: are

text: available,

text: such

text: as

text: in

text: medical

text: imaging,

text: DNNs

text: tend

text: to

text: overﬁt

text: to

text: the

text: data

text: and

text: may

text: not

text: be

text: competitive

text: with

text: methods

text: for

text: which

text: known

text: tractable

text: methods

text: for

text: stronger

text: regularization

text: are

text: e.g.

text: kernel

text: methods.

text: R

text: Generally

text: in

text: learning

text: theory,

text: one

text: wants

text: to

text: minimize

text: the

text: statistical

text: risk

text: (W)

text: (x),

text: y)dP(x,

text: y),

text: where

text: is

text: a

text: loss

text: a

text: function

text: and

text: P(x,

text: y)

text: is

text: the

text: data

text: distribution

text: of

text: in-

text: W

text: puts

text: and

text: outputs,

text: respectively.

text: However,

text: as

text: P

text: is

text: generally

text: unknown,

text: only

text: the

text: empirical

text: risk

text: R

text: (W)

text: N

text: (x

text: y

text: is

text: accessible.

text: Statistical

text: learning

text: theory

text: gives

text: conditions

text: for

text: N

text: N

text: W

text: i

text: i

text: the

text: empirical

text: risk

text: to

text: be

text: a

text: good

text: approximation

text: of

text: the

text: statistical

text: risk,

text: by

text: giving

text: a

text: bound

text: of

text: the

text: form:

text: R

text: R

text: (W)

text: (W)

text: g(h)/

text: N,

text: N

text: where

text: g(h)

text: is

text: an

text: increasing

text: function

text: of

text: h,

text: the

text: Vapnik–Chervonenkis

text: dimension

text: (VC

text: dimension).

text: This

text: measure

text: depends

text: on

text: the

text: function

text: set.

text: When

text: g(h)

text: is

text: small,

text: the

text: minimizer

text: of

text: the

text: empirical

text: risk

text: gives

text: a

text: small

text: statistical

text: risk

text: even

text: when

text: N

text: is

text: small.

text: Through

text: their

text: activation

text: function

text: and

text: their

text: weights,

text: DNNs

text: ART

text: is

text: also

text: afﬁliated

text: with

text: KU

text: Leuven.

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: index very efﬁciently a large function space with few parameters [9]. However, the VC dimension of this function set tends to be high. Indeed, many researchers have studied the VC-dimension of some simple neural networks [2, 14, 22]. It has been shown that, for example, for networks O with only sigmoidal activation functions, the optimal bound on VC dimension is (ρ2), where ρ is the number of weights. The authors of [2] also show that a deep network implement functions of higher complexity than a shallow network with the same number of weights (i.e. with the same generalization power according to the VC-dimension bounds), proving one of the many advantages of DNNs. Nevertheless, the DNNs that have become popular recently tend to have a high number of weights (For example, the Oxford VGG networks have more that 110 million parameters [21]), which makes their generalization power decrease signiﬁcantly for small databases. For this reason, an appropriate regularization is required. Training a DNN with regularization is solving the following problem: R arg min (W) + λΩ(f ). (2) N W W Much research has approached the idea of regularized DNN training. However, until now, this regularization tends to be poor compared for examples to regularization in kernel methods [20]. The common idea behind regularization is the application of Occam’s razor: if several solutions exist for a given problem, we favor the “simplest” solution. Which notion of simplicity is employed typically is determined by prior knowledge or computational considerations. Two categories of regularization can be distinguished in the existing literature: (i) regularizing by controlling the magnitude of the weights W((cid:96) and (cid:96) regularization [11, 16]); (ii) regularizing by controlling the network architecture 1 2 (DropOut [13] and DropConnect [25]). In order to make the regularization stronger, we study in this (cid:90) paper the feasibility of using L function norm: 2 (cid:107) (cid:107) (cid:104) (cid:105) Ω(f ) = f 2 = f (x), f (x) dx. (3) W W 2 W W As the exact value of this norm, and more importantly of its gradient, are not accessible, a good approximation is needed. Two approximations are considered. The ﬁrst idea is to consider the function norm with respect to the probability measure (we suppose (cid:90) for simplicity that all the distributions admit density functions): (cid:104) (cid:105) Ω(f ) = f (x), f (x) dP(x), (4) W W W (cid:90) where P is the marginal data distribution: P(x) := P(x, y)dy. (5) One can easily see that this is a valid function norm [10], and furthermore is just a weighted canonical function norm such that the regularization has a higher effect on the data that has the higher probability to be observed. The second idea is to use a Markov-chain sampling method for numerical integration [1]. Here, the (cid:107) (cid:107) idea is to generate samples from a probability density proportional to f (x) 2 (vector (cid:96) norm), W 2 2 and to use them to ap(cid:90)proximate the integral by: (cid:88) (cid:104) (cid:105) ≈ (cid:107) (cid:107)− n (cid:104) (cid:105) f (x), f (x) dx α 1 α f (x ), f (x ) (6) W W 1 i W i W i i=1 for some positive weights α and samples x . The generated samples will be concentrated around the (cid:107) (cid:107) i i higher values of f (x) 2, and the regularization will have a higher effect on these regions. W 2 2

text: index

text: very

text: efﬁciently

text: a

text: large

text: function

text: space

text: with

text: few

text: parameters

text: However,

text: the

text: VC

text: dimension

text: of

text: this

text: function

text: set

text: tends

text: to

text: be

text: high.

text: Indeed,

text: many

text: researchers

text: have

text: studied

text: the

text: VC-dimension

text: of

text: some

text: simple

text: neural

text: networks

text: It

text: has

text: been

text: shown

text: that,

text: for

text: example,

text: for

text: networks

text: O

text: with

text: only

text: sigmoidal

text: activation

text: functions,

text: the

text: optimal

text: bound

text: on

text: VC

text: dimension

text: is

text: where

text: ρ

text: is

text: the

text: number

text: of

text: weights.

text: The

text: authors

text: of

text: also

text: show

text: that

text: a

text: deep

text: network

text: implement

text: functions

text: of

text: higher

text: complexity

text: than

text: a

text: shallow

text: network

text: with

text: the

text: same

text: number

text: of

text: weights

text: (i.e.

text: with

text: the

text: same

text: generalization

text: power

text: according

text: to

text: the

text: VC-dimension

text: bounds),

text: proving

text: one

text: of

text: the

text: many

text: advantages

text: of

text: DNNs.

text: Nevertheless,

text: the

text: DNNs

text: that

text: have

text: become

text: popular

text: recently

text: tend

text: to

text: have

text: a

text: high

text: number

text: of

text: weights

text: (For

text: example,

text: the

text: Oxford

text: VGG

text: networks

text: have

text: more

text: that

text: million

text: parameters

text: which

text: makes

text: their

text: generalization

text: power

text: decrease

text: signiﬁcantly

text: for

text: small

text: databases.

text: For

text: this

text: reason,

text: an

text: appropriate

text: regularization

text: is

text: required.

text: Training

text: a

text: DNN

text: with

text: regularization

text: is

text: solving

text: the

text: following

text: problem:

text: R

text: arg

text: min

text: (W)

text: λΩ(f

text: N

text: W

text: W

text: Much

text: research

text: has

text: approached

text: the

text: idea

text: of

text: regularized

text: DNN

text: training.

text: However,

text: until

text: now,

text: this

text: regularization

text: tends

text: to

text: be

text: poor

text: compared

text: for

text: examples

text: to

text: regularization

text: in

text: kernel

text: methods

text: The

text: common

text: idea

text: behind

text: regularization

text: is

text: the

text: application

text: of

text: Occam’s

text: razor:

text: if

text: several

text: solutions

text: exist

text: for

text: a

text: given

text: problem,

text: we

text: favor

text: the

text: “simplest”

text: solution.

text: Which

text: notion

text: of

text: simplicity

text: is

text: employed

text: typically

text: is

text: determined

text: by

text: prior

text: knowledge

text: or

text: computational

text: considerations.

text: Two

text: categories

text: of

text: regularization

text: can

text: be

text: distinguished

text: in

text: the

text: existing

text: literature:

text: (i)

text: regularizing

text: by

text: controlling

text: the

text: magnitude

text: of

text: the

text: weights

text: and

text: regularization

text: (ii)

text: regularizing

text: by

text: controlling

text: the

text: network

text: architecture

text: (DropOut

text: and

text: DropConnect

text: In

text: order

text: to

text: make

text: the

text: regularization

text: stronger,

text: we

text: study

text: in

text: this

text: paper

text: the

text: feasibility

text: of

text: using

text: L

text: function

text: norm:

text: Ω(f

text: f

text: f

text: (x),

text: f

text: (x)

text: dx.

text: W

text: W

text: W

text: W

text: As

text: the

text: exact

text: value

text: of

text: this

text: norm,

text: and

text: more

text: importantly

text: of

text: its

text: gradient,

text: are

text: not

text: accessible,

text: a

text: good

text: approximation

text: is

text: needed.

text: Two

text: approximations

text: are

text: considered.

text: The

text: ﬁrst

text: idea

text: is

text: to

text: consider

text: the

text: function

text: norm

text: with

text: respect

text: to

text: the

text: probability

text: measure

text: (we

text: suppose

text: for

text: simplicity

text: that

text: all

text: the

text: distributions

text: admit

text: density

text: functions):

text: Ω(f

text: f

text: (x),

text: f

text: (x)

text: dP(x),

text: W

text: W

text: W

text: where

text: P

text: is

text: the

text: marginal

text: data

text: distribution:

text: P(x)

text: P(x,

text: y)dy.

text: One

text: can

text: easily

text: see

text: that

text: this

text: is

text: a

text: valid

text: function

text: norm

text: and

text: furthermore

text: is

text: just

text: a

text: weighted

text: canonical

text: function

text: norm

text: such

text: that

text: the

text: regularization

text: has

text: a

text: higher

text: effect

text: on

text: the

text: data

text: that

text: has

text: the

text: higher

text: probability

text: to

text: be

text: observed.

text: The

text: second

text: idea

text: is

text: to

text: use

text: a

text: Markov-chain

text: sampling

text: method

text: for

text: numerical

text: integration

text: Here,

text: the

text: idea

text: is

text: to

text: generate

text: samples

text: from

text: a

text: probability

text: density

text: proportional

text: to

text: f

text: (x)

text: (vector

text: norm),

text: W

text: and

text: to

text: use

text: them

text: to

text: the

text: integral

text: by:

text: n

text: f

text: (x),

text: f

text: (x)

text: dx

text: α

text: α

text: f

text: (x

text: f

text: (x

text: W

text: W

text: i

text: W

text: i

text: W

text: i

text: for

text: some

text: positive

text: weights

text: α

text: and

text: samples

text: x

text: The

text: generated

text: samples

text: will

text: be

text: concentrated

text: around

text: the

text: i

text: i

text: higher

text: values

text: of

text: f

text: (x)

text: and

text: the

text: regularization

text: will

text: have

text: a

text: higher

text: effect

text: on

text: these

text: regions.

text: W


text: (cid:0) (cid:1) (cid:0) (cid:1) ∇ (cid:107) (cid:107) is replaced by (cid:96)(f(x ), y ) + λ f(r ) 2) where r is drawn from a density proportional (cid:107) (cid:107) W t t t 2 ∇ t (cid:107) (cid:107) to f(x)) 2, or equivalently, according to (10), (cid:96)(f(x ), y ) + λα(r , z ) f(r ) 2) where 2 W t t t t t 2 (r , z ) is drawn from the data distribution. If we suppose that we can permute integration and t t derivation, it is straightforward to show that the two updates obey to the condition (14). Much research considers the question of the convergence of this procedure. Many of them are based on the results of [19]. Nemirovsky et al [18] showed that if C(W) is convex, and if its gradient is O Lipshitz continuous and bounded, then the optimal convergence rate is (1) and is obtained for O t γ = (1). In this case, the speed of decrease of γ has the best compromise between the speed of t t t convergence and the control of the variance introduced by using one sample rather than all samples to approximate the gradient. Bottou [4, 5, 6] has shown this property holds for less constraining conditions on C. In [3], he has studied in particular the type of objective that one wants to minimize in neural net- (cid:80) works. These objectives are not convex and may admit several minimums. He shows that (cid:80) ∃ ∀ E ∇ as long as: (i) C , W, C(W) > C , (ii) ∼ [H(x, y, W)] = C(W), E min m≤in (x,y) P(x,y) ≥ W∞ (iii) ∼ [H(x, y, W)T H(x, y, W)] A + BC(W), A, B 0, (iv) γ = and (x,y) ∞P(x,y) ∇ t t γ2 < , C(W ) converges and C(W ) converges to 0, using SGD. We know that t t t W t the second condition holds for both of the cases we study. Thus, for our methods, knowing E that the two other conditions on C and H hold for C (W) = ∼ [(cid:96)(f (x), y)] and ∇ 0 (x,y) P(x,y) W H (x , y , W ) = (cid:96)(f (x), y ) (which are the classical settings for neural networks), we need to0protvetthattthey stWill hoWldt after tadding the regularization term. Let C , A and B be the min,0 0 0 parameters that ensures the ﬁrst and third conditions for C and H . 0 0 First case: In this case, we can rewrite the objective as : E (cid:107) (cid:107) C (W) = C (W) + λ ∼ [ f (x) 2] 1 0 (x,y) P(x,y) W 2 ∇ (cid:107) (cid:107) H (x , y , W ) = H (x , y , W ) + λ f (x ) 2 (15) 1 t t t 0 t t t W Wt t 2 Thus : H (x , y , W ) = H (x , y , W ) + 2λJ f (x )T f (x ). (16) 1 t t t 0 t t t W Wt t Wt t ≥ E (cid:107) (cid:107) ≥ Then, as λ 0 and ∼ [ f (x) 2] 0, it follows that C (W) > C . In addition, we (x,y) P(x,y) W 2 1 min,0 have: E E [H (x, y, W)T H (x, y, W)] = [H (x, y, W)T H (x, y, W)] 1 1 0 0 E + 2λ [H (x, y, W)T J f (x)T f (x) + f (x)T J f (x)H (x, y, W)] 0 W W W W W W 0 E + 4λ2 [f (x)T J f (x)J f (x)T f (x)] (17) W W W W W W We know that the ﬁrst term is bounded above by A + B C (W). Let us suppose that H , f and 0 0 0 0 W J f are bounded. This is a reasonable hypothesis as we want to control the L norm of f . W W 2 W Moreover, when J f is bounded, the condition on H is veriﬁed for many popular loss functions W W 0 such as the cross-entropy loss. All our experiments are conducted with this loss. Thus, we can ﬁnd A E 1 that bounds the second term, and A such that the third term is bounded by 4λ2A [f (x)T f (x)]. 2 − 2 W W Finally, it sufﬁces to ﬁnd λ such that 4λ2A B is positive, which will ﬁnish the proof of the third 2 0 condition for C and H . 1 1 Second case: In this case, we can rewrite the objective as : E (cid:107) (cid:107) C (W) = C (W) + λ ∼ [α(x, y) f (x) 2] 2 0 (x,y) P(x,y) W 2 ∇ (cid:107) (cid:107) H (x , y , W ) = H (x , y , W ) + λ α(x , y ) f (x ) 2 (18) 2 t t t 0 t t t W t t Wt t 2 5 ##LTLine## ##LTLine##

text: is

text: replaced

text: by

text: y

text: λ

text: f(r

text: where

text: r

text: is

text: drawn

text: from

text: a

text: density

text: proportional

text: W

text: t

text: t

text: t

text: t

text: to

text: f(x))

text: or

text: equivalently,

text: according

text: to

text: y

text: λα(r

text: z

text: f(r

text: where

text: W

text: t

text: t

text: t

text: t

text: t

text: (r

text: z

text: is

text: drawn

text: from

text: the

text: data

text: distribution.

text: If

text: we

text: suppose

text: that

text: we

text: can

text: permute

text: integration

text: and

text: t

text: t

text: derivation,

text: it

text: is

text: straightforward

text: to

text: show

text: that

text: the

text: two

text: updates

text: obey

text: to

text: the

text: condition

text: Much

text: research

text: considers

text: the

text: question

text: of

text: the

text: convergence

text: of

text: this

text: procedure.

text: Many

text: of

text: them

text: are

text: based

text: on

text: the

text: results

text: of

text: Nemirovsky

text: et

text: al

text: showed

text: that

text: if

text: C(W)

text: is

text: convex,

text: and

text: if

text: its

text: gradient

text: is

text: O

text: Lipshitz

text: continuous

text: and

text: bounded,

text: then

text: the

text: optimal

text: convergence

text: rate

text: is

text: and

text: is

text: obtained

text: for

text: O

text: t

text: γ

text: In

text: this

text: case,

text: the

text: speed

text: of

text: decrease

text: of

text: γ

text: has

text: the

text: best

text: compromise

text: between

text: the

text: speed

text: of

text: t

text: t

text: t

text: convergence

text: and

text: the

text: control

text: of

text: the

text: variance

text: introduced

text: by

text: using

text: one

text: sample

text: rather

text: than

text: all

text: samples

text: to

text: approximate

text: the

text: gradient.

text: Bottou

text: has

text: shown

text: this

text: property

text: holds

text: for

text: less

text: constraining

text: conditions

text: on

text: C.

text: In

text: he

text: has

text: studied

text: in

text: particular

text: the

text: type

text: of

text: objective

text: that

text: one

text: wants

text: to

text: minimize

text: in

text: neural

text: net-

text: works.

text: These

text: objectives

text: are

text: not

text: convex

text: and

text: may

text: admit

text: several

text: minimums.

text: He

text: shows

text: that

text: E

text: as

text: long

text: as:

text: (i)

text: C

text: W,

text: C(W)

text: C

text: (ii)

text: [H(x,

text: y,

text: W)]

text: C(W),

text: E

text: min

text: m≤in

text: (x,y)

text: P(x,y)

text: W∞

text: (iii)

text: [H(x,

text: y,

text: W)T

text: H(x,

text: y,

text: W)]

text: A

text: BC(W),

text: A,

text: B

text: (iv)

text: γ

text: and

text: (x,y)

text: ∞P(x,y)

text: t

text: t

text: γ2

text: C(W

text: converges

text: and

text: C(W

text: converges

text: to

text: using

text: SGD.

text: We

text: know

text: that

text: t

text: t

text: t

text: W

text: t

text: the

text: second

text: condition

text: holds

text: for

text: both

text: of

text: the

text: cases

text: we

text: study.

text: Thus,

text: for

text: our

text: methods,

text: knowing

text: E

text: that

text: the

text: two

text: other

text: conditions

text: on

text: C

text: and

text: H

text: hold

text: for

text: C

text: (W)

text: (x),

text: y)]

text: and

text: (x,y)

text: P(x,y)

text: W

text: H

text: (x

text: y

text: W

text: (x),

text: y

text: (which

text: are

text: the

text: classical

text: settings

text: for

text: neural

text: networks),

text: we

text: need

text: stWill

text: hoWldt

text: after

text: tadding

text: the

text: regularization

text: term.

text: Let

text: C

text: A

text: and

text: B

text: be

text: the

text: parameters

text: that

text: ensures

text: the

text: ﬁrst

text: and

text: third

text: conditions

text: for

text: C

text: and

text: H

text: First

text: case:

text: In

text: this

text: case,

text: we

text: can

text: rewrite

text: the

text: objective

text: as

text: E

text: C

text: (W)

text: C

text: (W)

text: λ

text: f

text: (x)

text: (x,y)

text: P(x,y)

text: W

text: H

text: (x

text: y

text: W

text: H

text: (x

text: y

text: W

text: λ

text: f

text: (x

text: t

text: t

text: t

text: t

text: t

text: t

text: W

text: Wt

text: t

text: Thus

text: H

text: (x

text: y

text: W

text: H

text: (x

text: y

text: W

text: f

text: (x

text: )T

text: f

text: (x

text: t

text: t

text: t

text: t

text: t

text: t

text: W

text: Wt

text: t

text: Wt

text: t

text: E

text: Then,

text: as

text: λ

text: and

text: f

text: (x)

text: it

text: follows

text: that

text: C

text: (W)

text: C

text: In

text: addition,

text: we

text: (x,y)

text: P(x,y)

text: W

text: have:

text: E

text: E

text: [H

text: (x,

text: y,

text: W)T

text: H

text: (x,

text: y,

text: W)]

text: [H

text: (x,

text: y,

text: W)T

text: H

text: (x,

text: y,

text: W)]

text: E

text: 2λ

text: [H

text: (x,

text: y,

text: W)T

text: J

text: f

text: (x)T

text: f

text: (x)

text: f

text: (x)T

text: J

text: f

text: (x)H

text: (x,

text: y,

text: W)]

text: W

text: W

text: W

text: W

text: W

text: W

text: E

text: [f

text: (x)T

text: J

text: f

text: (x)J

text: f

text: (x)T

text: f

text: (x)]

text: W

text: W

text: W

text: W

text: W

text: W

text: We

text: know

text: that

text: the

text: ﬁrst

text: term

text: is

text: bounded

text: above

text: by

text: A

text: B

text: C

text: (W).

text: Let

text: us

text: suppose

text: that

text: H

text: f

text: and

text: W

text: J

text: f

text: are

text: bounded.

text: This

text: is

text: a

text: reasonable

text: hypothesis

text: as

text: we

text: want

text: to

text: control

text: the

text: L

text: norm

text: of

text: f

text: W

text: W

text: W

text: Moreover,

text: when

text: J

text: f

text: is

text: bounded,

text: the

text: condition

text: on

text: H

text: is

text: veriﬁed

text: for

text: many

text: popular

text: loss

text: functions

text: W

text: W

text: such

text: as

text: the

text: cross-entropy

text: loss.

text: All

text: our

text: experiments

text: are

text: conducted

text: with

text: this

text: loss.

text: Thus,

text: we

text: can

text: ﬁnd

text: A

text: E

text: that

text: bounds

text: the

text: second

text: term,

text: and

text: A

text: such

text: that

text: the

text: third

text: term

text: is

text: bounded

text: by

text: [f

text: (x)T

text: f

text: (x)].

text: W

text: W

text: Finally,

text: it

text: sufﬁces

text: to

text: ﬁnd

text: λ

text: such

text: that

text: B

text: is

text: positive,

text: which

text: will

text: ﬁnish

text: the

text: proof

text: of

text: the

text: third

text: condition

text: for

text: C

text: and

text: H

text: Second

text: case:

text: In

text: this

text: case,

text: we

text: can

text: rewrite

text: the

text: objective

text: as

text: E

text: C

text: (W)

text: C

text: (W)

text: λ

text: [α(x,

text: y)

text: f

text: (x)

text: (x,y)

text: P(x,y)

text: W

text: H

text: (x

text: y

text: W

text: H

text: (x

text: y

text: W

text: λ

text: α(x

text: y

text: f

text: (x

text: t

text: t

text: t

text: t

text: t

text: t

text: W

text: t

text: t

text: Wt

text: t

text: ##LTLine##

text: ##LTLine##


text: 6 [14] H.Shen, M. Dutta, J. Appl. Phys. 78, 2151 (1995). [20] Ajit Srivastava, Rahul Srivastava, Jigang Wang, and Junichiro Kono, Phys.Rev.Lett. 93, 157401 (2004). [15] J.K. Wahlstrand, J.E. Sipe, Phys. Rev. B82, 075206 (2010). [21] Y. Mizumoto, Y. Kayanuma, A. Srivastava, J. Kono, and A. H. Chin, Phys. Rev. B 74, 045216 (2006). [16] F. Duque-Gomez, J.E. Sipe, J. Phys. Chem. Solids 76, 138 (2[2021]5)S.. Ghimire, A.D. DiChiara, E. Sistrunk, U.B. Szafruga, P. Agostini, L. F. DiMauro, and D. A. Reis, Phys. Rev. Lett. 107, [17] Y. Yacoby, Phys. Rev. 169, 610 (1968). 167407 (2011). [18] A. P. Jauho and K. Johnsen, Phys.Rev. Lett. 76, 4576 (1996[)2.3] T. Otobe, Y. Shinohara, S. A. Sato, and K. Yabana, Phys. Rev. B 93, 045124 (2016). [19] K. B. Nordstrom, K. Johnsen, S. J. Allen, A. P. Jauho, B. B[2ir4n]irW, J. .VK. oHnoou,sTto.nN, oPdhay,s.HR. eAvk.i5ya1m, 1a8,4an(1d94H0.).Sakaki, Phys. Rev. Lett. 81, 457 (1998). [25] H. R. Reiss and V. P. Krainov, J. Phys. A: Math. Gen. 36, 5575 (2003).

text: H.Shen,

text: M.

text: Dutta,

text: J.

text: Appl.

text: Phys.

text: Ajit

text: Srivastava,

text: Rahul

text: Srivastava,

text: Jigang

text: Wang,

text: and

text: Junichiro

text: Kono,

text: Phys.Rev.Lett.

text: J.K.

text: Wahlstrand,

text: J.E.

text: Sipe,

text: Phys.

text: Rev.

text: Y.

text: Mizumoto,

text: Y.

text: Kayanuma,

text: A.

text: Srivastava,

text: J.

text: Kono,

text: and

text: A.

text: H.

text: Chin,

text: Phys.

text: Rev.

text: B

text: F.

text: Duque-Gomez,

text: J.E.

text: Sipe,

text: J.

text: Phys.

text: Chem.

text: Solids

text: Ghimire,

text: A.D.

text: DiChiara,

text: E.

text: Sistrunk,

text: U.B.

text: Szafruga,

text: P.

text: Agostini,

text: L.

text: F.

text: DiMauro,

text: and

text: D.

text: A.

text: Reis,

text: Phys.

text: Rev.

text: Lett.

text: Y.

text: Yacoby,

text: Phys.

text: Rev.

text: A.

text: P.

text: Jauho

text: and

text: K.

text: Johnsen,

text: Phys.Rev.

text: Lett.

text: T.

text: Otobe,

text: Y.

text: Shinohara,

text: S.

text: A.

text: Sato,

text: and

text: K.

text: Yabana,

text: Phys.

text: Rev.

text: B

text: K.

text: B.

text: Nordstrom,

text: K.

text: Johnsen,

text: S.

text: J.

text: Allen,

text: A.

text: P.

text: Jauho,

text: B.

text: J.

text: .VK.

text: oHnoou,sTto.nN,

text: oPdhay,s.HR.

text: Phys.

text: Rev.

text: Lett.

text: H.

text: R.

text: Reiss

text: and

text: V.

text: P.

text: Krainov,

text: J.

text: Phys.

text: A:

text: Math.

text: Gen.


text: (cid:34) (cid:35) (cid:88) | ∼ ∼ πsu∈pΠ Eπ (cid:34)n(cid:88)N=1 Un(Yn − c) (cid:35) Htheewreﬁhreθsrtjen∼Yijt,eNnm(θsµj.j,n, σNj2,n(θ)j,rexpλ2j,2rnes)enatnsdoθujr beliNef(µofj,nθ,jσaj2,fnte)r.  N · · · − Therefo(cid:40)re for each subproblem, after the arrival of the nth = sup Eπ U (x Y + + x Y c) ∈ (cid:88) (cid:88)n 1,n 1,n k,n k,n item, we can update our parameters as the following: π Π   n=1 = sup Eπ N k U (x Y − x (cid:107) c (cid:107)) . (3) µj,n = λ2βj,n−λ12µβjj,,nn−−11++xY2jj,,nn−−11x2j,n−1 if Uj,n−1 = 1; π∈Π n j,n j,n j,n Xn µj,n−1 if Uj,n−1 = 0. n=1 j=1 The precision of our beliefs (which is the inverse of the (cid:48) (cid:48) allNowowdiwffeeriennttrofdeuactueretws ocannewm(cid:48) apkoelictyhesiertsowΠn0 daencdisΠion, swUhijc,nh purpidoar/tpedosatesrifoorll(cid:40)ovwarsi:ance with initial value βj,0 = σj12,0 ) is for the nth item. Further, Π has an additional restriction that U1,n = · · · =U(cid:88)j,n.(cid:88)Base0d on the deﬁnition, we have βj,n = ββj,n−−1 + x2jλ,n2−1 iiff UUj,n−−1 == 10.; j,n 1 j,n 1 (3) = sup Eπ(cid:48)  N k U (x Y − x (cid:107) c (cid:107)) The jth single-feature subproblem can be solved using (cid:48)∈ (cid:48) (cid:88) (cid:88) j,n j,n j,n j,n X dynamic programming with a three-dimensional state space π Π0 n=1 j=1 n  (µ , σ , x ), where µ and σ are the mean and j,n j,n j,n j,n j,n ≤ (cid:48) N k − c variance of our current belief about θ and x is the sup Eπ U (x Y x (cid:107) (cid:107)) . j j,n π(cid:48)∈Π(cid:48) n=1 j=1 j,n j,n j,n j,n Xn cbuyrrtehnet icteomnd’sitijotnhalfedaitsutrreib. uItniiotnialloyf, µθj,0gaivnednσθj,−0 araendgivthene (4) ∼ j j prior distribution θ N(µ, Σ). Upon each item’s arrival, we move to another state based on the updating formula Since the supremum of a summation is less or equal to the sum(cid:88)mation of a su(cid:34)p(cid:88)remum, we have (cid:35) described above. Deﬁne Qj(µ, σ, x, 0) and Qj(µ, σ, x, 1) be the total reward to go if you decided to discard the item and forward the item respectively, ≤ k (cid:48) N − c (cid:88) (4) sup Eπ U (x Y x (cid:107) (cid:107)) . (cid:48)∈ (cid:48) j,n j,n j,n j,n X ∞ j=1 π Π n=1 n (cid:48)(cid:48) − − (5) Q (µ, σ, x, U) = sup Eπ [ γn 1U (x Y x c) j (cid:48)(cid:48)∈ j,x j,n j,n j,n |π ∼Πj n=1 j =Th1e,n2,b·a·s·e,dk,onwethkeno(cid:34)dweﬁnition of our policy set Πj, (cid:35)for θj N(µ, σ2), xj,1 = x, Uj,1 = U]. (cid:88) (cid:88) Then the Bellman equation for this problem is: ≤ k (cid:48)(cid:48) N − c (5) sup Eπ U (x Y x (cid:107) (cid:107)) , (cid:48)(cid:48)∈ j,n j,n j,n j,n X j=1 π Πj n=1 n V (µ, σ, x) = max Q (µ, σ, x, U). (7) j j U=0,1 which concludes the proof of the theorem. This calculation is summarized as Algorithm 1. We emphasize that this computational upper bound holds We may improve our upper bound by taking its minimum true in general, even when the different components of Xn with a hindsight upper bound, derived in the following are correlated. Numerical experiments in Section suggest way. We ﬁrst consider a larger class of policies that may that the optimality gap between this upper bound and the (cid:80) additionally base their decisions on full knowledge of θ. An best heuristic policy is typically small. optimal policy among this larger class of policies forwards For simplicity, in this paper we focus on the special case · the nth item to the user only if θ X > c, and the expected wuphpeerre bouki=nd1 xini,nT=heo1r.eWme1n.oTwo dcioscmupsustecotmhipsutqautiaonntitoyf, twhee total (cid:34)re(cid:88)ward of this optim(cid:35)al policy isn(cid:2) (cid:3) must solve(cid:34)th(cid:88)ese k subproblems: (cid:35) N · − γ · − E (θ X c)+ = − E (θ X c)+ . (8) N − · · · n=1 n 1 γ 1 sup Eπ U (x Y x c) , j = 1, 2, , k, ∈ j,n j,n j,n j,n π Πj n=1 Since (8) is the supremum of the same objective as (2), (6) but over a larger set of policies, it forms an upper bound. ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

label: equation

text: πsu∈pΠ

text: Eπ

text: Un(Yn

text: c)

text: Htheewreﬁhreθsrtjen∼Yijt,eNnm(θsµj.j,n,

text: N

text: for

text: each

text: subproblem,

text: after

text: the

text: arrival

text: of

text: the

text: nth

text: sup

text: Eπ

text: U

text: (x

text: Y

text: x

text: Y

text: c)

text: k,n

text: k,n

text: item,

text: we

text: can

text: update

text: our

text: parameters

text: as

text: the

text: following:

text: π

text: Π

text: sup

text: Eπ

text: N

text: k

text: U

text: (x

text: Y

text: x

text: c

text: µj,n

text: if

text: π∈Π

text: n

text: j,n

text: j,n

text: j,n

text: Xn

text: if

text: The

text: precision

text: of

text: our

text: beliefs

text: (which

text: is

text: the

text: inverse

text: of

text: the

text: allNowowdiwffeeriennttrofdeuactueretws

text: daencdisΠion,

text: swUhijc,nh

text: with

text: initial

text: value

text: is

text: for

text: the

text: nth

text: item.

text: Further,

text: Π

text: has

text: an

text: additional

text: restriction

text: that

text: on

text: the

text: deﬁnition,

text: we

text: have

text: βj,n

text: ββj,n−−1

text: iiff

text: j,n

text: j,n

text: sup

text: N

text: k

text: U

text: (x

text: Y

text: x

text: c

text: The

text: jth

text: single-feature

text: subproblem

text: can

text: be

text: solved

text: using

text: j,n

text: j,n

text: j,n

text: j,n

text: X

text: dynamic

text: programming

text: with

text: a

text: three-dimensional

text: state

text: space

text: π

text: Π0

text: n

text: (µ

text: σ

text: x

text: where

text: µ

text: and

text: σ

text: are

text: the

text: mean

text: and

text: j,n

text: j,n

text: j,n

text: j,n

text: j,n

text: N

text: k

text: c

text: variance

text: of

text: our

text: current

text: belief

text: about

text: θ

text: and

text: x

text: is

text: the

text: sup

text: Eπ

text: U

text: (x

text: Y

text: x

text: j

text: j,n

text: j,n

text: j,n

text: j,n

text: j,n

text: Xn

text: cbuyrrtehnet

text: icteomnd’sitijotnhalfedaitsutrreib.

text: uItniiotnialloyf,

text: araendgivthene

text: j

text: j

text: prior

text: distribution

text: θ

text: N(µ,

text: Σ).

text: Upon

text: each

text: item’s

text: arrival,

text: we

text: move

text: to

text: another

text: state

text: based

text: on

text: the

text: updating

text: formula

text: Since

text: the

text: supremum

text: of

text: a

text: summation

text: is

text: less

text: or

text: equal

text: to

text: the

text: of

text: a

text: we

text: have

text: described

text: above.

text: Deﬁne

text: Qj(µ,

text: σ,

text: x,

text: and

text: Qj(µ,

text: σ,

text: x,

text: be

text: the

text: total

text: reward

text: to

text: go

text: if

text: you

text: decided

text: to

text: discard

text: the

text: item

text: and

text: forward

text: the

text: item

text: respectively,

text: k

text: N

text: c

text: sup

text: Eπ

text: U

text: (x

text: Y

text: x

text: j,n

text: j,n

text: j,n

text: j,n

text: X

text: π

text: Π

text: n

text: Q

text: (µ,

text: σ,

text: x,

text: U)

text: sup

text: Eπ

text: γn

text: (x

text: Y

text: x

text: c)

text: j

text: j,x

text: j,n

text: j,n

text: j,n

text: |π

text: ∼Πj

text: j

text: of

text: our

text: policy

text: set

text: Πj,

text: θj

text: N(µ,

text: x,

text: U].

text: Then

text: the

text: Bellman

text: equation

text: for

text: this

text: problem

text: is:

text: k

text: N

text: c

text: sup

text: Eπ

text: U

text: (x

text: Y

text: x

text: j,n

text: j,n

text: j,n

text: j,n

text: X

text: π

text: Πj

text: n

text: V

text: (µ,

text: σ,

text: x)

text: max

text: Q

text: (µ,

text: σ,

text: x,

text: U).

text: j

text: j

text: which

text: concludes

text: the

text: proof

text: of

text: the

text: theorem.

text: This

text: calculation

text: is

text: summarized

text: as

text: Algorithm

text: We

text: emphasize

text: that

text: this

text: computational

text: upper

text: bound

text: holds

text: We

text: may

text: improve

text: our

text: upper

text: bound

text: by

text: taking

text: its

text: minimum

text: true

text: in

text: general,

text: even

text: when

text: the

text: different

text: components

text: of

text: Xn

text: with

text: a

text: hindsight

text: upper

text: bound,

text: derived

text: in

text: the

text: following

text: are

text: correlated.

text: Numerical

text: experiments

text: in

text: Section

text: suggest

text: way.

text: We

text: ﬁrst

text: consider

text: a

text: larger

text: class

text: of

text: policies

text: that

text: may

text: that

text: the

text: optimality

text: gap

text: between

text: this

text: upper

text: bound

text: and

text: the

text: additionally

text: base

text: their

text: decisions

text: on

text: full

text: knowledge

text: of

text: θ.

text: An

text: best

text: heuristic

text: policy

text: is

text: typically

text: small.

text: optimal

text: policy

text: among

text: this

text: larger

text: class

text: of

text: policies

text: forwards

text: For

text: simplicity,

text: in

text: this

text: paper

text: we

text: focus

text: on

text: the

text: special

text: case

text: the

text: nth

text: item

text: to

text: the

text: user

text: only

text: if

text: θ

text: X

text: c,

text: and

text: the

text: expected

text: wuphpeerre

text: twhee

text: total

text: of

text: this

text: policy

text: must

text: k

text: subproblems:

text: N

text: γ

text: E

text: (θ

text: X

text: c)+

text: E

text: (θ

text: X

text: c)+

text: N

text: n

text: γ

text: sup

text: Eπ

text: U

text: (x

text: Y

text: x

text: c)

text: j

text: k,

text: j,n

text: j,n

text: j,n

text: j,n

text: π

text: Πj

text: Since

text: is

text: the

text: supremum

text: of

text: the

text: same

text: objective

text: as

text: but

text: over

text: a

text: larger

text: set

text: of

text: policies,

text: it

text: forms

text: an

text: upper

text: bound.

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: Fig. 2: A telnet session of the PyRIDE interactive interface. It shows the array of PR2 speciﬁc functions provided through PyPR2 extension module. client running on an iPad device. Camera images are streaming to the client in realtime. B. Design Choices The design and development of PyRIDE represents a small important step to further improve the programming environments for the PR2 (and other) robots. We attempted to balance simplicity and easy to use with the need of Fig. 3: A live screenshot of a PyRIDE remote iPad client. programming ﬂexibility and support for extensive testing. A The screenshot shows a testing session for a human detection number of deliberate design decisions were made to achieve subsystem running on our PR2 robot. our goals. First, similar to ROS and other systems[6], we adopt Python as the main programming language due to its clear syntax and is easy to learn and use. There is already a large one uniﬁed programming interface so that programmers can population of Python developers. Unlike the existing systems immediately utilise them without any additional efforts to such as ROS that incorporate Python through the standard set up correct access. For example, to obtain the current module extension mechanism, PyRIDE embeds a Python joint positions of a PR2 robot arm, one can simply call interpreter directly at its core and it becomes a self-contained PyPR2.getArmJointPositions method in PyRIDE development environment. Combined with a remote interac- Python environment. In contrast, the standard approach in tive shell service that uses the standard telnet communication ROS requires many lines of boilerplate code to set up neces- protocol[7], PyRIDE provides the same interactive program- sary communication proxy channel to the appropriate ROS ming facility as the standard interactive Python interpreter node and run an event loop in order to retrieve the relevant (Fig. 2). This means that a programmer has direct remote data. The standard ROS programming approach becomes access to the functionalities provided by PyRIDE without cumbersome and error prone when a programmer starts to installing additional software on his/her local development develop complex robot behaviours that involve the coordi- machine, apart from telnet. No user speciﬁc account needs nation and execution of multiple underlying ROS nodes. to be created and maintained on the PR2 robot’s computer With the PyRIDE function integration framework, advanced in order for a new user to start programming on the robot. It PR2 developers can integrate their existing software modules also means that one can develop programs on PyRIDE from together under PyRIDE and expose their functionalities to a (almost any) personal computer with a network connection signiﬁcantly broader programmer audience. to the robot2. PyRIDE helps a PR2 programmer to focus Third, PyRIDE is designed to automatically search and on his/her work by minimising communication and adminis- load a default Python script on startup. Through this boot- trative overhead and removing distractions such as working strap mechanism, any scripts developed in the embedded with the underlying Linux system. Python environment can be immediately deployed. PyRIDE Second, the integration framework with robot function enables seamless transition between software development wrapper brings the existing robot functions together under and software deployment. A typical software development to deployment in PyRIDE starts with the development and 2The interactive shell service can be switched off so to prevent unautho- rised access after software development moves to the deployment phase. testing of code using the interactive mode. Afterwards, the ##LTFigure## ##LTFigure##

text: Fig.

text: A

label: caption

text: telnet

text: session

text: of

text: the

text: PyRIDE

text: interactive

text: interface.

text: It

text: shows

text: the

text: array

text: of

text: speciﬁc

text: functions

text: provided

text: through

text: extension

text: module.

text: client

label: list

text: running

text: on

text: an

text: iPad

text: device.

text: Camera

text: images

text: are

text: streaming

text: to

text: the

text: client

text: in

text: realtime.

text: B.

label: section

text: Design

text: Choices

text: The

text: design

text: and

text: development

text: of

text: PyRIDE

text: represents

text: a

text: small

text: important

text: step

text: to

text: further

text: improve

text: the

text: programming

text: environments

text: for

text: the

text: (and

text: other)

text: robots.

text: We

text: attempted

text: to

text: balance

text: simplicity

text: and

text: easy

text: to

text: use

text: with

text: the

text: need

text: of

text: Fig.

text: A

text: live

text: screenshot

text: of

text: a

text: PyRIDE

text: remote

text: iPad

text: client.

text: programming

text: ﬂexibility

text: and

text: support

text: for

text: extensive

text: testing.

text: A

text: The

text: screenshot

text: shows

text: a

text: testing

text: session

text: for

text: a

text: human

text: detection

text: number

text: of

text: deliberate

text: design

text: decisions

text: were

text: made

text: to

text: achieve

text: subsystem

text: running

text: on

text: our

text: robot.

text: our

text: goals.

text: First,

text: similar

text: to

text: ROS

text: and

text: other

text: we

text: adopt

text: Python

text: as

text: the

text: main

text: programming

text: language

text: due

text: to

text: its

text: clear

text: syntax

text: and

text: is

text: easy

text: to

text: learn

text: and

text: use.

text: There

text: is

text: already

text: a

text: large

text: one

text: uniﬁed

text: programming

text: interface

text: so

text: that

text: programmers

text: can

text: population

text: of

text: Python

text: developers.

text: Unlike

text: the

text: existing

text: systems

text: immediately

text: utilise

text: them

text: without

text: any

text: additional

text: efforts

text: to

text: such

text: as

text: ROS

text: that

text: incorporate

text: Python

text: through

text: the

text: standard

text: set

text: up

text: correct

text: access.

text: For

text: example,

text: to

text: obtain

text: the

text: current

text: module

text: extension

text: mechanism,

text: PyRIDE

text: embeds

text: a

text: Python

text: joint

text: positions

text: of

text: a

text: robot

text: arm,

text: one

text: can

text: simply

text: call

text: interpreter

text: directly

text: at

text: its

text: core

text: and

text: it

text: becomes

text: a

text: self-contained

text: method

text: in

text: PyRIDE

text: development

text: environment.

text: Combined

text: with

text: a

text: remote

text: interac-

text: Python

text: environment.

text: In

text: contrast,

text: the

text: standard

text: approach

text: in

text: tive

text: shell

text: service

text: that

text: uses

text: the

text: standard

text: telnet

text: communication

text: ROS

text: requires

text: many

text: lines

text: of

text: boilerplate

text: code

text: to

text: set

text: up

text: neces-

text: PyRIDE

text: provides

text: the

text: same

text: interactive

text: program-

text: sary

text: communication

text: proxy

text: channel

text: to

text: the

text: appropriate

text: ROS

text: ming

text: facility

text: as

text: the

text: standard

text: interactive

text: Python

text: interpreter

text: node

text: and

text: run

text: an

text: event

text: loop

text: in

text: order

text: to

text: retrieve

text: the

text: relevant

text: (Fig.

text: This

text: means

text: that

text: a

text: programmer

text: has

text: direct

text: remote

text: data.

text: The

text: standard

text: ROS

text: programming

text: approach

text: becomes

text: access

text: to

text: the

text: functionalities

text: provided

text: by

text: PyRIDE

text: without

text: cumbersome

text: and

text: error

text: prone

text: when

text: a

text: programmer

text: starts

text: to

text: installing

text: additional

text: software

text: on

text: his/her

text: local

text: development

text: develop

text: complex

text: robot

text: behaviours

text: that

text: involve

text: the

text: coordi-

text: machine,

text: apart

text: from

text: telnet.

text: No

text: user

text: speciﬁc

text: account

text: needs

text: nation

text: and

text: execution

text: of

text: multiple

text: underlying

text: ROS

text: nodes.

text: to

text: be

text: created

text: and

text: maintained

text: on

text: the

text: robot’s

text: computer

text: With

text: the

text: PyRIDE

text: function

text: integration

text: framework,

text: advanced

text: in

text: order

text: for

text: a

text: new

text: user

text: to

text: start

text: programming

text: on

text: the

text: robot.

text: It

text: developers

text: can

text: integrate

text: their

text: existing

text: software

text: modules

text: also

text: means

text: that

text: one

text: can

text: develop

text: programs

text: on

text: PyRIDE

text: from

text: together

text: under

text: PyRIDE

text: and

text: expose

text: their

text: functionalities

text: to

text: a

text: (almost

text: any)

text: personal

text: computer

text: with

text: a

text: network

text: connection

text: signiﬁcantly

text: broader

text: programmer

text: audience.

text: to

text: the

text: PyRIDE

text: helps

text: a

text: programmer

text: to

text: focus

text: Third,

text: PyRIDE

text: is

text: designed

text: to

text: automatically

text: search

text: and

text: on

text: his/her

text: work

text: by

text: minimising

text: communication

text: and

text: adminis-

text: load

text: a

text: default

text: Python

text: script

text: on

text: startup.

text: Through

text: this

text: boot-

text: trative

text: overhead

text: and

text: removing

text: distractions

text: such

text: as

text: working

text: strap

text: mechanism,

text: any

text: scripts

text: developed

text: in

text: the

text: embedded

text: with

text: the

text: underlying

text: Linux

text: system.

text: Python

text: environment

text: can

text: be

text: immediately

text: deployed.

text: PyRIDE

text: Second,

text: the

text: integration

text: framework

text: with

text: robot

text: function

text: enables

text: seamless

text: transition

text: between

text: software

text: development

text: wrapper

text: brings

text: the

text: existing

text: robot

text: functions

text: together

text: under

text: and

text: software

text: deployment.

text: A

text: typical

text: software

text: development

text: to

text: deployment

text: in

text: PyRIDE

text: starts

text: with

text: the

text: development

text: and

text: interactive

text: shell

text: service

text: can

text: be

text: switched

text: off

text: so

text: to

text: prevent

text: unautho-

text: rised

text: access

text: after

text: software

text: development

text: moves

text: to

text: the

text: deployment

text: phase.

text: testing

text: of

text: code

text: using

text: the

text: interactive

text: mode.

text: Afterwards,

text: the

text: ##LTFigure##

text: ##LTFigure##


text: given to important words.2 standard train/validation/test split, containing 550k, The idea of ”Inner-attention” was inspired by the 10k, and 10k samples, respectively. observation that when human read one sentence, 3.2 Parameter Setting people usually can roughly form an intuition about The training objective of our model is cross- which part of the sentence is more important accord- entropy loss, and we use minibatch SGD with the ing past experience. And we implemented this idea Rmsprop (Tieleman and Hinton, 2012) for opti- using attention mechanism in our model. The atten- mization. The batch size is 128. A dropout layer was tion mechanism is formalized as follows: applied in the output of the network with the dropout rate set to 0.25. In our model, we used pretrained ⊗ 300D Glove 840B vectors (Pennington et al., 2014) M = tanh(W yY + W hR e ) ave L to initialize the word embedding. Out-of-vocabulary α = softmax(wT M) words in the training set are randomly initialized by R = Y αT sampling values uniformly from (0.05, 0.05). All att of these embedding are not updated during train- ing . We didn’t tune representations of words for where Y is a matrix consisting of output vec- two reasons: 1. To reduced the number of param- tors of biLSTM, Rave is the output of mean pooling eters needed to train. 2. Keep their representa- layer, α denoted the attention vector and Ratt is the tion stays close to unseen similar words in inference attention-weighted sentence representation. time, which improved the model’s generation abil- 2.2 Sentence Matching Module ity. The model is implemented using open-source Once the sentence vectors are generated. Three framework Keras.3 matching methods were applied to extract relations 3.3 The Input Strategy between premise and hypothesis. • In this part, we investigated four strategies to Concatenation of the two representations • modify the input on our basic model which helps us Element-wise product • increase performance, the four strategies are: Element-wise difference • Inverting Premises (Sutskever et al., 2014) • Doubling Premises (Zaremba and Sutskever, This matching architecture was ﬁrst used by (Mou 2014) et al., 2015). Finally, we used a SoftMax layer over • Doubling Hypothesis the output of a non-linear projection of the generated • Differentiating Inputs (Removing same words matching vector for classiﬁcation. appeared in premises and hypothesis) 3 Experiments Experimental results were illustrated in Table 2. 3.1 DataSet As we can see from it, doubling hypothesis and differentiating inputs both improved our model’s To evaluate the performance of our model, we performance.While the hypothesises usually much conducted our experiments on Stanford Natural Lan- shorter than premises, doubling hypothesis may ab- guage Inference (SNLI) corpus (Bos and Markert, sorb this difference and emphasize the meaning 2005). At 570K pairs, SNLI is two orders of magni- twice via this strategy. Differentiating input strat- tude larger than all other resources of its type. The egy forces the model to focus on different part of dataset is constructed by crowdsourced efforts, each the two sentences which may help the classiﬁcation for Neutral and Contradiction examples as we ob- sentence written by humans. The target labels com- served that our model tended to assign unconﬁdent prise three classes: Entailment, Contradiction, and instances to Entailment. And the original input sen- Neutral (two irrelevant sentences). We applied the tences appeared in Figure 1 are: 2Recently, (Yang et al., 2016) proposed a Hierarchical At- Premise: Two man in polo shirts and tan pants im- tention model on the task of document classiﬁcation also used mersed in a pleasant conversation about photograph. for but the target representation in attention their mechanism is randomly initialized. 3http://keras.io/ ##LTLine## ##LTLine##

text: given

text: to

text: important

text: standard

text: train/validation/test

text: split,

text: containing

text: The

text: idea

text: of

text: ”Inner-attention”

text: was

text: inspired

text: by

text: the

text: and

text: samples,

text: respectively.

text: observation

text: that

text: when

text: human

text: read

text: one

text: sentence,

text: Parameter

text: Setting

text: people

text: usually

text: can

text: roughly

text: form

text: an

text: intuition

text: about

text: The

text: training

text: objective

text: of

text: our

text: model

text: is

text: cross-

text: which

text: part

text: of

text: the

text: sentence

text: is

text: more

text: important

text: accord-

text: entropy

text: loss,

text: and

text: we

text: use

text: minibatch

text: SGD

text: with

text: the

text: ing

text: past

text: experience.

text: And

text: we

text: implemented

text: this

text: idea

text: Rmsprop

text: (Tieleman

text: and

text: Hinton,

text: for

text: opti-

text: using

text: attention

text: mechanism

text: in

text: our

text: model.

text: The

text: atten-

text: mization.

text: The

text: batch

text: size

text: is

text: A

text: dropout

text: layer

text: was

text: tion

text: mechanism

text: is

text: formalized

text: as

text: follows:

text: applied

text: in

text: the

text: output

text: of

text: the

text: network

text: with

text: the

text: dropout

text: rate

text: set

text: to

text: In

text: our

text: model,

text: we

text: used

text: pretrained

text: Glove

text: vectors

text: (Pennington

text: et

text: al.,

text: M

text: tanh(W

text: yY

text: W

text: hR

text: e

text: ave

text: L

text: to

text: initialize

text: the

text: word

text: embedding.

text: Out-of-vocabulary

text: α

text: softmax(wT

text: M)

text: words

text: in

text: the

text: training

text: set

text: are

text: randomly

text: initialized

text: by

text: R

text: Y

text: αT

text: sampling

text: values

text: uniformly

text: from

text: All

text: att

text: of

text: these

text: embedding

text: are

text: not

text: updated

text: during

text: train-

text: ing

text: We

text: didn’t

text: tune

text: representations

text: of

text: words

text: for

text: where

text: Y

text: is

text: a

text: matrix

text: consisting

text: of

text: output

text: vec-

text: two

text: reasons:

text: To

text: reduced

text: the

text: number

text: of

text: param-

text: tors

text: of

text: biLSTM,

text: Rave

text: is

text: the

text: output

text: of

text: mean

text: pooling

text: eters

text: needed

text: to

text: train.

text: Keep

text: their

text: representa-

text: layer,

text: α

text: denoted

text: the

text: attention

text: vector

text: and

text: Ratt

text: is

text: the

text: tion

text: stays

text: close

text: to

text: unseen

text: similar

text: words

text: in

text: inference

text: attention-weighted

text: sentence

text: representation.

text: time,

text: which

text: improved

text: the

text: model’s

text: generation

text: abil-

text: Sentence

text: Matching

text: Module

text: ity.

text: The

text: model

text: is

text: implemented

text: using

text: open-source

text: Once

text: the

text: sentence

text: vectors

text: are

text: generated.

text: Three

text: framework

text: matching

text: methods

text: were

text: applied

text: to

text: extract

text: relations

text: The

text: Input

text: Strategy

text: between

text: premise

text: and

text: hypothesis.

text: In

text: this

text: part,

text: we

text: investigated

text: four

text: strategies

text: to

text: Concatenation

text: of

text: the

text: two

text: representations

text: modify

text: the

text: input

text: on

text: our

text: basic

text: model

text: which

text: helps

text: us

text: Element-wise

text: product

text: increase

text: performance,

text: the

text: four

text: strategies

text: are:

text: Element-wise

text: difference

text: Inverting

text: Premises

text: (Sutskever

text: et

text: al.,

text: Doubling

text: Premises

text: (Zaremba

text: and

text: Sutskever,

text: This

text: matching

text: architecture

text: was

text: ﬁrst

text: used

text: by

text: (Mou

text: et

text: al.,

text: Finally,

text: we

text: used

text: a

text: SoftMax

text: layer

text: over

text: Doubling

text: Hypothesis

text: the

text: output

text: of

text: a

text: non-linear

text: projection

text: of

text: the

text: generated

text: Differentiating

text: Inputs

text: (Removing

text: same

text: words

text: matching

text: vector

text: for

text: classiﬁcation.

text: appeared

text: in

text: premises

text: and

text: hypothesis)

text: Experiments

text: Experimental

text: results

text: were

text: illustrated

text: in

text: Table

text: DataSet

text: As

text: we

text: can

text: see

text: from

text: it,

text: doubling

text: hypothesis

text: and

text: differentiating

text: inputs

text: both

text: improved

text: our

text: model’s

text: To

text: evaluate

text: the

text: performance

text: of

text: our

text: model,

text: we

text: performance.While

text: the

text: hypothesises

text: usually

text: much

text: conducted

text: our

text: experiments

text: on

text: Stanford

text: Natural

text: Lan-

text: shorter

text: than

text: premises,

text: doubling

text: hypothesis

text: may

text: ab-

text: guage

text: Inference

text: (SNLI)

text: corpus

text: (Bos

text: and

text: Markert,

text: sorb

text: this

text: difference

text: and

text: emphasize

text: the

text: meaning

text: At

text: pairs,

text: SNLI

text: is

text: two

text: orders

text: of

text: magni-

text: twice

text: via

text: this

text: strategy.

text: Differentiating

text: input

text: strat-

text: tude

text: larger

text: than

text: all

text: other

text: resources

text: of

text: its

text: type.

text: The

text: egy

text: forces

text: the

text: model

text: to

text: focus

text: on

text: different

text: part

text: of

text: dataset

text: is

text: constructed

text: by

text: crowdsourced

text: efforts,

text: each

text: the

text: two

text: sentences

text: which

text: may

text: help

text: the

text: classiﬁcation

text: for

text: Neutral

text: and

text: Contradiction

text: examples

text: as

text: we

text: ob-

text: sentence

text: written

text: by

text: humans.

text: The

text: target

text: labels

text: com-

text: served

text: that

text: our

text: model

text: tended

text: to

text: assign

text: unconﬁdent

text: prise

text: three

text: classes:

text: Entailment,

text: Contradiction,

text: and

text: instances

text: to

text: Entailment.

text: And

text: the

text: original

text: input

text: sen-

text: Neutral

text: (two

text: irrelevant

text: sentences).

text: We

text: applied

text: the

text: tences

text: appeared

text: in

text: Figure

text: are:

text: (Yang

text: et

text: al.,

text: proposed

text: a

text: Hierarchical

text: At-

text: Premise:

text: Two

text: man

text: in

text: polo

text: shirts

text: and

text: tan

text: pants

text: im-

text: tention

text: model

text: on

text: the

text: task

text: of

text: document

text: classiﬁcation

text: also

text: used

text: mersed

text: in

text: a

text: pleasant

text: conversation

text: about

text: photograph.

text: for

text: but

text: the

text: target

text: representation

text: in

text: attention

text: their

text: mechanism

text: is

text: randomly

text: initialized.

text: ##LTLine##

text: ##LTLine##


text: TABLE III: The branching ratio(in %) of exclusive semileptonic and nonleptonic decay B to c X(3940), X(4160) with a = 1.14. 1 Mode Ours [12] Mode Ours → × × → × B+ X(3940)e+ν 1.02 10−2 1.9+0.2+0.1+0.0+0.8+0.7+0.0 10−2 B+ X(4160)e+ν 2.39 10−3 c e −0.1−0.1−0.0−0.9−0.7−0.0 c e → × × → × B+ X(3940)τ+ν 3.00 10−4 5.7+0.6+0.7+0.3+2.4+2.0+0.0 10−4 B+ X(4160)τ+ν 1.78 10−5 c τ −0.3−0.4−0.3−2.7−2.2−0.1 c τ → × → × B+ X(3940)+π 6.78 10−3 – B+ X(4160)+π 2.05 10−3 c c → × → × B+ X(3940)+K 4.94 10−4 – B+ X(4160)+K 1.44 10−4 c c → × → × B+ X(3940)+ρ 1.34 10−2 – B+ X(4160)+ρ 3.73 10−3 c c → × → × B+ X(3940)+K∗ 7.44 10−4 – B+ X(4160)+K∗ 2.00 10−4 c c 10-4 10-5 1.08 2.7 1.06 2.6 1.04 2.5 1.02 2.4 1.00 2.3 0.98 2.2 0.96 2.1 3.92 3.93 3.94 3.95 3.96 4.12 4.14 4.16 4.18 4.20 M GeV M GeV FIG. 7: The relations of branching ratios to the mass of ﬁnal mesons. In conclusion, considering X(3940) and X(4160) as η (3S) and η (4S) states, we study c c the semileptonic and nonleptonic B decays to X(3940) and X(4160) by the improved B-S c method. The corresponding decay form factors are calculated and the corresponding decay widths and branching ratios are obtained. The exclusive decay widths and branching ratios are very small, because of the minus value in the wavefunctions. But now the Large Hadron × Collider (LHC) will produce as much as 5 1010 B events per year [42, 43], if we can c observe the suﬃcient events, some channels will provide us a sizable ratios, and may be we will detect the productions of X(3940) and X(4160) in B exclusive weak semileptonic and c nonleptonic decay. That will provide us a new way to observe the X(3940) and X(4160) in the future. 12 ) +e)0493(X B(r e c ) +e)0614(X B(r e c ##LTFigure## ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: TABLE

text: III:

text: The

text: branching

text: ratio(in

text: of

text: exclusive

text: semileptonic

text: and

text: nonleptonic

text: decay

text: B

text: to

text: c

text: with

text: a

text: Mode

label: table

text: Ours

text: Mode

text: Ours

text: B+

text: B+

text: c

text: e

text: c

text: e

text: B+

text: B+

text: c

text: τ

text: c

text: τ

text: B+

text: B+

text: c

text: c

text: B+

text: B+

text: c

text: c

text: B+

text: B+

text: c

text: c

text: B+

text: B+

text: c

text: c

text: M

text: GeV

text: M

text: GeV

text: FIG.

text: The

text: relations

text: of

text: branching

text: ratios

text: to

text: the

text: mass

text: of

text: ﬁnal

text: mesons.

text: In

text: conclusion,

text: considering

text: and

text: as

text: η

text: and

text: η

text: states,

text: we

text: study

text: c

text: c

text: the

text: semileptonic

text: and

text: nonleptonic

text: B

text: decays

text: to

text: and

text: by

text: the

text: improved

text: B-S

text: c

text: method.

text: The

text: corresponding

text: decay

text: form

text: factors

text: are

text: calculated

text: and

text: the

text: corresponding

text: decay

text: widths

text: and

text: branching

text: ratios

text: are

text: obtained.

text: The

text: exclusive

text: decay

text: widths

text: and

text: branching

text: ratios

text: are

text: very

text: small,

text: because

text: of

text: the

text: minus

text: value

text: in

text: the

text: wavefunctions.

text: But

text: now

text: the

text: Large

text: Hadron

text: Collider

text: (LHC)

text: will

text: produce

text: as

text: much

text: as

text: B

text: events

text: per

text: year

text: if

text: we

text: can

text: c

text: observe

text: the

text: suﬃcient

text: events,

text: some

text: channels

text: will

text: provide

text: us

text: a

text: sizable

text: ratios,

text: and

text: may

text: be

text: we

text: will

text: detect

text: the

text: productions

text: of

text: and

text: in

text: B

text: exclusive

text: weak

text: semileptonic

text: and

text: c

text: nonleptonic

text: decay.

text: That

text: will

text: provide

text: us

text: a

text: new

text: way

text: to

text: observe

text: the

text: and

text: in

text: the

text: future.

text: B(r

text: e

text: c

text: B(r

text: e

text: c

text: ##LTFigure##

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: For the diﬀerent JPC (or JP ) states, we give the general form of wave functions. Reducing the wave functions by the last equation of Eq. (A7), then solving the ﬁrst and second equations in Eq. (A7) to get the wave functions and mass spectrum. We have discussed the solution of the Salpeter equation in detail in Ref. [55, 56]. The normalization condition(cid:34) for BS wave function is: (cid:35) (cid:90) q2 dq P/ P/ − P/ P/ T T T r ϕ++ ϕ++ ϕ−− ϕ−− = 2P . (A8) 2π2 M M M M 0 In our model, the instantaneous interaction kernel V is Cornell potential, which is the sum of a linear scalar interaction and a vector interaction: ⊗ − ⊗ 4 α V (r) = V (r) + V + γ γ0V (r) = λr + V γ γ0 s , (A9) s 0 0 v 0 0 3 r where λ is the string constant and α ((cid:126)q) is the running coupling constant. In order to ﬁt s the data of heavy quarkonia, a constant V is often added to conﬁne potential. One can see 0 that V (r) diverges at r = 0, we introduce a factor e−αr to avoid the divergence: v λ − −4 α V (r) = (1 e−αr) , V (r) = s e−αr . (A10) s α v 3 r (cid:28) It is easy to know that when αr 1, the potential becomes to Eq. (A9). In the momentum space and the C.M.S of the bound state, the potential reads : ⊗ V ((cid:126)q) = V ((cid:126)q) + γ γ0V ((cid:126)q) , s 0 v − λ λ 1 − 2 α ((cid:126)q) V ((cid:126)q) = ( + V )δ3((cid:126)q) + , V ((cid:126)q) = s , (A11) s α 0 π2 ((cid:126)q2 + α2)2 v 3π2 ((cid:126)q2 + α2) where the running coupling constant α ((cid:126)q) is : s 12π 1 α ((cid:126)q) = − . s 33 2N log(a + q(cid:126)2 ) f Λ2 QCD We introduce a small parameter a to avoid the divergence in the denominator. The constants ¯ λ, α, V and Λ are the parameters that characterize the potential. N = 3 for bq (and 0 QCD f c¯q) system. 15 ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: For

text: the

text: diﬀerent

text: JPC

text: (or

text: JP

text: states,

text: we

text: give

text: the

text: general

text: form

text: of

text: wave

text: functions.

text: Reducing

text: the

text: wave

text: functions

text: by

text: the

text: last

text: equation

text: of

text: Eq.

text: then

text: solving

text: the

text: ﬁrst

text: and

text: second

text: equations

text: in

text: Eq.

text: to

text: get

text: the

text: wave

text: functions

text: and

text: mass

text: spectrum.

text: We

text: have

text: discussed

text: the

text: solution

text: of

text: the

text: Salpeter

text: equation

text: in

text: detail

text: in

text: Ref.

text: The

text: normalization

text: for

text: BS

text: wave

text: function

text: is:

text: dq

text: P/

text: P/

text: P/

text: P/

text: T

text: T

text: T

text: r

text: ϕ++

text: ϕ++

text: ϕ−−

text: ϕ−−

text: M

text: M

text: M

text: M

text: In

text: our

text: model,

text: the

text: instantaneous

text: interaction

text: kernel

text: V

text: is

text: Cornell

text: potential,

text: which

text: is

text: the

text: sum

text: of

text: a

text: linear

text: scalar

text: interaction

text: and

text: a

text: vector

text: interaction:

text: α

text: V

text: (r)

text: V

text: (r)

text: V

text: γ

text: (r)

text: λr

text: V

text: γ

text: γ0

text: s

text: s

text: v

text: r

text: where

text: λ

text: is

text: the

text: string

text: constant

text: and

text: α

text: is

text: the

text: running

text: coupling

text: constant.

text: In

text: order

text: to

text: ﬁt

text: s

text: the

text: data

text: of

text: heavy

text: quarkonia,

text: a

text: constant

text: V

text: is

text: often

text: added

text: to

text: conﬁne

text: potential.

text: One

text: can

text: see

text: that

text: V

text: (r)

text: diverges

text: at

text: r

text: we

text: introduce

text: a

text: factor

text: e−αr

text: to

text: avoid

text: the

text: divergence:

text: v

text: λ

text: −4

text: α

text: V

text: (r)

text: e−αr)

text: V

text: (r)

text: s

text: e−αr

text: s

text: α

text: v

text: r

text: It

text: is

text: easy

text: to

text: know

text: that

text: when

text: αr

text: the

text: potential

text: becomes

text: to

text: Eq.

text: In

text: the

text: momentum

text: space

text: and

text: the

text: C.M.S

text: of

text: the

text: bound

text: state,

text: the

text: potential

text: reads

text: V

text: V

text: γ

text: s

text: v

text: λ

text: λ

text: α

text: V

text: V

text: V

text: s

text: s

text: α

text: π2

text: v

text: where

text: the

text: running

text: coupling

text: constant

text: α

text: is

text: s

text: α

text: s

text: log(a

text: f

text: Λ2

text: QCD

text: We

text: introduce

text: a

text: small

text: parameter

text: a

text: to

text: avoid

text: the

text: divergence

text: in

text: the

text: denominator.

text: The

text: constants

text: λ,

text: α,

text: V

text: and

text: Λ

text: are

text: the

text: parameters

text: that

text: characterize

text: the

text: potential.

text: N

text: for

text: bq

text: (and

text: QCD

text: f

text: c¯q)

text: system.

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##