text: L1 SCHEME FOR TIME-FRACTIONAL DIFFUSION 19 Table 2. The L2-norm of the error for the subdiﬀusion equation with − initial data (a) and (b) at t = 0.1 with h = 2 13, τ = 1/N. α N 10 20 40 80 160 320 rate ≈ 0.1 (a) 1.46e-4 7.18e-5 3.55e-5 1.77e-5 8.82e-6 4.40e-6 1.01 (1.00) ≈ (b) 3.95e-4 1.93e-4 9.57e-5 4.76e-5 2.38e-5 1.19e-5 1.00 (1.00) ≈ 0.5 (a) 1.22e-3 5.89e-4 2.88e-4 1.43e-4 7.08e-5 3.52e-5 1.01 (1.00) ≈ (b) 3.65e-3 1.73e-3 8.36e-4 4.09e-4 2.02e-4 1.00e-4 1.02 (1.00) ≈ 0.9 (a) 7.01e-3 3.05e-3 1.39e-3 6.53e-4 3.12e-4 1.50e-4 1.07 (1.00) ≈ (b) 1.54e-2 7.67e-3 3.79e-3 1.87e-3 9.23e-4 4.55e-4 1.02 (1.00) Table 3. The L2-norm of the error for the subdiﬀusion equation with − initial data (a) and (b) with α = 0.5, h = 2 13 and N = 10. t 1e-5 1e-6 1e-7 1e-8 1e-9 1e-10 rate ≈ (a) 2.94e-3 1.05e-3 3.45e-4 1.11e-4 3.51e-5 1.11e-5 0.50 (0.50) ≈ (b) 3.02e-3 2.56e-3 2.18e-3 1.86e-3 1.58e-3 1.35e-3 0.07 (0.06) Now we present numerical results for the space time fractional problem. Since the exact solution is not available in closed form, we compute the reference solution by the second-order backward diﬀerence scheme from [11] on a much ﬁner mesh, i.e., N = 1000. The numerical results for case (a) with diﬀerent α and β values are presented in Table 4. A ﬁrst-order convergence is observed, and the convergence rate is independent of the time- and space-fractional orders. Interestingly, the observation is valid also for the case β = 5/4, for which the theory in Section does not cover, awaiting further study. For a ﬁxed α value, the error decreases with the increase of the fractional order β, which indicates that the solution decays faster as β approaches two. This is also consistent with the fact that the closer is the β value to unity, the more singular is the solution, and thus more challenging to approximate numerically, cf. [8]. For the nonsmooth case (b), we are particularly interested in the case of small t. Thus we present the numerical results for t = 0.1, t = 0.01 and t = 0.001 in Table 5. The experiment shows the ﬁrst order convergence is robust for nonsmooth data even if t is close to zero. Like before, for → ﬁxed n, let t 0, the error behaves like tα for case (a), which is fully conﬁrmed by the n n numerical results, cf. Table 6, indicating the sharpness of the estimate in Theorem 4.1. Table 4. The L2-norm of the error for the time-space fractional prob- − → lem with initial data (a) at t = 0.1 with h = 2 13, τ 0. \ α β N 5 10 20 40 80 rate ≈ 1.25 1.37e-3 6.55e-4 3.21e-4 1.59e-4 7.90e-5 1.01 (1.00) ≈ 0.1 1.5 8.41e-4 4.03e-4 1.98e-4 9.78e-5 4.87e-5 1.01 (1.00) ≈ 1.75 5.08e-4 2.44e-4 1.19e-4 5.92e-5 2.94e-5 1.01 (1.00) ≈ 1.25 1.52e-2 6.69e-3 3.12e-3 1.50e-3 7.32e-4 1.03 (1.00) ≈ 0.5 1.5 8.22e-3 3.70e-3 1.75e-3 8.49e-4 4.17e-4 1.05 (1.00) ≈ 1.75 4.69e-3 2.14e-3 1.02e-3 4.97e-4 2.45e-4 1.03 (1.00) ≈ 1.25 6.01e-2 3.19e-2 1.62e-2 8.07e-3 3.99e-4 1.01 (1.00) ≈ 0.9 1.5 5.61e-2 2.86e-2 1.42e-2 6.95e-3 3.39e-3 1.03 (1.00) ≈ 1.75 3.58e-2 1.66e-2 7.74e-3 3.66e-3 1.75e-3 1.07 (1.00) 6. Conclusions ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: SCHEME

text: FOR

text: TIME-FRACTIONAL

text: DIFFUSION

text: Table

text: The

label: caption

text: of

text: the

text: error

text: for

text: the

text: subdiﬀusion

text: equation

text: with

text: initial

text: data

text: (a)

text: and

text: (b)

text: at

text: t

text: with

text: h

text: τ

text: α

label: table

text: N

text: rate

text: (a)

text: (b)

text: (a)

text: (b)

text: (a)

text: (b)

text: Table

text: The

text: of

text: the

text: error

text: for

text: the

text: subdiﬀusion

text: equation

text: with

text: initial

text: data

text: (a)

text: and

text: (b)

text: with

text: α

text: h

text: and

text: N

text: t

text: rate

text: (a)

text: (b)

text: Now

text: we

text: present

text: numerical

text: results

text: for

text: the

text: space

text: time

text: fractional

text: problem.

text: Since

text: the

text: exact

text: solution

text: is

text: not

text: available

text: in

text: closed

text: form,

text: we

text: compute

text: the

text: reference

text: solution

text: by

text: the

text: second-order

text: backward

text: diﬀerence

text: scheme

text: from

text: on

text: a

text: much

text: ﬁner

text: mesh,

text: i.e.,

text: N

text: The

text: numerical

text: results

text: for

text: case

text: (a)

text: with

text: diﬀerent

text: α

text: and

text: β

text: values

text: are

text: presented

text: in

text: Table

text: A

text: ﬁrst-order

text: convergence

text: is

text: observed,

text: and

text: the

text: convergence

text: rate

text: is

text: independent

text: of

text: the

text: time-

text: and

text: space-fractional

text: orders.

text: Interestingly,

text: the

text: observation

text: is

text: valid

text: also

text: for

text: the

text: case

text: β

text: for

text: which

text: the

text: theory

text: in

text: Section

text: does

text: not

text: cover,

text: awaiting

text: further

text: study.

text: For

text: a

text: ﬁxed

text: α

text: value,

text: the

text: error

text: decreases

text: with

text: the

text: increase

text: of

text: the

text: fractional

text: order

text: β,

text: which

text: indicates

text: that

text: the

text: solution

text: decays

text: faster

text: as

text: β

text: approaches

text: two.

text: This

text: is

text: also

text: consistent

text: with

text: the

text: fact

text: that

text: the

text: closer

text: is

text: the

text: β

text: value

text: to

text: unity,

text: the

text: more

text: singular

text: is

text: the

text: solution,

text: and

text: thus

text: more

text: challenging

text: to

text: approximate

text: numerically,

text: cf.

text: For

text: the

text: nonsmooth

text: case

text: (b),

text: we

text: are

text: particularly

text: interested

text: in

text: the

text: case

text: of

text: small

text: t.

text: Thus

text: we

text: present

text: the

text: numerical

text: results

text: for

text: t

text: t

text: and

text: t

text: in

text: Table

text: The

text: experiment

text: shows

text: the

text: ﬁrst

text: order

text: convergence

text: is

text: robust

text: for

text: nonsmooth

text: data

text: even

text: if

text: t

text: is

text: close

text: to

text: zero.

text: Like

text: before,

text: for

text: ﬁxed

text: n,

text: let

text: t

text: the

text: error

text: behaves

text: like

text: tα

text: for

text: case

text: (a),

text: which

text: is

text: fully

text: conﬁrmed

text: by

text: the

text: n

text: n

text: numerical

text: results,

text: cf.

text: Table

text: indicating

text: the

text: sharpness

text: of

text: the

text: estimate

text: in

text: Theorem

text: Table

text: The

text: of

text: the

text: error

text: for

text: the

text: time-space

text: fractional

text: prob-

text: lem

text: with

text: initial

text: data

text: (a)

text: at

text: t

text: with

text: h

text: τ

text: α

text: β

text: N

text: rate

text: Conclusions

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: elpmaxe rep ssoL )%( egatnecrep elpmaS elpmaxe rep ssoL 40 35 30 25 20 15 10 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 iteration # Figure 5: Percentage of samples per iteration. to achieve convergence in each iteration. Figure 5 depicts the sampling ratio when an iteration is halted. While the sampling ratio is well below 5% in the ﬁrst iterations, it increases drastically once we are getting close to the min- imum. This happens because more data have to be seen for the estimators to converge. If a ﬁxed-size sub-sample is taken and model calibration is executed on the sub-sample, two events can occur. If the sample size is too large, unnecessary processing is executed. For a smaller sample, convergence to the true minimum cannot be achieved, no matter how long the process is executed. Since online aggregation chooses the size of the sample adaptively, it avoids these problems altogether. mini-batch OLA lock # batch sizes mini-batch OLA lock # step sizes & # batch sizes 1 1 1 2 4 8 16 1 step & 1 batch 2 steps & 2 batches 0.8 0.8 3 steps & 3 batches 0.6 0.6 4 steps & 4 batches 0.4 0.4 0.2 0.2 0 0 0 500 1000 1500 2000 0 500 1000 1500 2000 Time (seconds) Time (seconds) (a) (b) Figure 6: Convergence as a function of batch and step size for mini-batch OLA on classify50M. (a) LR with ﬁxed step size and different batch sizes. (b) LR with different step sizes and different batch sizes. 7.4 Two-Parameter Experiments We examine how the proposed techniques apply to a scenario where two parameters have to be calibrated. Specif- ically, we consider mini-batch gradient descent (Section ) as a variant of IGD in which the gradient is approximated using a variable number of examples, rather than a single example. The number of examples, i.e., the batch size, is the second parameter – together with the step size – to tune. Since these two parameters are correlated – the larger 22 ##LTFigure## ##LTFigure## ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: elpmaxe

label: paragraph

font: default

text: rep

text: ssoL

text: egatnecrep

text: elpmaS

text: elpmaxe

text: rep

text: ssoL

text: iteration

text: Figure

text: Percentage

text: of

text: samples

text: per

text: iteration.

text: to

text: achieve

text: convergence

text: in

text: each

text: iteration.

text: Figure

text: depicts

text: the

text: sampling

text: ratio

text: when

text: an

text: iteration

text: is

text: halted.

text: While

text: the

text: sampling

text: ratio

text: is

text: well

text: below

text: in

text: the

text: ﬁrst

text: iterations,

text: it

text: increases

text: drastically

text: once

text: we

text: are

text: getting

text: close

text: to

text: the

text: min-

text: imum.

text: This

text: happens

text: because

text: more

text: data

text: have

text: to

text: be

text: seen

text: for

text: the

text: estimators

text: to

text: converge.

text: If

text: a

text: ﬁxed-size

text: sub-sample

text: is

text: taken

text: and

text: model

text: calibration

text: is

text: executed

text: on

text: the

text: sub-sample,

text: two

text: events

text: can

text: occur.

text: If

text: the

text: sample

text: size

text: is

text: too

text: large,

text: unnecessary

text: processing

text: is

text: executed.

text: For

text: a

text: smaller

text: sample,

text: convergence

text: to

text: the

text: true

text: minimum

text: cannot

text: be

text: achieved,

text: no

text: matter

text: how

text: long

text: the

text: process

text: is

text: executed.

text: Since

text: online

text: aggregation

text: chooses

text: the

text: size

text: of

text: the

text: sample

text: adaptively,

text: it

text: avoids

text: these

text: problems

text: altogether.

text: mini-batch

text: OLA

text: lock

text: batch

text: sizes

text: mini-batch

text: OLA

text: lock

text: step

text: sizes

text: batch

text: sizes

text: step

text: batch

text: steps

text: batches

text: steps

text: batches

text: steps

text: batches

text: Time

text: (seconds)

text: Time

text: (seconds)

text: (a)

text: (b)

text: Figure

text: Convergence

text: as

text: a

text: function

text: of

text: batch

text: and

text: step

text: size

text: for

text: mini-batch

text: OLA

text: on

text: (a)

text: LR

text: with

text: ﬁxed

text: step

text: size

text: and

text: different

text: batch

text: sizes.

text: (b)

text: LR

text: with

text: different

text: step

text: sizes

text: and

text: different

text: batch

text: sizes.

text: Two-Parameter

text: Experiments

text: We

text: examine

text: how

text: the

text: proposed

text: techniques

text: apply

text: to

text: a

text: scenario

text: where

text: two

text: parameters

text: have

text: to

text: be

text: calibrated.

text: Specif-

text: ically,

text: we

text: consider

text: mini-batch

text: gradient

text: descent

text: (Section

text: as

text: a

text: variant

text: of

text: IGD

text: in

text: which

text: the

text: gradient

text: is

text: approximated

text: using

text: a

text: variable

text: number

text: of

text: examples,

text: rather

text: than

text: a

text: single

text: example.

text: The

text: number

text: of

text: examples,

text: i.e.,

text: the

text: batch

text: size,

text: is

text: the

text: second

text: parameter

text: together

text: with

text: the

text: step

text: size

text: to

text: tune.

text: Since

text: these

text: two

text: parameters

text: are

text: correlated

text: the

text: larger

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: an iteration. To the best of our knowledge, other types of parallelism [16], e.g., pipelining, super-scalar execution, and vectorization, are not considered in the literature. 4.1 Distributed BGD The straightforward strategy to parallelize BGD is to overlap gradient computation (line 5 in Algorithm 1) across the processing nodes storing the M data subsets [9]. The partial gradients are subsequently aggregated at a coordinator node holding the current model w(cid:126)(k), where a single global update step is performed in order to generate the new model w(cid:126)(k+1) based on Eq. (2). The update step requires completion of partial gradient computation across all the nodes, i.e., update model is a synchronization barrier. Once the new model w(cid:126)(k+1) is computed – using line search (cid:104) (cid:105) or a ﬁxed step size – it is(cid:80)disseminated to the pro(cid:80)cessing nodes for a new iteration over the data. Parallel BGD works because the objective function Λ is linearly separable, i.e., the gradient of a sum is the sum of the gradient ∇ ∇ applied to each term: N f (w(cid:126), x(cid:126) ; y ) = N f (w(cid:126), x(cid:126) ; y ). It provides linear processing speedup and i=1 i i i=1 i i logarithmic communication in the number of nodes since aggregation can be executed on a tree structure. In terms of convergence though, there is no improvement with respect to the sequential algorithm—only faster iterations for gradient computation. An alternative parallel algorithm is given in [22]. Each node solves to convergence an independent BGD on its corresponding data subset. The joint model is produced by averaging the partial models computed at each node. As a result, communication between nodes is reduced to minimum. From a convergence standpoint, this ensemble solution provides only variance reduction, but does not eliminate the bias when compared to the single node sub- sample solution [55]. The execution time is dominated by the convergence time of the slowest node. 4.2 Distributed IGD Intuitively, parallelizing IGD cannot be any different from the batch version since the difference between the two algorithms is minimal. It should be even simpler since no global gradient computation is required. Essentially, a model update is executed for every data example based solely on its value. At a closer look though, we observe that the updates in Algorithm 2 are strictly sequential, i.e., the input of the current update is the output of the previous one. Moreover, the order in which updates are executed affects the ﬁnal result, i.e., model update is not commutative. These reasons seem to preclude any type of parallelism altogether. Unless we drop the requirement that all the data examples have to be used to update the model in one iteration. Similar to the approach in [22], the standard method to parallelize IGD [55] is to solve a separate model for every data partition and then to average the M models in order to generate the ﬁnal model at each iteration. The resulting model is then passed as a starting point to the subsequent iteration and the process is repeated until convergence, which is theoretically achieved in a logarithmic number of iterations [55]. An alternative that minimizes communication is to execute IGD until convergence on each partition and average only the ﬁnal models. Model averaging. The following tradeoff has to be considered in the case of parallel IGD. When the number of data partitions M is large, so is the degree of parallelism. This results in higher speedup. It also results in more models computed over fewer examples to be averaged. This is a potential problem since there is a lower bound on the number of examples required for a given model to achieve the asymptotic regime [55]. Moreover, the examples seen by every model have to represent a random sample from the entire population. The larger the number of partitions M, the higher the communication required to generate random samples. The combined effect of non-asymptotic behavior and non- random examples is higher variance across the partial models, which ﬁnally results in slower convergence [19, 38]. Thus, choosing the optimal M value has deeper implications in the case of parallel IGD. Shared-memory parallelization. Exactly because of this reason, the preferred solution to parallelize IGD in a shared-memory setting is to discard model merging completely. A single model is shared across all the threads, while concurrent updates to the model are serialized through locking. Unfortunately, this solution reduces speedup dramatically even when light locking mechanisms, e.g., compare&swap, are used. The solution proposed in [35] to solve this problem is to discard locking altogether and allow for contentious updates to the shared model. As expected, this results in linear speedup. What about convergence? As long as the model is highly-dimensional and the gradients used in updates are sparse, i.e., only a small number of model dimensions are updated, the difference in convergence compared to the serial solution is minimal. 7

text: an

text: iteration.

text: To

text: the

text: best

text: of

text: our

text: knowledge,

text: other

text: types

text: of

text: parallelism

text: e.g.,

text: pipelining,

text: super-scalar

text: execution,

text: and

text: vectorization,

text: are

text: not

text: considered

text: in

text: the

text: literature.

label: section

text: Distributed

text: BGD

text: The

text: straightforward

text: strategy

text: to

text: parallelize

text: BGD

text: is

text: to

text: overlap

text: gradient

text: computation

text: (line

text: in

text: Algorithm

text: across

text: the

text: processing

text: nodes

text: storing

text: the

text: M

text: data

text: subsets

text: The

text: partial

text: gradients

text: are

text: subsequently

text: aggregated

text: at

text: a

text: coordinator

text: node

text: holding

text: the

text: current

text: model

text: where

text: a

text: single

text: global

text: update

text: step

text: is

text: performed

text: in

text: order

text: to

text: generate

text: the

text: new

text: model

text: based

text: on

text: Eq.

text: The

text: update

text: step

text: requires

text: completion

text: of

text: partial

text: gradient

text: computation

text: across

text: all

text: the

text: nodes,

text: i.e.,

text: update

text: model

text: is

text: a

text: synchronization

text: barrier.

text: Once

text: the

text: new

text: model

text: is

text: computed

text: using

text: line

text: search

text: or

text: a

text: ﬁxed

text: step

text: size

text: it

text: to

text: the

text: nodes

text: for

text: a

text: new

text: iteration

text: over

text: the

text: data.

text: Parallel

text: BGD

text: works

text: because

text: the

text: objective

text: function

text: Λ

text: is

text: linearly

text: separable,

text: i.e.,

text: the

text: gradient

text: of

text: a

text: sum

text: is

text: the

text: sum

text: of

text: the

text: gradient

text: applied

text: to

text: each

text: term:

text: N

text: f

text: y

text: N

text: f

text: y

text: It

text: provides

text: linear

text: processing

text: speedup

text: and

text: i

text: i

text: i

text: i

text: logarithmic

text: communication

text: in

text: the

text: number

text: of

text: nodes

text: since

text: aggregation

text: can

text: be

text: executed

text: on

text: a

text: tree

text: structure.

text: In

text: terms

text: of

text: convergence

text: though,

text: there

text: is

text: no

text: improvement

text: with

text: respect

text: to

text: the

text: sequential

text: algorithm—only

text: faster

text: iterations

text: for

text: gradient

text: computation.

text: An

text: alternative

text: parallel

text: algorithm

text: is

text: given

text: in

text: Each

text: node

text: solves

text: to

text: convergence

text: an

text: independent

text: BGD

text: on

text: its

text: corresponding

text: data

text: subset.

text: The

text: joint

text: model

text: is

text: produced

text: by

text: averaging

text: the

text: partial

text: models

text: computed

text: at

text: each

text: node.

text: As

text: a

text: result,

text: communication

text: between

text: nodes

text: is

text: reduced

text: to

text: minimum.

text: From

text: a

text: convergence

text: standpoint,

text: this

text: ensemble

text: solution

text: provides

text: only

text: variance

text: reduction,

text: but

text: does

text: not

text: eliminate

text: the

text: bias

text: when

text: compared

text: to

text: the

text: single

text: node

text: sub-

text: sample

text: solution

text: The

text: execution

text: time

text: is

text: dominated

text: by

text: the

text: convergence

text: time

text: of

text: the

text: slowest

text: node.

text: Distributed

text: IGD

text: Intuitively,

text: parallelizing

text: IGD

text: cannot

text: be

text: any

text: different

text: from

text: the

text: batch

text: version

text: since

text: the

text: difference

text: between

text: the

text: two

text: algorithms

text: is

text: minimal.

text: It

text: should

text: be

text: even

text: simpler

text: since

text: no

text: global

text: gradient

text: computation

text: is

text: required.

text: Essentially,

text: a

text: model

text: update

text: is

text: executed

text: for

text: every

text: data

text: example

text: based

text: solely

text: on

text: its

text: value.

text: At

text: a

text: closer

text: look

text: though,

text: we

text: observe

text: that

text: the

text: updates

text: in

text: Algorithm

text: are

text: strictly

text: sequential,

text: i.e.,

text: the

text: input

text: of

text: the

text: current

text: update

text: is

text: the

text: output

text: of

text: the

text: previous

text: one.

text: Moreover,

text: the

text: order

text: in

text: which

text: updates

text: are

text: executed

text: affects

text: the

text: ﬁnal

text: result,

text: i.e.,

text: model

text: update

text: is

text: not

text: commutative.

text: These

text: reasons

text: seem

text: to

text: preclude

text: any

text: type

text: of

text: parallelism

text: altogether.

text: Unless

text: we

text: drop

text: the

text: requirement

text: that

text: all

text: the

text: data

text: examples

text: have

text: to

text: be

text: used

text: to

text: update

text: the

text: model

text: in

text: one

text: iteration.

text: Similar

text: to

text: the

text: approach

text: in

text: the

text: standard

text: method

text: to

text: parallelize

text: IGD

text: is

text: to

text: solve

text: a

text: separate

text: model

text: for

text: every

text: data

text: partition

text: and

text: then

text: to

text: average

text: the

text: M

text: models

text: in

text: order

text: to

text: generate

text: the

text: ﬁnal

text: model

text: at

text: each

text: iteration.

text: The

text: resulting

text: model

text: is

text: then

text: passed

text: as

text: a

text: starting

text: point

text: to

text: the

text: subsequent

text: iteration

text: and

text: the

text: process

text: is

text: repeated

text: until

text: convergence,

text: which

text: is

text: theoretically

text: achieved

text: in

text: a

text: logarithmic

text: number

text: of

text: iterations

text: An

text: alternative

text: that

text: minimizes

text: communication

text: is

text: to

text: execute

text: IGD

text: until

text: convergence

text: on

text: each

text: partition

text: and

text: average

text: only

text: the

text: ﬁnal

text: models.

text: Model

text: averaging.

text: The

text: following

text: tradeoff

text: has

text: to

text: be

text: considered

text: in

text: the

text: case

text: of

text: parallel

text: IGD.

text: When

text: the

text: number

text: of

text: data

text: partitions

text: M

text: is

text: large,

text: so

text: is

text: the

text: degree

text: of

text: parallelism.

text: This

text: results

text: in

text: higher

text: speedup.

text: It

text: also

text: results

text: in

text: more

text: models

text: computed

text: over

text: fewer

text: examples

text: to

text: be

text: averaged.

text: This

text: is

text: a

text: potential

text: problem

text: since

text: there

text: is

text: a

text: lower

text: bound

text: on

text: the

text: number

text: of

text: examples

text: required

text: for

text: a

text: given

text: model

text: to

text: achieve

text: the

text: asymptotic

text: regime

text: Moreover,

text: the

text: examples

text: seen

text: by

text: every

text: model

text: have

text: to

text: represent

text: a

text: random

text: sample

text: from

text: the

text: entire

text: population.

text: The

text: larger

text: the

text: number

text: of

text: partitions

text: M,

text: the

text: higher

text: the

text: communication

text: required

text: to

text: generate

text: random

text: samples.

text: The

text: combined

text: effect

text: of

text: non-asymptotic

text: behavior

text: and

text: non-

text: random

text: examples

text: is

text: higher

text: variance

text: across

text: the

text: partial

text: models,

text: which

text: ﬁnally

text: results

text: in

text: slower

text: convergence

text: Thus,

text: choosing

text: the

text: optimal

text: M

text: value

text: has

text: deeper

text: implications

text: in

text: the

text: case

text: of

text: parallel

text: IGD.

text: Shared-memory

text: parallelization.

text: Exactly

text: because

text: of

text: this

text: reason,

text: the

text: preferred

text: solution

text: to

text: parallelize

text: IGD

text: in

text: a

text: shared-memory

text: setting

text: is

text: to

text: discard

text: model

text: merging

text: completely.

text: A

text: single

text: model

text: is

text: shared

text: across

text: all

text: the

text: threads,

text: while

text: concurrent

text: updates

text: to

text: the

text: model

text: are

text: serialized

text: through

text: locking.

text: Unfortunately,

text: this

text: solution

text: reduces

text: speedup

text: dramatically

text: even

text: when

text: light

text: locking

text: mechanisms,

text: e.g.,

text: compare&swap,

text: are

text: used.

text: The

text: solution

text: proposed

text: in

text: to

text: solve

text: this

text: problem

text: is

text: to

text: discard

text: locking

text: altogether

text: and

text: allow

text: for

text: contentious

text: updates

text: to

text: the

text: shared

text: model.

text: As

text: expected,

text: this

text: results

text: in

text: linear

text: speedup.

text: What

text: about

text: convergence?

text: As

text: long

text: as

text: the

text: model

text: is

text: highly-dimensional

text: and

text: the

text: gradients

text: used

text: in

text: updates

text: are

text: sparse,

text: i.e.,

text: only

text: a

text: small

text: number

text: of

text: model

text: dimensions

text: are

text: updated,

text: the

text: difference

text: in

text: convergence

text: compared

text: to

text: the

text: serial

text: solution

text: is

text: minimal.


text: Hybrid solution. In a distributed setting with multi-core nodes, a hybrid approach is regarded as the optimal solu- tion [39]. A single model is created for each node. It is shared across all the local threads or processes and updated using one of the shared-memory strategies, chosen based on the actual model properties. Model merging across nodes is executed using the standard averaging technique along an aggregation tree. The hybrid solution aims to maximize the degree of parallelism available in the system without dramatically impacting the convergence. 4.3 Summary Parallelizing gradient descent methods in a distributed environment poses different challenges. For the batch method, gradient computation is trivially parallelizable due to the objective function being linearly separable. The main difﬁculty is posed by model update which is a strictly sequential operation. The main issue when parallelizing IGD is the tradeoff between the degree of parallelism and the convergence rate. More parallelism does not automatically result in better convergence. A hybrid approach in which a shared model is created for each node, while averaging is used to merge models across nodes, is likely to be the optimal solution in general. 5 Speculative Iterations In this section, we address two fundamental problems speciﬁc to model calibration—convergence detection and parameter tuning. As with any iterative method, gradient descent convergence is achieved when there is no more decrease in the objective function, i.e., the loss, across consecutive iterations. While it is obvious that convergence detection requires loss evaluation at every iteration, the standard practice, e.g., Vowpal Wabbit [2], MLLib [17], is to discard detection altogether and execute the algorithm for a ﬁxed number of iterations. The reason is simple: loss computation requires a complete pass over the data, which doubles the execution time. This approach suffers from at least two problems. First, it is impossible to detect convergence before the speciﬁed number of iterations ﬁnishes. And second, it is impossible to identify bad parameter conﬁgurations, i.e., conﬁgurations that do not lead to model convergence. Recall that both BGD and IGD depend on a series of parameters, the most important of which is the step size. Finding the optimal step size typically requires many trials. Discarding loss computation increases both the number of trials as well as the duration of each trial. High-level approach. We propose a uniﬁed solution for convergence detection and parameter tuning based on spec- ulative processing. The main idea is to overlap gradient and loss computation for multiple parameter conﬁgurations across every data traversal. This allows for timely convergence detection and early bad conﬁguration identiﬁcation since many trials are executed simultaneously. Overall, faster model training. The intuition behind our approach is that modern CPU architectures provide extensive parallelization opportunities, e.g., multi-core, hyper-threading, vec- torization, that are difﬁcult to use at full potential in disk-based workloads speciﬁc to Big Data analytics over massive datasets. With the right mix of tasks and judicious scheduling, the degree of parallelism supported in hardware can be fully utilized by executing multiple tasks concurrently. Moreover, the overall execution time is similar to the time it takes to execute each task separately. Our contribution is to design speculative gradient descent algorithms that test multiple step sizes simultaneously and overlap gradient and loss computation. The number of step sizes used at each iteration is determined adaptively and dynamically at runtime. The step sizes are drawn from a parametric distribution that is continuously updated using a Bayesian statistics [23] approach. Only the model with the minimum loss survives each iteration, while the others are discarded (Figure 1). As long as the execution time does not dramatically increase when multiple step sizes are tested, speculative execution leads to faster model training. To the best of our knowledge, this is the ﬁrst solution that uses speculative execution for gradient descent parameter tuning and loss computation. Speculative execution is applied in Vowpal Wabbit [2] and MLlib [17], but only to deal with the problem of slow nodes. Not to test multiple step sizes simultaneously. Vowpal Wabbit also supports progressive loss estimation which overlaps gradient and loss computation. Progressive loss provides only local estimates. It never computes the exact loss at the end of an iteration. This can be done only with an additional pass over the data. Our solution is exact – not approximate – and works in a distributed setting. 5.1 Speculative BGD 8

text: Hybrid

text: solution.

text: In

text: a

text: distributed

text: setting

text: with

text: multi-core

text: nodes,

text: a

text: hybrid

text: approach

text: is

text: regarded

text: as

text: the

text: optimal

text: solu-

text: tion

text: A

text: single

text: model

text: is

text: created

text: for

text: each

text: node.

text: It

text: is

text: shared

text: across

text: all

text: the

text: local

text: threads

text: or

text: processes

text: and

text: updated

text: using

text: one

text: of

text: the

text: shared-memory

text: strategies,

text: chosen

text: based

text: on

text: the

text: actual

text: model

text: properties.

text: Model

text: merging

text: across

text: nodes

text: is

text: executed

text: using

text: the

text: standard

text: averaging

text: technique

text: along

text: an

text: aggregation

text: tree.

text: The

text: hybrid

text: solution

text: aims

text: to

text: maximize

text: the

text: degree

text: of

text: parallelism

text: available

text: in

text: the

text: system

text: without

text: dramatically

text: impacting

text: the

text: convergence.

text: Summary

text: Parallelizing

text: gradient

text: descent

text: methods

text: in

text: a

text: distributed

text: environment

text: poses

text: different

text: challenges.

text: For

text: the

text: batch

text: method,

text: gradient

text: computation

text: is

text: trivially

text: parallelizable

text: due

text: to

text: the

text: objective

text: function

text: being

text: linearly

text: separable.

text: The

text: main

text: difﬁculty

text: is

text: posed

text: by

text: model

text: update

text: which

text: is

text: a

text: strictly

text: sequential

text: operation.

text: The

text: main

text: issue

text: when

text: parallelizing

text: IGD

text: is

text: the

text: tradeoff

text: between

text: the

text: degree

text: of

text: parallelism

text: and

text: the

text: convergence

text: rate.

text: More

text: parallelism

text: does

text: not

text: automatically

text: result

text: in

text: better

text: convergence.

text: A

text: hybrid

text: approach

text: in

text: which

text: a

text: shared

text: model

text: is

text: created

text: for

text: each

text: node,

text: while

text: averaging

text: is

text: used

text: to

text: merge

text: models

text: across

text: nodes,

text: is

text: likely

text: to

text: be

text: the

text: optimal

text: solution

text: in

text: general.

text: Speculative

text: Iterations

text: In

text: this

text: section,

text: we

text: address

text: two

text: fundamental

text: problems

text: speciﬁc

text: to

text: model

text: calibration—convergence

text: detection

text: and

text: parameter

text: tuning.

text: As

text: with

text: any

text: iterative

text: method,

text: gradient

text: descent

text: convergence

text: is

text: achieved

text: when

text: there

text: is

text: no

text: more

text: decrease

text: in

text: the

text: objective

text: function,

text: i.e.,

text: the

text: loss,

text: across

text: consecutive

text: iterations.

text: While

text: it

text: is

text: obvious

text: that

text: convergence

text: detection

text: requires

text: loss

text: evaluation

text: at

text: every

text: iteration,

text: the

text: standard

text: practice,

text: e.g.,

text: Vowpal

text: Wabbit

text: MLLib

text: is

text: to

text: discard

text: detection

text: altogether

text: and

text: execute

text: the

text: algorithm

text: for

text: a

text: ﬁxed

text: number

text: of

text: iterations.

text: The

text: reason

text: is

text: simple:

text: loss

text: computation

text: requires

text: a

text: complete

text: pass

text: over

text: the

text: data,

text: which

text: doubles

text: the

text: execution

text: time.

text: This

text: approach

text: suffers

text: from

text: at

text: least

text: two

text: problems.

text: First,

text: it

text: is

text: impossible

text: to

text: detect

text: convergence

text: before

text: the

text: speciﬁed

text: number

text: of

text: iterations

text: ﬁnishes.

text: And

text: second,

text: it

text: is

text: impossible

text: to

text: identify

text: bad

text: parameter

text: conﬁgurations,

text: i.e.,

text: conﬁgurations

text: that

text: do

text: not

text: lead

text: to

text: model

text: convergence.

text: Recall

text: that

text: both

text: BGD

text: and

text: IGD

text: depend

text: on

text: a

text: series

text: of

text: parameters,

text: the

text: most

text: important

text: of

text: which

text: is

text: the

text: step

text: size.

text: Finding

text: the

text: optimal

text: step

text: size

text: typically

text: requires

text: many

text: trials.

text: Discarding

text: loss

text: computation

text: increases

text: both

text: the

text: number

text: of

text: trials

text: as

text: well

text: as

text: the

text: duration

text: of

text: each

text: trial.

text: High-level

text: approach.

text: We

text: propose

text: a

text: uniﬁed

text: solution

text: for

text: convergence

text: detection

text: and

text: parameter

text: tuning

text: based

text: on

text: spec-

text: ulative

text: processing.

text: The

text: main

text: idea

text: is

text: to

text: overlap

text: gradient

text: and

text: loss

text: computation

text: for

text: multiple

text: parameter

text: conﬁgurations

text: across

text: every

text: data

text: traversal.

text: This

text: allows

text: for

text: timely

text: convergence

text: detection

text: and

text: early

text: bad

text: conﬁguration

text: identiﬁcation

text: since

text: many

text: trials

text: are

text: executed

text: simultaneously.

text: Overall,

text: faster

text: model

text: training.

text: The

text: intuition

text: behind

text: our

text: approach

text: is

text: that

text: modern

text: CPU

text: architectures

text: provide

text: extensive

text: parallelization

text: opportunities,

text: e.g.,

text: multi-core,

text: hyper-threading,

text: vec-

text: torization,

text: that

text: are

text: difﬁcult

text: to

text: use

text: at

text: full

text: potential

text: in

text: disk-based

text: workloads

text: speciﬁc

text: to

text: Big

text: Data

text: analytics

text: over

text: massive

text: datasets.

text: With

text: the

text: right

text: mix

text: of

text: tasks

text: and

text: judicious

text: scheduling,

text: the

text: degree

text: of

text: parallelism

text: supported

text: in

text: hardware

text: can

text: be

text: fully

text: utilized

text: by

text: executing

text: multiple

text: tasks

text: concurrently.

text: Moreover,

text: the

text: overall

text: execution

text: time

text: is

text: similar

text: to

text: the

text: time

text: it

text: takes

text: to

text: execute

text: each

text: task

text: separately.

text: Our

text: contribution

text: is

text: to

text: design

text: speculative

text: gradient

text: descent

text: algorithms

text: that

text: test

text: multiple

text: step

text: sizes

text: simultaneously

text: and

text: overlap

text: gradient

text: and

text: loss

text: computation.

text: The

text: number

text: of

text: step

text: sizes

text: used

text: at

text: each

text: iteration

text: is

text: determined

text: adaptively

text: and

text: dynamically

text: at

text: runtime.

text: The

text: step

text: sizes

text: are

text: drawn

text: from

text: a

text: parametric

text: distribution

text: that

text: is

text: continuously

text: updated

text: using

text: a

text: Bayesian

text: statistics

text: approach.

text: Only

text: the

text: model

text: with

text: the

text: minimum

text: loss

text: survives

text: each

text: iteration,

text: while

text: the

text: others

text: are

text: discarded

text: (Figure

text: As

text: long

text: as

text: the

text: execution

text: time

text: does

text: not

text: dramatically

text: increase

text: when

text: multiple

text: step

text: sizes

text: are

text: tested,

text: speculative

text: execution

text: leads

text: to

text: faster

text: model

text: training.

text: To

text: the

text: best

text: of

text: our

text: knowledge,

text: this

text: is

text: the

text: ﬁrst

text: solution

text: that

text: uses

text: speculative

text: execution

text: for

text: gradient

text: descent

text: parameter

text: tuning

text: and

text: loss

text: computation.

text: Speculative

text: execution

text: is

text: applied

text: in

text: Vowpal

text: Wabbit

text: and

text: MLlib

text: but

text: only

text: to

text: deal

text: with

text: the

text: problem

text: of

text: slow

text: nodes.

text: Not

text: to

text: test

text: multiple

text: step

text: sizes

text: simultaneously.

text: Vowpal

text: Wabbit

text: also

text: supports

text: progressive

text: loss

text: estimation

text: which

text: overlaps

text: gradient

text: and

text: loss

text: computation.

text: Progressive

text: loss

text: provides

text: only

text: local

text: estimates.

text: It

text: never

text: computes

text: the

text: exact

text: loss

text: at

text: the

text: end

text: of

text: an

text: iteration.

text: This

text: can

text: be

text: done

text: only

text: with

text: an

text: additional

text: pass

text: over

text: the

text: data.

text: Our

text: solution

text: is

text: exact

text: not

text: approximate

text: and

text: works

text: in

text: a

text: distributed

text: setting.

text: Speculative

text: BGD


text: [26] S. Aoki et al., astro-ph/1202.2529 (2012) [27] A.M. Galper et al. (GAMMA-400), AIPCP, 1516, 288 (2013) [28] A. Furniss et al., MNRAS, 446, 2267 (2015) [29] W. Chen et al., astro-ph/1410.7717 (2014) 6

label: reference

text: S.

text: Aoki

text: et

text: al.,

text: A.M.

text: Galper

text: et

text: al.

text: AIPCP,

text: A.

text: Furniss

text: et

text: al.,

text: MNRAS,

text: W.

text: Chen

text: et

text: al.,


text: { } with y , θ B a sample from the joint distribution of the response and parameters. Intuitively, the “inner l l l=1 sample” of size B˜ is used to approximate the two marginal likelihoods in (4), the ﬁrst marginal to γ and the second to both γ and θ, and the “outer sample” of size B is then used to approximate the expected utility with respect to the joint distribution of y and θ. This approximation is biased for US(δ) due to the bias | | in log π˜(y θ, δ) and log π˜(y δ). However, under regularity conditions satisﬁed by most models of practical − importance (Severini, 2000, pp. 80–81), this bias is of order B˜ 1 (Ryan, 2003) and hence asymptotically negligible. | (cid:80) For NSEL, E(θ y, δ) in (5) can be approximated via importance sampling: w (cid:80) | B˜ θ˜ π(y | θ˜ , γ˜ , δ) E˜(θ y, δ) = b=1 lw b b , (cid:110) (cid:111) w B˜ π(y | θ˜ , γ˜ , δ) b=1 b b B˜ where θ˜ , γ˜ is a random sample from the prior distribution of ψ, and θ˜ is the wth element of θ˜ . b b bw b Hence, the follobw=i1ng nested Monte Carlo appr(cid:88)oxim(cid:88)ati(cid:104)on of the expected(cid:105)utility is obtained: − 1 B p − | 2 U˜V (δ) = θ E˜(θ y , δ) , B lw w l l=1 w=1 where θ is the wth element of θ . Here, the inner sample is used to approximate the posterior expectation, lw l and the outer sample used to approximate the expected utility. Importance sampling has commonly been used to estimate posterior quantities for Bayesian design (see Ryan et al., 2016 and references therein), | although the approximation of the expected utility will again be biased due to bias in E˜(θ y, δ)2. | w In the examples, we set B˜ = B = 1000 for the evaluation of U˜(δ δC ) in Step 1b of the ACE algorithm i (i) (chosen from practical experience). For the comparisons in Steps 1d and 3e, we set B = B˜ = 20, 000. 3.1.2 Evaluating the expected utility via normal approximation The following approximations to US(δ) and UV (δ) are commonly used (Atkinson et al., 2007, ch. 18), justiﬁed via a normal approximation to the posterior(cid:90)distribution of ψ: φS(δ) = E (lo(cid:2)g |I(cid:8)(θ; δ, γ)|) =(cid:9)(cid:3) log |(cid:90)I(θ; δ(cid:8), γ)|π(ψ)dψ,(cid:9) ψ Θ − I − − I − φV (δ) = E tr (θ; δ, γ) 1 = tr (θ; δ, γ) 1 π(ψ)dψ, ψ Θ I with (θ; δ, γ) the Fisher information matrix for θ, or an approximation thereof. Designs that maximise φS and φV are sometimes referred to as pseudo-Bayesian D- and A-optimal designs, respectively. Note that these expressions also result from taking expectations of the utility functions |I | − {I }− uD(ψ, y, δ) = log (θ; δ, γ) , uA(ψ, y, δ) = tr (θ; δ, γ) 1 , which do not depend on y. Unbiased Monte Carlo approximations to φS(δ) and φV (δ) can be obtained via sampling from the prior distrib(cid:88)ution for ψ: (cid:88) (cid:8) (cid:9) 1 B |I | 1 B I − φˆS(δ) = log (θ ; δ, γ ) , φˆV (δ) = tr (θ ; δ, γ ) 1 . B l l B l l l=1 l=1 (cid:110)(cid:104) (cid:105) (cid:111) For comparison of designs, the D-eﬃciency of design δ relative to design δ is deﬁned as 1 2 × − Eﬀ (δ , δ ) = 100 exp φˆS(δ ) φˆS(δ ) /p . (6) D 1 2 1 2 9 ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: with

text: y

text: θ

text: B

text: a

text: sample

text: from

text: the

text: joint

text: distribution

text: of

text: the

text: response

text: and

text: parameters.

text: Intuitively,

text: the

text: “inner

text: l

text: l

text: sample”

text: of

text: size

text: B˜

text: is

text: used

text: to

text: approximate

text: the

text: two

text: marginal

text: likelihoods

text: in

text: the

text: ﬁrst

text: marginal

text: to

text: γ

text: and

text: the

text: second

text: to

text: both

text: γ

text: and

text: θ,

text: and

text: the

text: “outer

text: sample”

text: of

text: size

text: B

text: is

text: then

text: used

text: to

text: approximate

text: the

text: expected

text: utility

text: with

text: respect

text: to

text: the

text: joint

text: distribution

text: of

text: y

text: and

text: θ.

text: This

text: approximation

text: is

text: biased

text: for

text: US(δ)

text: due

text: to

text: the

text: bias

text: in

text: log

text: π˜(y

text: θ,

text: δ)

text: and

text: log

text: π˜(y

text: δ).

text: However,

text: under

text: regularity

text: conditions

text: satisﬁed

text: by

text: most

text: models

text: of

text: practical

text: importance

text: (Severini,

text: pp.

text: this

text: bias

text: is

text: of

text: order

text: B˜

text: (Ryan,

text: and

text: hence

text: asymptotically

text: negligible.

text: For

text: NSEL,

text: E(θ

text: y,

text: δ)

text: in

text: can

text: be

text: approximated

text: via

text: importance

text: sampling:

text: w

text: B˜

text: θ˜

text: π(y

text: θ˜

text: γ˜

text: δ)

text: E˜(θ

text: y,

text: δ)

text: lw

text: b

text: b

text: w

text: B˜

text: π(y

text: θ˜

text: γ˜

text: δ)

text: b

text: b

text: B˜

text: where

text: θ˜

text: γ˜

text: is

text: a

text: random

text: sample

text: from

text: the

text: prior

text: distribution

text: of

text: ψ,

text: and

text: θ˜

text: is

text: the

text: wth

text: element

text: of

text: θ˜

text: b

text: b

text: bw

text: b

text: Hence,

text: the

text: nested

text: Monte

text: Carlo

text: of

text: the

text: is

text: obtained:

text: B

text: p

text: U˜V

text: (δ)

text: θ

text: E˜(θ

text: y

text: δ)

text: B

text: lw

text: w

text: l

text: where

text: θ

text: is

text: the

text: wth

text: element

text: of

text: θ

text: Here,

text: the

text: inner

text: sample

text: is

text: used

text: to

text: approximate

text: the

text: posterior

text: expectation,

text: lw

text: l

text: and

text: the

text: outer

text: sample

text: used

text: to

text: approximate

text: the

text: expected

text: utility.

text: Importance

text: sampling

text: has

text: commonly

text: been

text: used

text: to

text: estimate

text: posterior

text: quantities

text: for

text: Bayesian

text: design

text: (see

text: Ryan

text: et

text: al.,

text: and

text: references

text: therein),

text: although

text: the

text: approximation

text: of

text: the

text: expected

text: utility

text: will

text: again

text: be

text: biased

text: due

text: to

text: bias

text: in

text: E˜(θ

text: y,

text: w

text: In

text: the

text: examples,

text: we

text: set

text: B˜

text: B

text: for

text: the

text: evaluation

text: of

text: U˜(δ

text: δC

text: in

text: Step

text: of

text: the

text: ACE

text: algorithm

text: i

text: (i)

text: (chosen

text: from

text: practical

text: experience).

text: For

text: the

text: comparisons

text: in

text: Steps

text: and

text: we

text: set

text: B

text: B˜

text: Evaluating

text: the

text: expected

text: utility

text: via

text: normal

text: approximation

text: The

text: following

text: approximations

text: to

text: US(δ)

text: and

text: UV

text: (δ)

text: are

text: commonly

text: used

text: (Atkinson

text: et

text: al.,

text: ch.

text: justiﬁed

text: via

text: a

text: normal

text: approximation

text: to

text: the

text: of

text: ψ:

text: φS(δ)

text: E

text: δ,

text: γ)|)

text: log

text: ψ

text: Θ

text: I

text: I

text: φV

text: (δ)

text: E

text: tr

text: (θ;

text: δ,

text: γ)

text: tr

text: (θ;

text: δ,

text: γ)

text: π(ψ)dψ,

text: ψ

text: Θ

text: I

text: with

text: (θ;

text: δ,

text: γ)

text: the

text: Fisher

text: information

text: matrix

text: for

text: θ,

text: or

text: an

text: approximation

text: thereof.

text: Designs

text: that

text: maximise

text: φS

text: and

text: φV

text: are

text: sometimes

text: referred

text: to

text: as

text: pseudo-Bayesian

text: D-

text: and

text: A-optimal

text: designs,

text: respectively.

text: Note

text: that

text: these

text: expressions

text: also

text: result

text: from

text: taking

text: expectations

text: of

text: the

text: utility

text: functions

text: |I

text: {I

text: uD(ψ,

text: y,

text: δ)

text: log

text: (θ;

text: δ,

text: γ)

text: uA(ψ,

text: y,

text: δ)

text: tr

text: (θ;

text: δ,

text: γ)

text: which

text: do

text: not

text: depend

text: on

text: y.

text: Unbiased

text: Monte

text: Carlo

text: approximations

text: to

text: φS(δ)

text: and

text: φV

text: (δ)

text: can

text: be

text: obtained

text: via

text: sampling

text: from

text: the

text: prior

text: for

text: ψ:

text: B

text: |I

text: B

text: I

text: φˆS(δ)

text: log

text: (θ

text: δ,

text: γ

text: φˆV

text: (δ)

text: tr

text: (θ

text: δ,

text: γ

text: B

text: l

text: l

text: B

text: l

text: l

text: For

text: comparison

text: of

text: designs,

text: the

text: D-eﬃciency

text: of

text: design

text: δ

text: relative

text: to

text: design

text: δ

text: is

text: deﬁned

text: as

text: Eﬀ

text: (δ

text: δ

text: exp

text: φˆS(δ

text: φˆS(δ

text: /p

text: D

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: 16 son Lab PAC 39 Proposal, Timelike Compton Scattering J. Wagner, Phys. Rev. D 87, no. 5, 054029 (2013) − and J/Ψ photoproduction on the proton in e+e pair [26] B. Pire, L. Szymanowski and J. Wagner, Phys. Rev. D production with CLAS12 at 11 GeV (2012). 79, 014010 (2009) [23] B. Pire, L. Szymanowski and J. Wagner, Phys. Rev. D [27] A. Accardi, J. L. Albacete, M. Anselmino, N. Armesto, 83, 034009 (2011) E. C. Aschenauer, A. Bacchetta, D. Boer and W. Brooks [24] D. Mueller, B. Pire, L. Szymanowski and J. Wagner, et al., arXiv:1212.1701 [nucl-ex]. Phys. Rev. D 86, 031502 (2012) [25] H. Moutarde, B. Pire, F. Sabatie, L. Szymanowski and

text: son

text: Lab

text: PAC

text: Proposal,

text: Timelike

text: Compton

text: Scattering

text: J.

text: Wagner,

text: Phys.

text: Rev.

text: D

text: no.

text: and

text: J/Ψ

text: photoproduction

text: on

text: the

text: proton

text: in

text: e+e

text: pair

text: B.

text: Pire,

text: L.

text: Szymanowski

text: and

text: J.

text: Wagner,

text: Phys.

text: Rev.

text: D

text: production

text: with

text: at

text: GeV

text: B.

text: Pire,

text: L.

text: Szymanowski

text: and

text: J.

text: Wagner,

text: Phys.

text: Rev.

text: D

text: A.

text: Accardi,

text: J.

text: L.

text: Albacete,

text: M.

text: Anselmino,

text: N.

text: Armesto,

text: E.

text: C.

text: Aschenauer,

text: A.

text: Bacchetta,

text: D.

text: Boer

text: and

text: W.

text: Brooks

text: D.

text: Mueller,

text: B.

text: Pire,

text: L.

text: Szymanowski

text: and

text: J.

text: Wagner,

text: et

text: al.,

text: [nucl-ex].

text: Phys.

text: Rev.

text: D

text: H.

text: Moutarde,

text: B.

text: Pire,

text: F.

text: Sabatie,

text: L.

text: Szymanowski

text: and


text: 3 light-cone frame, introducing the + components of the ∆ and q¯ four-vectors, ξ˜ and ξ˜(cid:48) respectively, the various D four-vectors involved can be decomposed as: D q - q + 2 ~ 2~ q + ~ k ~x ~x x x x (- ' + ) (- ' - ) (x - ' ) m¯ 2 Pµ = p˜µ + nµ, (7) 2 D D q¯µ = −−ξ˜(cid:48) p˜µ − 2q¯ξ˜2(cid:48) nµ, (8) (xk +- ~2x ) k( x+ - 2~x ) ∆µ = 2ξ˜p˜µ + ξ˜m¯ 2nµ + ∆µ⊥, (9) − with m¯ 2 = m2 ∆2 and m is the proton mass. We 4 D D rmeolamteentthuemﬁ:nal photon virtuality to the average photon q~ + 2~ ~ q - 2~ x x x x (- ' - ) (- ' + ) (cid:48) − k - q q¯2 = Q 2/2 ∆2/4. (10) D ~ D x k - (x + ') k + 2~ ~2 (x + x ) (x - x ) We have to relate the light-cone momentum fractions ξ˜ and ξ˜(cid:48) to the kinematical variables which are experi- mentally accessible. To do so, we introduce the variables (cid:48) ξ and ξ : → Figure 2: Leading order diagrams for the process γq γq. We indicate between parenthesis the + components of the four-vectors. − (cid:48) (cid:48) − q¯2 Q 2 + ∆2/2 ξ = = − − (cid:48) , (11) 2P.q¯ 2(s m2) + ∆2 Q 2 (cid:48) − ∆.q¯ Q 2 ξ = = − − (cid:48) . (12) 2P.q¯ 2(s m2) + ∆2 Q 2 The light-cone momentum fractions ξ˜ and ξ˜(cid:48) are re- TTCS ∝ 1 γν (k/ + /q¯) γµ (cid:48) H (k + q¯)2 + i(cid:15) lated to the kinematical variables ξ and ξ by: (cid:20) 1 − + − γµ (k/ /q¯) γν (k q¯)2 + i(cid:15) · 1 + ξ˜(cid:48)2 m¯ 2/q¯2 ∝ 1 − 1 (cid:21) 1 ξ˜ = ξ 1 − ξ˜(cid:48)(cid:112)2 m¯ 2/q¯2 , (13) 2 n/ (p˜µnν + p˜νnµ gµν) . (x − ξ˜(cid:48) + i(cid:15) + x + ξ˜(cid:48) − i(cid:15)) (cid:48) (cid:48) · 2 − 1 − 1 ξ˜ = ξ 1 + 1 − 4ξ(cid:48)2 m¯ 2/q¯2 . (14) n/γ5 i(cid:15)µνκλ nλ p˜κ (x − ξ˜(cid:48) + i(cid:15) x + ξ˜(cid:48) − i(cid:15)) . (16) In the asymptotic limit, where mass and ∆ terms are neglected relatively to Q(cid:48)2, we have: The full TCS amplitude, corresponding to the diagram of Fig. 1-bottom (plus the associated crossed diagram) reads then (cid:48) − (cid:48) − (cid:48) Q 2 ξ˜ = ξ = ξ˜ = ξ = − (cid:48) . (15) 2s Q 2 − e3 (cid:48) TTCS = (cid:48) u¯(k) γν v(k ) (cid:15)µ(q) HTCS, (17) q 2 µν B. Timelike Compton amplitude The two diagrams of Fig. 2 have to be calculated. The − (cid:48) leading order amplitude reads: with, in the Bjorken limit where ξ = ξ˜ , ##LTFigure## ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: light-cone

text: frame,

text: introducing

text: the

text: components

text: of

text: the

text: and

text: q¯

text: four-vectors,

text: ξ˜

text: and

text: respectively,

text: the

text: various

text: D

text: four-vectors

text: involved

text: can

text: be

text: decomposed

text: as:

text: D

text: q

text: q

text: q

text: k

text: ~x

text: ~x

text: x

text: x

text: x

text: (x

text: m¯

label: equation

text: Pµ

text: p˜µ

text: nµ,

text: D

text: D

text: q¯µ

text: p˜µ

text: nµ,

text: (xk

text: k(

text: x+

text: ∆µ

text: 2ξ˜p˜µ

text: ξ˜m¯

text: ∆µ⊥,

text: with

text: m¯

text: ∆2

text: and

text: m

text: is

text: the

text: proton

text: mass.

text: We

text: D

text: D

text: rmeolamteentthuemﬁ:nal

text: photon

text: virtuality

text: to

text: the

text: average

text: photon

text: q~

text: q

text: x

text: x

text: x

text: x

text: k

text: q

text: Q

text: D

text: D

text: x

text: k

text: (x

text: k

text: (x

text: x

text: (x

text: x

text: We

text: have

text: to

text: relate

text: the

text: light-cone

text: momentum

text: fractions

text: ξ˜

text: and

text: to

text: the

text: kinematical

text: variables

text: which

text: are

text: experi-

text: mentally

text: accessible.

text: To

text: do

text: so,

text: we

text: introduce

text: the

text: variables

text: ξ

text: and

text: ξ

text: Figure

text: Leading

text: order

text: diagrams

text: for

text: the

text: process

text: γq

text: γq.

text: We

text: indicate

text: between

text: parenthesis

text: the

text: components

text: of

text: the

text: four-vectors.

text: Q

text: ξ

text: ∆2

text: Q

text: ∆.q¯

text: Q

text: ξ

text: ∆2

text: Q

text: The

text: light-cone

text: momentum

text: fractions

text: ξ˜

text: and

text: are

text: re-

text: TTCS

text: γν

text: (k/

text: /q¯)

text: γµ

text: H

text: (k

text: lated

text: to

text: the

text: kinematical

text: variables

text: ξ

text: and

text: ξ

text: by:

text: γµ

text: (k/

text: /q¯)

text: γν

text: (k

text: m¯

text: ξ˜

text: ξ

text: m¯

text: n/

text: (p˜µnν

text: p˜νnµ

text: gµν)

text: (x

text: x

text: ξ˜

text: ξ

text: m¯

text: nλ

text: p˜κ

text: (x

text: x

text: In

text: the

text: asymptotic

text: limit,

text: where

text: mass

text: and

text: terms

text: are

text: neglected

text: relatively

text: to

text: we

text: have:

text: The

text: full

text: TCS

text: amplitude,

text: corresponding

text: to

text: the

text: diagram

text: of

text: Fig.

text: (plus

text: the

text: associated

text: crossed

text: diagram)

text: reads

text: then

text: Q

text: ξ˜

text: ξ

text: ξ˜

text: ξ

text: Q

text: TTCS

text: u¯(k)

text: γν

text: v(k

text: HTCS,

text: q

text: µν

text: B.

text: Timelike

text: Compton

text: amplitude

text: The

text: two

text: diagrams

text: of

text: Fig.

text: have

text: to

text: be

text: calculated.

text: The

text: leading

text: order

text: amplitude

text: reads:

text: with,

text: in

text: the

text: Bjorken

text: limit

text: where

text: ξ

text: ξ˜

text: ##LTFigure##

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: − F = F 10 0.4m. (3) ν ν,0 We used F = 4 260 Jy for the B ﬁlter, and F = 3 640 Jy for the V ﬁlter. To derive an ν,0 ν,0 × upper limit for the full cocoon size, we scaled our ﬂux to extend over an area of 45 12 arcmin2, making the implicit assumption that the background is roughly similar for the region centred ∼ × − − − on the pulsar and the cocoon region. Our νF upper limits are 9.6 10 8 erg s 1 cm 2 for ∼ × − − − ν the B ﬁlter, and 7.5 10 8 erg s 1 cm 2 for the V ﬁlter. In order to compare with other optical studies, we converted U, B, V, R, and I upper limits found [29] using Hubble Space Telescope (HST), New Technology Telescope (NTT), and Very Large Telescope (VLT) to νF values assuming the optical source size to be the same as that ν of the X-ray cocoon. These upper limits are indicated as red triangles in Figure 1, along with multiwavelength data and models for the broadband spectrum of Vela X [20]. 3. Conclusions The search for the Vela X PWN in optical is crucial to understand its morphology and how energy is distributed throughout the multiwavelength spectrum. It is a challenge to resolve the extended structure of Vela X in the optical waveband, given that many sources contribute to the background radiation, as well as its apparent intrinsic faintness. We only observed a background level of optical emission near the Vela pulsar position, and there is no structure visible that could be associated with a possible Vela X optical counterpart, apart from some hints of brighter emission near the position of the southern radio lobe. However, this may be due to bright stars that were not cleanly removed. Our νF upper limits of ∼ − − − ν 10 7 erg s 1 cm 2 are clearly not constraining for a multiwavelength model of Vela X’s emission. We note that our upper limits suﬀer from several observational factors that introduce uncertainties. Removal of bright or even fainter unresolved point sources remain problematic. Our integration times were also very short. Furthermore, the ﬁeld of view of the telescope is too small to capture the full extension of the source, and we had to use a method of mosaicing several single images to obtain one large image that encapsulates the probable extension of the optical source. Changing conditions between the constituent frames add complexity and uncertainty to our analysis. Neither our optical limits nor those of others prove to be constraining for the spectral model of Vela X. Barring the gamma-ray data for the moment, one may model the radio, optical and X-ray data with either a single or double synchrotron component, implying either a single or multi-component electron spectrum interacting with the PWN B-ﬁeld. However, the Fermi and H.E.S.S. data are consistent with a two-component leptonic interpretation [2], ﬁrst put forward by [10] and subsequently elaborated on by [11]. A second model [20] invokes a single electron ◦ × ◦ population, but then assumes diﬀerent diﬀusive properties for the 2 3 extended radio nebula (cid:48) × (cid:48) (ERN) and 45 12 cocoon regions (although the H.E.S.S. “coccoon” is quite more extensive). The particles are assumed to be contained within the ERN until the reverse shock from the SNR interacts with the ERN, and diﬀusive transport starts to take place. Older particles escape then from the ERN, while newly-injected particles are thought to suﬀer signiﬁcant cooling in the high- B-ﬁeld region near the pulsar before being injected into the low-B-ﬁeld cocoon. Within such a scenario, each substructure (the ERN and cocoon) is responsible for diﬀerent components of the multiwavelength spectrum (radio and GeV vs. X-ray and TeV). In essence, this is a diﬀerent way of constructing a two-component electron injection spectrum. Our optical limits are not violated by these models, although the available limits support the idea of particle escape from the ERN (i.e., a lower synchrotron ﬂux component). Deeper optical observations as well as reﬁned modelling should continue to enhance our understanding of this complex source. References

text: F

text: F

text: ν

text: We

text: used

text: F

text: Jy

text: for

text: the

text: B

text: ﬁlter,

text: and

text: F

text: Jy

text: for

text: the

text: V

text: ﬁlter.

text: To

text: derive

text: an

text: upper

text: limit

text: for

text: the

text: full

text: cocoon

text: size,

text: we

text: scaled

text: our

text: ﬂux

text: to

text: extend

text: over

text: an

text: area

text: of

text: making

text: the

text: implicit

text: assumption

text: that

text: the

text: background

text: is

text: roughly

text: similar

text: for

text: the

text: region

text: centred

text: on

text: the

text: pulsar

text: and

text: the

text: cocoon

text: region.

text: Our

text: νF

text: upper

text: limits

text: are

text: erg

text: s

text: cm

text: for

text: ν

text: the

text: B

text: ﬁlter,

text: and

text: erg

text: s

text: cm

text: for

text: the

text: V

text: ﬁlter.

text: In

text: order

text: to

text: compare

text: with

text: other

text: optical

text: studies,

text: we

text: converted

text: U,

text: B,

text: V,

text: R,

text: and

text: I

text: upper

text: limits

text: found

text: using

text: Hubble

text: Space

text: Telescope

text: (HST),

text: New

text: Technology

text: Telescope

text: (NTT),

text: and

text: Very

text: Large

text: Telescope

text: (VLT)

text: to

text: νF

text: values

text: assuming

text: the

text: optical

text: source

text: size

text: to

text: be

text: the

text: same

text: as

text: that

text: ν

text: of

text: the

text: X-ray

text: cocoon.

text: These

text: upper

text: limits

text: are

text: indicated

text: as

text: red

text: triangles

text: in

text: Figure

text: along

text: with

text: multiwavelength

text: data

text: and

text: models

text: for

text: the

text: broadband

text: spectrum

text: of

text: Vela

text: X

text: Conclusions

text: The

text: search

text: for

text: the

text: Vela

text: X

text: PWN

text: in

text: optical

text: is

text: crucial

text: to

text: understand

text: its

text: morphology

text: and

text: how

text: energy

text: is

text: distributed

text: throughout

text: the

text: multiwavelength

text: spectrum.

text: It

text: is

text: a

text: challenge

text: to

text: resolve

text: the

text: extended

text: structure

text: of

text: Vela

text: X

text: in

text: the

text: optical

text: waveband,

text: given

text: that

text: many

text: sources

text: contribute

text: to

text: the

text: background

text: radiation,

text: as

text: well

text: as

text: its

text: apparent

text: intrinsic

text: faintness.

text: We

text: only

text: observed

text: a

text: background

text: level

text: of

text: optical

text: emission

text: near

text: the

text: Vela

text: pulsar

text: position,

text: and

text: there

text: is

text: no

text: structure

text: visible

text: that

text: could

text: be

text: associated

text: with

text: a

text: possible

text: Vela

text: X

text: optical

text: counterpart,

text: apart

text: from

text: some

text: hints

text: of

text: brighter

text: emission

text: near

text: the

text: position

text: of

text: the

text: southern

text: radio

text: lobe.

text: However,

text: this

text: may

text: be

text: due

text: to

text: bright

text: stars

text: that

text: were

text: not

text: cleanly

text: removed.

text: Our

text: νF

text: upper

text: limits

text: of

text: ν

text: erg

text: s

text: cm

text: are

text: clearly

text: not

text: constraining

text: for

text: a

text: multiwavelength

text: model

text: of

text: Vela

text: X’s

text: emission.

text: We

text: note

text: that

text: our

text: upper

text: limits

text: suﬀer

text: from

text: several

text: observational

text: factors

text: that

text: introduce

text: uncertainties.

text: Removal

text: of

text: bright

text: or

text: even

text: fainter

text: unresolved

text: point

text: sources

text: remain

text: problematic.

text: Our

text: integration

text: times

text: were

text: also

text: very

text: short.

text: Furthermore,

text: the

text: ﬁeld

text: of

text: view

text: of

text: the

text: telescope

text: is

text: too

text: small

text: to

text: capture

text: the

text: full

text: extension

text: of

text: the

text: source,

text: and

text: we

text: had

text: to

text: use

text: a

text: method

text: of

text: mosaicing

text: several

text: single

text: images

text: to

text: obtain

text: one

text: large

text: image

text: that

text: encapsulates

text: the

text: probable

text: extension

text: of

text: the

text: optical

text: source.

text: Changing

text: conditions

text: between

text: the

text: constituent

text: frames

text: add

text: complexity

text: and

text: uncertainty

text: to

text: our

text: analysis.

text: Neither

text: our

text: optical

text: limits

text: nor

text: those

text: of

text: others

text: prove

text: to

text: be

text: constraining

text: for

text: the

text: spectral

text: model

text: of

text: Vela

text: X.

text: Barring

text: the

text: gamma-ray

text: data

text: for

text: the

text: moment,

text: one

text: may

text: model

text: the

text: radio,

text: optical

text: and

text: X-ray

text: data

text: with

text: either

text: a

text: single

text: or

text: double

text: synchrotron

text: component,

text: implying

text: either

text: a

text: single

text: or

text: multi-component

text: electron

text: spectrum

text: interacting

text: with

text: the

text: PWN

text: B-ﬁeld.

text: However,

text: the

text: Fermi

text: and

text: H.E.S.S.

text: data

text: are

text: consistent

text: with

text: a

text: two-component

text: leptonic

text: interpretation

text: ﬁrst

text: put

text: forward

text: by

text: and

text: subsequently

text: elaborated

text: on

text: by

text: A

text: second

text: model

text: invokes

text: a

text: single

text: electron

text: population,

text: but

text: then

text: assumes

text: diﬀerent

text: diﬀusive

text: properties

text: for

text: the

text: extended

text: radio

text: nebula

text: (ERN)

text: and

text: cocoon

text: regions

text: (although

text: the

text: H.E.S.S.

text: “coccoon”

text: is

text: quite

text: more

text: extensive).

text: The

text: particles

text: are

text: assumed

text: to

text: be

text: contained

text: within

text: the

text: ERN

text: until

text: the

text: reverse

text: shock

text: from

text: the

text: SNR

text: interacts

text: with

text: the

text: ERN,

text: and

text: diﬀusive

text: transport

text: starts

text: to

text: take

text: place.

text: Older

text: particles

text: escape

text: then

text: from

text: the

text: ERN,

text: while

text: newly-injected

text: particles

text: are

text: thought

text: to

text: suﬀer

text: signiﬁcant

text: cooling

text: in

text: the

text: high-

text: B-ﬁeld

text: region

text: near

text: the

text: pulsar

text: before

text: being

text: injected

text: into

text: the

text: low-B-ﬁeld

text: cocoon.

text: Within

text: such

text: a

text: scenario,

text: each

text: substructure

text: (the

text: ERN

text: and

text: cocoon)

text: is

text: responsible

text: for

text: diﬀerent

text: components

text: of

text: the

text: multiwavelength

text: spectrum

text: (radio

text: and

text: GeV

text: vs.

text: X-ray

text: and

text: TeV).

text: In

text: essence,

text: this

text: is

text: a

text: diﬀerent

text: way

text: of

text: constructing

text: a

text: two-component

text: electron

text: injection

text: spectrum.

text: Our

text: optical

text: limits

text: are

text: not

text: violated

text: by

text: these

text: models,

text: although

text: the

text: available

text: limits

text: support

text: the

text: idea

text: of

text: particle

text: escape

text: from

text: the

text: ERN

text: (i.e.,

text: a

text: lower

text: synchrotron

text: ﬂux

text: component).

text: Deeper

text: optical

text: observations

text: as

text: well

text: as

text: reﬁned

text: modelling

text: should

text: continue

text: to

text: enhance

text: our

text: understanding

text: of

text: this

text: complex

text: source.

text: References


text: 2 bation between our bubble and others, which can give arise to the localized features in the CMB [20, 21]. But Such a potential can be derived from brane dynamics more detail data analysis [22] showed that the expected of string theory [26]. We will nevertheless not import number of bubble collision is too few to account for the anything other than this potential from string theory, features in the CMB. Alternatively, a cold spot may fol- but consider it as a phenomenological example. (cid:39) low from a diﬀerent trajectory during multi-stream in- To ﬁt the observations, we will take µ = 0.01 and V × − 0 ﬂation [23, 24] or modulated reheating after multi-ﬁeld 4.89 10 14 [27]. At the horizon crossing scale of the cold (cid:39) (cid:39) inﬂation [25]. spot, the inﬂaton ﬁeld is φ∗ 0.153 for N∗ 4 (counting The above physical or astronomical interpretations of from the start of the observable stage of inﬂation), and (cid:39) × − cold spot either fail at some level, or require ﬁne tuning the Hubble parameter is H∗ 1.28 10 7. The feature or exotic scenarios of the early universe. Economically, δV is put at φ = φ∗. 0 some cosmologists would prefer to interpret the cold spot In the isocurvature direction, the initial value of the merely as a “3σ” statistical ﬂuke. In this paper, we will (cid:104) (cid:105) ﬁeld χ∗ has no classical preferred value. We set χ∗ = 0 provide a natural and physically plausible explanation of (one can always shift the χ parameter to satisfy this re- 0 the cold spot, through multiple-ﬁeld inﬂation. If the in- quirement if needed). The quantum ﬂuctuation of χ∗(x) ﬂationary trajectory is scattered by a feature hidden in will be crucial in the analysis. Following the cosmic per- the isocurvature direction, the inﬂaton loses some energy turbation theory, χ∗(x) has a scale invariant power spec- and thus inﬂation tends to be longer. Therefore, it is trum in Fourier space and we normalize the power spec- possible that only a small portion of the sky hits the fea- trum to be P = [H/(2π)]2. In position space, χ∗(x) χ ture due to stochastic ﬂuctuations, then that local patch behaves as a Gaussian random ﬁeld. As we will show, of the universe experiences longer period of inﬂation and the nonlinear mapping from χ∗(x) to ∆T(x) generates thus produces a cold spot. The mechanism is illustrated the cold spot on the CMB. in Fig. 1. To explore the parameter space of δV , we will choose This paper is organized as follows. In Section , we pro- three sets of parameters as three illustrative examples of vide an explicit example of feature scattering, from mass- feature scattering. less isocurvature directions. Our predictions are com- • Benchmark 1: negative short feature. We set pared with the measurement of the cold-spot in Planck’s − × − A = 1.1 10 5V µ4/φ4∗, σ = 10H∗/(2π), σ = SMICA map. In Section , we consider isocurvature direc- 0 φ χ ∼ 0.7H∗/(2π) and χ = 4H∗/(2π). The procedure of tions with mass m H. We conclude in Section and 0 √ the calculation is also introduced in this subsection. discuss possible future directions. Throughout the pa- per, the unit Mpl = 1/ 8πG = 1 is used unless otherwise • stated. Bench−mark× 2: − negative long feature. We set II. COLD SPOT FROM FEATURE A = 1.4 10 5V0µ4/φ4∗, σφ = 50H∗/(2π), σχ = SCATTERING 0.7H∗/(2π) and χ = 5.2H∗/(2π). The diﬀerences 0 from Benchmark 1 are discussed. A. Potential choice and method of calculation • × Benchmark 3: positive feature. We set A = 1.0 − In this section, we provide an explicit example of fea- 10 4V0µ4/φ4∗, σφ = 50H∗/(2π), σχ = 0.5H∗/(2π) ture scattering to illustrate the physical mechanism. The and χ0 = 3.5H∗/(2π). The diﬀerences from Bench- generalization to other inﬂationary potentials should be marks 1 and 2 are discussed. straightforward. Consider inﬂation with two ﬁelds, both have standard kinetic term, and the potential is given by Note that in the above benchmarks, the randomness ∼ (cid:34) (cid:35) in the potential needs not to be large. In fact, δV/V − 10 5 is already signiﬁcant enough to bring observable V = V (φ) + δV , sr diﬀerence. This is because, the isocurvature ﬂuctuation − − δV = A exp −(φ φ0)2 − (χ χ0)2 . (1) control whether the inﬂation trajectory hit the feature. 2σ2 2σ2 And once the trajectory hits the feature, the change of φ χ ∼ ∼ − curvature perturbation will be order ζ δV/V 10 5. As a result, our mechanism is extremely sensitive to small The classical initial trajectory is chosen to be on φ features in the potential. direction, i.e. χ = 0. Here V (φ) is the slow-roll part of sr the potential. Our analysis is not sensitive in the form of The δN formalism [32–34] will be used to investigate V (φ), as long as it successfully drives slow-roll inﬂation. the cosmological perturbations from the feature scatter- sr For illustration purpose, we choose a small ﬁeld potential ing. The δN formalism uses the observation that diﬀer- for V (φ): (cid:18) (cid:19) ent Hubble patches during inﬂation can be approximated sr as diﬀerent local FRW universes. Thus the cosmological observables can be calculated by exploring those local − µ4 V (φ) = V 1 . (2) FRW universes. Especially, the curvature perturbation sr 0 φ4 in the uniform energy density slice can be calculated by ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: bation

text: between

text: our

text: bubble

text: and

text: others,

text: which

text: can

text: give

text: arise

text: to

text: the

text: localized

text: features

text: in

text: the

text: CMB

text: But

text: Such

text: a

text: potential

text: can

text: be

text: derived

text: from

text: brane

text: dynamics

text: more

text: detail

text: data

text: analysis

text: showed

text: that

text: the

text: expected

text: of

text: string

text: theory

text: We

text: will

text: nevertheless

text: not

text: import

text: number

text: of

text: bubble

text: collision

text: is

text: too

text: few

text: to

text: account

text: for

text: the

text: anything

text: other

text: than

text: this

text: potential

text: from

text: string

text: theory,

text: features

text: in

text: the

text: CMB.

text: Alternatively,

text: a

text: cold

text: spot

text: may

text: fol-

text: but

text: consider

text: it

text: as

text: a

text: phenomenological

text: example.

text: low

text: from

text: a

text: diﬀerent

text: trajectory

text: during

text: multi-stream

text: in-

text: To

text: ﬁt

text: the

text: observations,

text: we

text: will

text: take

text: µ

text: and

text: V

text: ﬂation

text: or

text: modulated

text: reheating

text: after

text: multi-ﬁeld

text: At

text: the

text: horizon

text: crossing

text: scale

text: of

text: the

text: cold

text: inﬂation

text: spot,

text: the

text: inﬂaton

text: ﬁeld

text: is

text: φ∗

text: for

text: N∗

text: (counting

text: The

text: above

text: physical

text: or

text: astronomical

text: interpretations

text: of

text: from

text: the

text: start

text: of

text: the

text: observable

text: stage

text: of

text: inﬂation),

text: and

text: cold

text: spot

text: either

text: fail

text: at

text: some

text: level,

text: or

text: require

text: ﬁne

text: tuning

text: the

text: Hubble

text: parameter

text: is

text: H∗

text: The

text: feature

text: or

text: exotic

text: scenarios

text: of

text: the

text: early

text: universe.

text: Economically,

text: δV

text: is

text: put

text: at

text: φ

text: φ∗.

text: some

text: cosmologists

text: would

text: prefer

text: to

text: interpret

text: the

text: cold

text: spot

text: In

text: the

text: isocurvature

text: direction,

text: the

text: initial

text: value

text: of

text: the

text: merely

text: as

text: a

text: “3σ”

text: statistical

text: ﬂuke.

text: In

text: this

text: paper,

text: we

text: will

text: ﬁeld

text: χ∗

text: has

text: no

text: classical

text: preferred

text: value.

text: We

text: set

text: χ∗

text: provide

text: a

text: natural

text: and

text: physically

text: plausible

text: explanation

text: of

text: (one

text: can

text: always

text: shift

text: the

text: χ

text: parameter

text: to

text: satisfy

text: this

text: re-

text: the

text: cold

text: spot,

text: through

text: multiple-ﬁeld

text: inﬂation.

text: If

text: the

text: in-

text: quirement

text: if

text: needed).

text: The

text: quantum

text: ﬂuctuation

text: of

text: χ∗(x)

text: ﬂationary

text: trajectory

text: is

text: scattered

text: by

text: a

text: feature

text: hidden

text: in

text: will

text: be

text: crucial

text: in

text: the

text: analysis.

text: Following

text: the

text: cosmic

text: per-

text: the

text: isocurvature

text: direction,

text: the

text: inﬂaton

text: loses

text: some

text: energy

text: turbation

text: theory,

text: χ∗(x)

text: has

text: a

text: scale

text: invariant

text: power

text: spec-

text: and

text: thus

text: inﬂation

text: tends

text: to

text: be

text: longer.

text: Therefore,

text: it

text: is

text: trum

text: in

text: Fourier

text: space

text: and

text: we

text: normalize

text: the

text: power

text: spec-

text: possible

text: that

text: only

text: a

text: small

text: portion

text: of

text: the

text: sky

text: hits

text: the

text: fea-

text: trum

text: to

text: be

text: P

text: In

text: position

text: space,

text: χ∗(x)

text: χ

text: ture

text: due

text: to

text: stochastic

text: ﬂuctuations,

text: then

text: that

text: local

text: patch

text: behaves

text: as

text: a

text: Gaussian

text: random

text: ﬁeld.

text: As

text: we

text: will

text: show,

text: of

text: the

text: universe

text: experiences

text: longer

text: period

text: of

text: inﬂation

text: and

text: the

text: nonlinear

text: mapping

text: from

text: χ∗(x)

text: to

text: ∆T(x)

text: generates

text: thus

text: produces

text: a

text: cold

text: spot.

text: The

text: mechanism

text: is

text: illustrated

text: the

text: cold

text: spot

text: on

text: the

text: CMB.

text: in

text: Fig.

text: To

text: explore

text: the

text: parameter

text: space

text: of

text: δV

text: we

text: will

text: choose

text: This

text: paper

text: is

text: organized

text: as

text: follows.

text: In

text: Section

text: we

text: pro-

text: three

text: sets

text: of

text: parameters

text: as

text: three

text: illustrative

text: examples

text: of

text: vide

text: an

text: explicit

text: example

text: of

text: feature

text: scattering,

text: from

text: mass-

text: feature

text: scattering.

text: less

text: isocurvature

text: directions.

text: Our

text: predictions

text: are

text: com-

text: Benchmark

text: negative

text: short

text: feature.

text: We

text: set

text: pared

text: with

text: the

text: measurement

text: of

text: the

text: cold-spot

text: in

text: Planck’s

text: A

text: σ

text: σ

text: SMICA

text: map.

text: In

text: Section

text: we

text: consider

text: isocurvature

text: direc-

text: φ

text: χ

text: and

text: χ

text: The

text: procedure

text: of

text: tions

text: with

text: mass

text: m

text: H.

text: We

text: conclude

text: in

text: Section

text: and

text: the

text: calculation

text: is

text: also

text: introduced

text: in

text: this

text: subsection.

text: discuss

text: possible

text: future

text: directions.

text: Throughout

text: the

text: pa-

text: per,

text: the

text: unit

text: Mpl

text: is

text: used

text: unless

text: otherwise

text: stated.

text: Bench−mark×

text: negative

text: long

text: feature.

text: We

text: set

text: II.

text: COLD

text: SPOT

text: FROM

text: FEATURE

text: A

text: σφ

text: σχ

text: SCATTERING

text: and

text: χ

text: The

text: diﬀerences

text: from

text: Benchmark

text: are

text: discussed.

text: A.

text: Potential

text: choice

text: and

text: method

text: of

text: calculation

text: Benchmark

text: positive

text: feature.

text: We

text: set

text: A

text: In

text: this

text: section,

text: we

text: provide

text: an

text: explicit

text: example

text: of

text: fea-

text: σφ

text: σχ

text: ture

text: scattering

text: to

text: illustrate

text: the

text: physical

text: mechanism.

text: The

text: and

text: χ0

text: The

text: diﬀerences

text: from

text: Bench-

text: generalization

text: to

text: other

text: inﬂationary

text: potentials

text: should

text: be

text: marks

text: and

text: are

text: discussed.

text: straightforward.

text: Consider

text: inﬂation

text: with

text: two

text: ﬁelds,

text: both

text: have

text: standard

text: kinetic

text: term,

text: and

text: the

text: potential

text: is

text: given

text: by

text: Note

text: that

text: in

text: the

text: above

text: benchmarks,

text: the

text: randomness

text: in

text: the

text: potential

text: needs

text: not

text: to

text: be

text: large.

text: In

text: fact,

text: δV/V

text: is

text: already

text: signiﬁcant

text: enough

text: to

text: bring

text: observable

text: V

text: V

text: (φ)

text: δV

text: sr

text: diﬀerence.

text: This

text: is

text: because,

text: the

text: isocurvature

text: ﬂuctuation

text: δV

text: A

text: exp

text: −(φ

text: (χ

text: control

text: whether

text: the

text: inﬂation

text: trajectory

text: hit

text: the

text: feature.

text: And

text: once

text: the

text: trajectory

text: hits

text: the

text: feature,

text: the

text: change

text: of

text: φ

text: χ

text: curvature

text: perturbation

text: will

text: be

text: order

text: ζ

text: δV/V

text: As

text: a

text: result,

text: our

text: mechanism

text: is

text: extremely

text: sensitive

text: to

text: small

text: The

text: classical

text: initial

text: trajectory

text: is

text: chosen

text: to

text: be

text: on

text: φ

text: features

text: in

text: the

text: potential.

text: direction,

text: i.e.

text: χ

text: Here

text: V

text: (φ)

text: is

text: the

text: slow-roll

text: part

text: of

text: sr

text: the

text: potential.

text: Our

text: analysis

text: is

text: not

text: sensitive

text: in

text: the

text: form

text: of

text: The

text: δN

text: formalism

text: will

text: be

text: used

text: to

text: investigate

text: V

text: (φ),

text: as

text: long

text: as

text: it

text: successfully

text: drives

text: slow-roll

text: inﬂation.

text: the

text: cosmological

text: perturbations

text: from

text: the

text: feature

text: scatter-

text: sr

text: For

text: illustration

text: purpose,

text: we

text: choose

text: a

text: small

text: ﬁeld

text: potential

text: ing.

text: The

text: δN

text: formalism

text: uses

text: the

text: observation

text: that

text: diﬀer-

text: for

text: V

text: (φ):

text: ent

text: Hubble

text: patches

text: during

text: inﬂation

text: can

text: be

text: approximated

text: sr

text: as

text: diﬀerent

text: local

text: FRW

text: universes.

text: Thus

text: the

text: cosmological

text: observables

text: can

text: be

text: calculated

text: by

text: exploring

text: those

text: local

text: µ4

text: V

text: (φ)

text: V

text: FRW

text: universes.

text: Especially,

text: the

text: curvature

text: perturbation

text: sr

text: φ4

text: in

text: the

text: uniform

text: energy

text: density

text: slice

text: can

text: be

text: calculated

text: by

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##