text: - TVAL3 (Total-Variation minimization by Augmented Lagrangian and ALter- nating direction ALgorithms) is a MATLAB software for the solution of sig- nal reconstruction problems regularized with the total-variation semi-norm. TVAL3 reformulates problem (1.1) to the equivalent problem (cid:88) l (cid:107) (cid:124) (cid:107) 1 (cid:107) − (cid:107) (7.3) minimize Ω x + Ax b 2, i 2 2c 2 i=1 ∈ R × where Ω = [ReW , ImW ] n 2. Then it solves the augmented Lagrangian i i i reformulation of problem (7.3), which is (cid:88) l (cid:107) (cid:107) β (cid:107) (cid:124) − (cid:107) − (cid:124) (cid:124) − 1 (cid:107) − (cid:107) (7.4) minimize ( u + Ω x u 2 v (Ω x u )) + Ax b 2, i 2 2 i i 2 i i i 2c 2 i=1 ∈ R where u , v 2 and β, c are positive constants. The augmented Lagrangian i i ∈ R · · · in (7.4) is minimized for variables x n and u i = 1, 2, , l. The param- · · · i eters v i = 1, 2, , l are handled by the method. i - TwIST (Two-step Iterative Soft Thresholding): is also a MATLAB software for signal/image processing problems. TwIST solves problem (1.1). TwIST is a nonlinear two-step iterative version of IST, which according to its authors is more eﬀective on ill-conditioned and ill-posed problems. Another solver is NestA [4] (by the same authors as TFOCS) which can also solve (cid:124) − (1.1) but it is applicable only in the case that (AA ) 1 is available. Additionally, TFOCS is the sucessor of NestA, since both apply the similar techniques but TFOCS is a newer and allows of more control options. Another method is the Primal-Dual Hybrid Gradient (PDHG) of [14]. PDHG has been reported to be very eﬃcient for imaging applications such as denoising and deblurring, for which matrix A is the identity or a square and full-rank matrix which is inexpensively diagonalizable. Unfortunately, this is not the case for the CS problems that we are interested in. However, for all earlier mentioned methods the matrix inversion can be replaced with a solution of a linear system at every iteration of the methods or a one-time cost of a factorization. To the best of our knowledge, there are no available implementations with such modiﬁcations for these methods. There exists also a generic proximal algorithm for total-variation [12] and the Generalized Iterative Soft Thresholding (GISTA) in [20] for which we do not have generic implementations for CS problems. Solvers pdNCG, TFOCS unc, TVAL3 and TwIST solve the penalized least squares problem (1.1), while TFOCS con solves the constrained least squares problem (7.2). In our experiments we put signiﬁcant eﬀort in calibrating the parameters c and (cid:15) for the penalized and constrained least-squares problems, respectively, such that all (cid:107) − (cid:107) methods solve similar problems. First, we set (cid:15) = b ˜b in (7.2), where ˜b is the 2 noiseless sampled signal. Hence problem (7.2) is parameterized with the optimal (cid:15). Then we ﬁnd an approximation of the optimal c. By optimal c we mean the value of (cid:107) − (cid:107) c for which problems (1.1) and (7.2) are equivalent if (cid:15) = b ˜b and µ = 0. Let ω denote the optimal Lagrange multiplier of (7.2). If (cid:15) = (cid:107)b − ˜b(cid:107) 2 and µ T2= 0, then 2 T2 it is easy to show that for c := 2/ω problems (1.1) and (7.2) are equivalent. 21 ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

label: list

text: (Total-Variation

text: minimization

text: by

text: Augmented

text: Lagrangian

text: and

text: ALter-

text: nating

text: direction

text: ALgorithms)

text: is

text: a

text: MATLAB

text: software

text: for

text: the

text: solution

text: of

text: sig-

text: nal

text: reconstruction

text: problems

text: regularized

text: with

text: the

text: total-variation

text: semi-norm.

text: reformulates

text: problem

text: to

text: the

text: equivalent

text: problem

text: l

text: minimize

text: Ω

text: x

text: Ax

text: b

text: i

text: R

text: where

text: Ω

text: [ReW

text: ImW

text: n

text: Then

text: it

text: solves

text: the

text: augmented

text: Lagrangian

text: i

text: i

text: i

text: reformulation

text: of

text: problem

text: which

text: is

text: l

text: β

text: minimize

text: u

text: Ω

text: x

text: u

text: v

text: (Ω

text: x

text: u

text: Ax

text: b

text: i

text: i

text: i

text: i

text: i

text: i

text: R

text: where

text: u

text: v

text: and

text: β,

text: c

text: are

text: positive

text: constants.

text: The

text: augmented

text: Lagrangian

text: i

text: i

text: R

text: in

text: is

text: minimized

text: for

text: variables

text: x

text: n

text: and

text: u

text: i

text: l.

text: The

text: param-

text: i

text: eters

text: v

text: i

text: l

text: are

text: handled

text: by

text: the

text: method.

text: i

text: TwIST

text: (Two-step

text: Iterative

text: Soft

text: Thresholding):

text: is

text: also

text: a

text: MATLAB

text: software

text: for

text: signal/image

text: processing

text: problems.

text: TwIST

text: solves

text: problem

text: TwIST

text: is

text: a

text: nonlinear

text: two-step

text: iterative

text: version

text: of

text: IST,

text: which

text: according

text: to

text: its

text: authors

text: is

text: more

text: eﬀective

text: on

text: ill-conditioned

text: and

text: ill-posed

text: problems.

text: Another

text: solver

text: is

text: NestA

text: (by

text: the

text: same

text: authors

text: as

text: TFOCS)

text: which

text: can

text: also

text: solve

text: but

text: it

text: is

text: applicable

text: only

text: in

text: the

text: case

text: that

text: (AA

text: is

text: available.

text: Additionally,

text: TFOCS

text: is

text: the

text: sucessor

text: of

text: NestA,

text: since

text: both

text: apply

text: the

text: similar

text: techniques

text: but

text: TFOCS

text: is

text: a

text: newer

text: and

text: allows

text: of

text: more

text: control

text: options.

text: Another

text: method

text: is

text: the

text: Primal-Dual

text: Hybrid

text: Gradient

text: (PDHG)

text: of

text: PDHG

text: has

text: been

text: reported

text: to

text: be

text: very

text: eﬃcient

text: for

text: imaging

text: applications

text: such

text: as

text: denoising

text: and

text: deblurring,

text: for

text: which

text: matrix

text: A

text: is

text: the

text: identity

text: or

text: a

text: square

text: and

text: full-rank

text: matrix

text: which

text: is

text: inexpensively

text: diagonalizable.

text: Unfortunately,

text: this

text: is

text: not

text: the

text: case

text: for

text: the

text: CS

text: problems

text: that

text: we

text: are

text: interested

text: in.

text: However,

text: for

text: all

text: earlier

text: mentioned

text: methods

text: the

text: matrix

text: inversion

text: can

text: be

text: replaced

text: with

text: a

text: solution

text: of

text: a

text: linear

text: system

text: at

text: every

text: iteration

text: of

text: the

text: methods

text: or

text: a

text: one-time

text: cost

text: of

text: a

text: factorization.

text: To

text: the

text: best

text: of

text: our

text: knowledge,

text: there

text: are

text: no

text: available

text: implementations

text: with

text: such

text: modiﬁcations

text: for

text: these

text: methods.

text: There

text: exists

text: also

text: a

text: generic

text: proximal

text: algorithm

text: for

text: total-variation

text: and

text: the

text: Generalized

text: Iterative

text: Soft

text: Thresholding

text: (GISTA)

text: in

text: for

text: which

text: we

text: do

text: not

text: have

text: generic

text: implementations

text: for

text: CS

text: problems.

text: Solvers

text: pdNCG,

text: TFOCS

text: unc,

text: and

text: TwIST

text: solve

text: the

text: penalized

text: least

text: squares

text: problem

text: while

text: TFOCS

text: con

text: solves

text: the

text: constrained

text: least

text: squares

text: problem

text: In

text: our

text: experiments

text: we

text: put

text: signiﬁcant

text: eﬀort

text: in

text: calibrating

text: the

text: parameters

text: c

text: and

text: for

text: the

text: penalized

text: and

text: constrained

text: least-squares

text: problems,

text: respectively,

text: such

text: that

text: all

text: methods

text: solve

text: similar

text: problems.

text: First,

text: we

text: set

text: b

text: ˜b

text: in

text: where

text: ˜b

text: is

text: the

text: noiseless

text: sampled

text: signal.

text: Hence

text: problem

text: is

text: parameterized

text: with

text: the

text: optimal

text: Then

text: we

text: ﬁnd

text: an

text: approximation

text: of

text: the

text: optimal

text: c.

text: By

text: optimal

text: c

text: we

text: mean

text: the

text: value

text: of

text: c

text: for

text: which

text: problems

text: and

text: are

text: equivalent

text: if

text: b

text: ˜b

text: and

text: µ

text: Let

text: ω

text: denote

text: the

text: optimal

text: Lagrange

text: multiplier

text: of

text: If

text: and

text: µ

text: then

text: it

text: is

text: easy

text: to

text: show

text: that

text: for

text: c

text: problems

text: and

text: are

text: equivalent.

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: RNS (a) pdNCG, SNR 64.7, 525 sec (b) TFOCS con, SNR 62.3, 556 sec 60 50 40 30 20 10 pdNCG TFOCS_con 0 50 100 150 200 250 300 350 400 450 500 550 CPU time (sec) (c) SNR against CPU time Fig. 7.4: Reconstruction of two radio-frequency radar tones by TFOCS con and pdNCG. In Figure 7.4a the reconstructed signal by pdNCG is shown. The signal has SNR 64.7 dB and it required 525 seconds of CPU time to be reconstructed. In Figure 7.4b the reconstructed signal by TFOCS con is shown. The signal has PSNR 62.3 dB and it required 556 seconds of CPU time to be reconstructed. In Figure 7.4c we present the SNR for each iteration of pdNCG and TFOCS con against CPU time. [5] S. R. Becker, E. J. Cand´es, and M. C. Grant. Templates for convex cone problems with applications to sparse signal recovery. Mathematical Programming Computation, 3(3):165– 218, 2011. http://cvxr.com/tfocs/. [6] J. M. Bioucas-Dias and M. A. T. Figueiredo. A new TwIST: Two-step iterative shrink- age/thresholding algorithms for image restoration. IEEE Transactions on Image Pro- cessing, 16(12):2992–3004, 2007. [7] E. J. Cand´es. The restricted isometry property and its implications for compressed sensing. C. R. Acad. Sci. Paris, 346:589–592, 2008. [8] E. J. Cand´es and D. L. Donoho. New tight frames of curvelets and optimal representations of objects with piecewise c2 singularities. Comm. Pure Appl. Math., 57:219–266, 2004. [9] E. J. Cand´es, Y. C. Eldar, and D. Needell. Compressed sensing with coherent and redundant dictionaries. Applied and Computational Harmonic Analysis, 31(1):59–73, 2011. [10] R. H. Chan, T. F. Chan, and H. M. Zhou. Advanced signal processing algorithms. in Proceedings 32 ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: RNS

label: paragraph

font: default

text: (a)

text: pdNCG,

label: caption

text: SNR

text: sec

text: (b)

text: TFOCS

text: con,

text: SNR

text: sec

text: pdNCG

text: CPU

text: time

text: (sec)

text: (c)

text: SNR

text: against

text: CPU

text: time

text: Fig.

text: Reconstruction

text: of

text: two

text: radio-frequency

text: radar

text: tones

text: by

text: TFOCS

text: con

text: and

text: pdNCG.

text: In

text: Figure

text: the

text: reconstructed

text: signal

text: by

text: pdNCG

text: is

text: shown.

text: The

text: signal

text: has

text: SNR

text: dB

text: and

text: it

text: required

text: seconds

text: of

text: CPU

text: time

text: to

text: be

text: reconstructed.

text: In

text: Figure

text: the

text: reconstructed

text: signal

text: by

text: TFOCS

text: con

text: is

text: shown.

text: The

text: signal

text: has

text: PSNR

text: dB

text: and

text: it

text: required

text: seconds

text: of

text: CPU

text: time

text: to

text: be

text: reconstructed.

text: In

text: Figure

text: we

text: present

text: the

text: SNR

text: for

text: each

text: iteration

text: of

text: pdNCG

text: and

text: TFOCS

text: con

text: against

text: CPU

text: time.

text: S.

text: R.

text: Becker,

text: E.

text: J.

text: Cand´es,

text: and

text: M.

text: C.

text: Grant.

text: Templates

text: for

text: convex

text: cone

text: problems

text: with

text: applications

text: to

text: sparse

text: signal

text: recovery.

text: Mathematical

text: Programming

text: Computation,

text: J.

text: M.

text: Bioucas-Dias

text: and

text: M.

text: A.

text: T.

text: Figueiredo.

text: A

text: new

text: TwIST:

text: Two-step

text: iterative

text: shrink-

text: age/thresholding

text: algorithms

text: for

text: image

text: restoration.

text: IEEE

text: Transactions

text: on

text: Image

text: Pro-

text: cessing,

text: E.

text: J.

text: Cand´es.

text: The

text: restricted

text: isometry

text: property

text: and

text: its

text: implications

text: for

text: compressed

text: sensing.

text: C.

text: R.

text: Acad.

text: Sci.

text: Paris,

text: E.

text: J.

text: Cand´es

text: and

text: D.

text: L.

text: Donoho.

text: New

text: tight

text: frames

text: of

text: curvelets

text: and

text: optimal

text: representations

text: of

text: objects

text: with

text: piecewise

text: singularities.

text: Comm.

text: Pure

text: Appl.

text: Math.,

text: E.

text: J.

text: Cand´es,

text: Y.

text: C.

text: Eldar,

text: and

text: D.

text: Needell.

text: Compressed

text: sensing

text: with

text: coherent

text: and

text: redundant

text: dictionaries.

text: Applied

text: and

text: Computational

text: Harmonic

text: Analysis,

text: R.

text: H.

text: Chan,

text: T.

text: F.

text: Chan,

text: and

text: H.

text: M.

text: Zhou.

text: Advanced

text: signal

text: processing

text: algorithms.

text: in

text: Proceedings

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: O components of (ω). A consequence of the previous statement is that the compo- ∗ nents of W x split into the following disjoint sets c,µ B { ∈ { · · · } | | ∗ | (cid:29) O } |B| | ∗ | := i 1, 2, , l W x (ω) , = q = supp(W x ) , i c,µ c B { ∈ { · · · } | | ∗ | ≈ O } |B | − c := i 1, 2, , l W x (ω) , c = l q. i c,µ ∗ ∇ ∗ The behaviour of W x has a crucial inﬂuence on matrix 2ψ (W x ) in (2.6). c,µ µ c,µ Notice that the components of the diagonal matrix D, deﬁned in (2.5) as part of ∇ ∗ 2ψ (W x ), split into two disjoint sets. In particular, q components are non-zeros µ c,µ O − O much less than (1/ω), while the majority, l q, of its components are of (1/ω), (cid:28) O 1 ∀ ∈ B O 1 ∀ ∈ B (5.1) D ( ) i and D = ( ) i c. i ω i ω ∇ Hence, for points close to x and small µ, matrix 2fµ(x) in (2.8) consists of a ∇ c,µ (cid:124) c dominant matrix c 2ψ (x) and of matrix A A with moderate largest eigenvalue. The µ (cid:124) (cid:124) (cid:124) previous argument for A A is due to (1.3). Observe that λ (A A) = λ (AA ), m(cid:124)ax ≤ max hence, if δ in (1.3) is not a very large constant, then λ (A A) 1 + δ. According max ∇ to Remark 4.2, the symmetric matrix sym(B˜) in (2.8) tends to matrix 2ψ (x) as → µ x x . Therefore, matrix sym(B˜) is the dominant matrix in Bˆ. For this reason, in c,µ (cid:124) the proposed preconditioning technique, matrix A A in (2.8) is replaced by a scaled identity ρI , ρ > 0, while the dominant matrix sym(B˜) is maintained. Based on these n observations we propose the following preconditioner (5.2) N˜ := c sym(B˜) + ρI . n In order to capture the approximate separability of the diagonal components of matrix D for points close to x , when µ is suﬃciently small, we will work with B B c,µ approximate guess of and c. For this reason, we introduce the positive constant ν, such that #(D < ν) = σ. i ∗ Here σ might be diﬀerent from the sparsity of W x . Furthermore, according to the c above deﬁnition we have the sets B { ∈ { · · · } | } B { · · · }\B (5.3) := i 1, 2, , l D < ν and c := 1, 2, , l , ν i ν ν |B | |B | − with = σ and c = l σ. This notation is being used in the following ν ν theorem, in which we analyze the behaviour of the spectral properties of precondi- ∇ ∇ ∗ tioned 2fµ(x), with preconditioner N := c 2ψ (W x) + ρI . However, according c ∇ µ n → to Remark 4.2 matrices Bˆ and N˜ tend to 2fµ(x) and N, respectively, as x x . c c,µ Therefore, the following theorem is useful for the analysis of the limiting behaviour of the spectrum of preconditioned Bˆ. 9 ##LTLine## ##LTLine##

text: O

text: components

text: of

text: (ω).

text: A

text: consequence

text: of

text: the

text: previous

text: statement

text: is

text: that

text: the

text: compo-

text: nents

text: of

text: W

text: x

text: split

text: into

text: the

text: following

text: disjoint

text: sets

text: c,µ

text: B

text: O

text: |B|

text: i

text: l

text: W

text: x

text: (ω)

text: q

text: supp(W

text: x

text: i

text: c,µ

text: c

text: B

text: O

text: |B

text: c

text: i

text: l

text: W

text: x

text: (ω)

text: c

text: l

text: q.

text: i

text: c,µ

text: The

text: behaviour

text: of

text: W

text: x

text: has

text: a

text: crucial

text: inﬂuence

text: on

text: matrix

text: 2ψ

text: (W

text: x

text: in

text: c,µ

text: µ

text: c,µ

text: Notice

text: that

text: the

text: components

text: of

text: the

text: diagonal

text: matrix

text: D,

text: deﬁned

text: in

text: as

text: part

text: of

text: 2ψ

text: (W

text: x

text: split

text: into

text: two

text: disjoint

text: sets.

text: In

text: particular,

text: q

text: components

text: are

text: non-zeros

text: µ

text: c,µ

text: O

text: O

text: much

text: less

text: than

text: while

text: the

text: majority,

text: l

text: q,

text: of

text: its

text: components

text: are

text: of

text: O

text: B

text: O

text: B

text: D

text: i

text: and

text: D

text: i

text: c.

text: i

text: ω

text: i

text: ω

text: Hence,

text: for

text: points

text: close

text: to

text: x

text: and

text: small

text: µ,

text: matrix

text: in

text: consists

text: of

text: a

text: c,µ

text: c

text: dominant

text: matrix

text: c

text: 2ψ

text: (x)

text: and

text: of

text: matrix

text: A

text: A

text: with

text: moderate

text: largest

text: eigenvalue.

text: The

text: µ

text: previous

text: argument

text: for

text: A

text: A

text: is

text: due

text: to

text: Observe

text: that

text: λ

text: (A

text: A)

text: λ

text: (AA

text: max

text: hence,

text: if

text: δ

text: in

text: is

text: not

text: a

text: very

text: large

text: constant,

text: then

text: λ

text: (A

text: A)

text: δ.

text: According

text: max

text: to

text: Remark

text: the

text: symmetric

text: matrix

text: sym(B˜)

text: in

text: tends

text: to

text: matrix

text: 2ψ

text: (x)

text: as

text: µ

text: x

text: x

text: Therefore,

text: matrix

text: sym(B˜)

text: is

text: the

text: dominant

text: matrix

text: in

text: Bˆ.

text: For

text: this

text: reason,

text: in

text: c,µ

text: the

text: proposed

text: preconditioning

text: technique,

text: matrix

text: A

text: A

text: in

text: is

text: replaced

text: by

text: a

text: scaled

text: identity

text: ρI

text: ρ

text: while

text: the

text: dominant

text: matrix

text: sym(B˜)

text: is

text: maintained.

text: Based

text: on

text: these

text: n

text: observations

text: we

text: propose

text: the

text: following

text: preconditioner

text: N˜

text: c

text: sym(B˜)

text: ρI

text: n

text: In

text: order

text: to

text: capture

text: the

text: approximate

text: separability

text: of

text: the

text: diagonal

text: components

text: of

text: matrix

text: D

text: for

text: points

text: close

text: to

text: x

text: when

text: µ

text: is

text: suﬃciently

text: small,

text: we

text: will

text: work

text: with

text: B

text: B

text: c,µ

text: approximate

text: guess

text: of

text: and

text: c.

text: For

text: this

text: reason,

text: we

text: introduce

text: the

text: positive

text: constant

text: ν,

text: such

text: that

text: #(D

text: ν)

text: σ.

text: i

text: Here

text: σ

text: might

text: be

text: diﬀerent

text: from

text: the

text: sparsity

text: of

text: W

text: x

text: Furthermore,

text: according

text: to

text: the

text: c

text: above

text: deﬁnition

text: we

text: have

text: the

text: sets

text: B

text: B

text: }\B

text: i

text: l

text: D

text: ν

text: and

text: c

text: l

text: ν

text: i

text: ν

text: ν

text: |B

text: |B

text: with

text: σ

text: and

text: c

text: l

text: σ.

text: This

text: notation

text: is

text: being

text: used

text: in

text: the

text: following

text: ν

text: ν

text: theorem,

text: in

text: which

text: we

text: analyze

text: the

text: behaviour

text: of

text: the

text: spectral

text: properties

text: of

text: precondi-

text: tioned

text: with

text: preconditioner

text: N

text: c

text: 2ψ

text: (W

text: x)

text: ρI

text: However,

text: according

text: c

text: µ

text: n

text: to

text: Remark

text: matrices

text: Bˆ

text: and

text: N˜

text: tend

text: to

text: and

text: N,

text: respectively,

text: as

text: x

text: x

text: c

text: c,µ

text: Therefore,

text: the

text: following

text: theorem

text: is

text: useful

text: for

text: the

text: analysis

text: of

text: the

text: limiting

text: behaviour

text: of

text: the

text: spectrum

text: of

text: preconditioned

text: Bˆ.

text: ##LTLine##

text: ##LTLine##


text: 4 Extension to Generalized Quadratic Regression Mod- els 4.1 Generalized Quadratic Regression A standard generalized linear model (GLM) assumes that the conditional distribution of y given X belongs to the canonical exponential family with density (cid:26) (cid:20) (cid:21)(cid:27) (cid:89) (cid:89) − n n y θ b(θ ) i i i f (y, X, β) = f (y ; θ ) = c(y ) exp , n 0 i i i φ i=1 i=1 (cid:62) where φ > 0 is a dispersion parameter, β = (β , ..., β ) are the regression coeﬃcients, 1 p and (cid:62) θ = (θ , ..., θ ) = Xβ. (10) 1 n The function b(θ) is twice continuously diﬀerentiable with a positive second-order deriva- tive. In sparse high dimensional modeling, β is a long vector with a small number of nonzero ◦ entries. In the context of QR, the design matrix is (X, X 2). A natural generalization of GLM is to modify (10) as (cid:62) ◦ θ = (θ , ..., θ ) = XβM + X 2βI. (11) 1 n In the literature, there are very few computational tools available to ﬁt high dimensional GQR models. Next, we illustrate how the aforementioned algorithms can be used for GQR. 12 ##LTLine##

label: section

text: Extension

text: to

text: Generalized

text: Quadratic

text: Regression

text: Mod-

text: els

text: Generalized

text: Quadratic

text: Regression

text: A

text: standard

text: generalized

text: linear

text: model

text: (GLM)

text: assumes

text: that

text: the

text: conditional

text: distribution

text: of

text: y

text: given

text: X

text: belongs

text: to

text: the

text: canonical

text: exponential

text: family

text: with

text: density

label: equation

text: n

text: n

text: y

text: θ

text: b(θ

text: i

text: i

text: i

text: f

text: (y,

text: X,

text: β)

text: f

text: (y

text: θ

text: c(y

text: exp

text: n

text: i

text: i

text: i

text: φ

text: where

text: φ

text: is

text: a

text: dispersion

text: parameter,

text: β

text: (β

text: β

text: are

text: the

text: regression

text: coeﬃcients,

text: p

text: and

text: θ

text: (θ

text: θ

text: Xβ.

text: n

text: The

text: function

text: b(θ)

text: is

text: twice

text: continuously

text: diﬀerentiable

text: with

text: a

text: positive

text: second-order

text: deriva-

text: tive.

text: In

text: sparse

text: high

text: dimensional

text: modeling,

text: β

text: is

text: a

text: long

text: vector

text: with

text: a

text: small

text: number

text: of

text: nonzero

text: entries.

text: In

text: the

text: context

text: of

text: QR,

text: the

text: design

text: matrix

text: is

text: (X,

text: X

text: A

text: natural

text: generalization

text: of

text: GLM

text: is

text: to

text: modify

text: as

text: θ

text: (θ

text: θ

text: XβM

text: X

text: n

text: In

text: the

text: literature,

text: there

text: are

text: very

text: few

text: computational

text: tools

text: available

text: to

text: ﬁt

text: high

text: dimensional

text: GQR

text: models.

text: Next,

text: we

text: illustrate

text: how

text: the

text: aforementioned

text: algorithms

text: can

text: be

text: used

text: for

text: GQR.

text: ##LTLine##


text: where we consider a relatively small p to make the comparison possible with the hierarchical lasso (Bien et al., 2013). Example 4 considers a QR model with a heavy tail error distribution to demonstrate the robustness of our methods. For comparison, we consider RAMP and two two-stage methods, i.e., two-stage LASSO (2-LASSO) and two-stage SCAD (2-SCAD). We also include existing methods iFORT and iFORM (Hao & Zhang, 2014a), the hierarchical lasso (Bien et al., 2013), and the benchmark method ORACLE for which the true sparse model is known. When computing the solution paths of two-stage methods and RAMP, we choose the tuning parameter by EBIC with γ = 1 (Chen & Chen, 2008). We also implemented other parameter tuning criteria including AIC, BIC, and GIC (Fan & Tang, 2013), and observed that the EBIC tends to work the best among most of the simulation settings that we con- sidered. For easy presentation, we report only the results for EBIC. S { (cid:54) } T { (cid:54) } |S| |T | Let = j : β = 0 and = (j, k) : β = 0 with cardinality s = and t = . j j,k (cid:98) (cid:98) For each example, we run M = 100 Monte-Carlo simulations for each method and make (cid:98) (cid:98) S T a comparison. For the m-th simulation, denote the estimated subsets as (m) and (m), (cid:98) the estimated coeﬃcient vector as β(m), the main eﬀects and interaction eﬀects as β(m) and j β(m). We evaluate the performance on variable selection and model estimation based on the j,k (cid:80) (cid:98) following criteria. • − (cid:80) S ⊂ S Main eﬀects coverage percentage (main.cov): M 1 M I( (m)).(cid:98) m=1 • − (cid:80) T ⊂ T Interaction eﬀects coverage percentage (inter.cov): M 1 M I( (m(cid:98))). m=1 • − (cid:80) S S Main eﬀects exact selection percentage (main.exact): M 1 M I( = (m)).(cid:98) m=1 • (cid:80) − T T Interaction eﬀects exact selectio(cid:98)n percen(cid:98)tage (inter.exact): M 1 M I( = (m)). m=1 • − |S | |T (cid:80) | (cid:80) (cid:80) (cid:80) Model size (size): M 1 M ( (m) + (m) ). m=1 • { − − − Root mean squared error (RMSE): M 1 M [ p (βˆ(m) β )2 + p p (βˆ(m) m=1 j=0 j j j=1 k=j j,k } β )2] 1/2. j,k { } ∼ N Example 1 Set (n, p, s, t) = (400, 5000, 10, 10). Generate the covariates x n i.i.d. (0, Σ) i i=1 | − | S { · · · } with Σ = 0.5 j k and generate the response y by model (1). = 1, 2, , 10 with the jk 15

text: where

text: we

text: consider

text: a

text: relatively

text: small

text: p

text: to

text: make

text: the

text: comparison

text: possible

text: with

text: the

text: hierarchical

text: lasso

text: (Bien

text: et

text: al.,

text: Example

text: considers

text: a

text: QR

text: model

text: with

text: a

text: heavy

text: tail

text: error

text: distribution

text: to

text: demonstrate

text: the

text: robustness

text: of

text: our

text: methods.

text: For

text: comparison,

text: we

text: consider

text: RAMP

text: and

text: two

text: two-stage

text: methods,

text: i.e.,

text: two-stage

text: LASSO

text: and

text: two-stage

text: SCAD

text: We

text: also

text: include

text: existing

text: methods

text: iFORT

text: and

text: iFORM

text: (Hao

text: Zhang,

text: the

text: hierarchical

text: lasso

text: (Bien

text: et

text: al.,

text: and

text: the

text: benchmark

text: method

text: ORACLE

text: for

text: which

text: the

text: true

text: sparse

text: model

text: is

text: known.

text: When

text: computing

text: the

text: solution

text: paths

text: of

text: two-stage

text: methods

text: and

text: RAMP,

text: we

text: choose

text: the

text: tuning

text: parameter

text: by

text: EBIC

text: with

text: γ

text: (Chen

text: Chen,

text: We

text: also

text: implemented

text: other

text: parameter

text: tuning

text: criteria

text: including

text: AIC,

text: BIC,

text: and

text: GIC

text: (Fan

text: Tang,

text: and

text: observed

text: that

text: the

text: EBIC

text: tends

text: to

text: work

text: the

text: best

text: among

text: most

text: of

text: the

text: simulation

text: settings

text: that

text: we

text: con-

text: sidered.

text: For

text: easy

text: presentation,

text: we

text: report

text: only

text: the

text: results

text: for

text: EBIC.

text: S

text: T

text: |S|

text: |T

text: Let

text: j

text: β

text: and

text: (j,

text: k)

text: β

text: with

text: cardinality

text: s

text: and

text: t

text: j

text: j,k

text: For

text: each

text: example,

text: we

text: run

text: M

text: Monte-Carlo

text: simulations

text: for

text: each

text: method

text: and

text: make

text: S

text: T

text: a

text: comparison.

text: For

text: the

text: m-th

text: simulation,

text: denote

text: the

text: estimated

text: subsets

text: as

text: (m)

text: and

text: (m),

text: the

text: estimated

text: coeﬃcient

text: vector

text: as

text: β(m),

text: the

text: main

text: eﬀects

text: and

text: interaction

text: eﬀects

text: as

text: β(m)

text: and

text: j

text: β(m).

text: We

text: evaluate

text: the

text: performance

text: on

text: variable

text: selection

text: and

text: model

text: estimation

text: based

text: on

text: the

text: j,k

text: following

text: criteria.

text: S

text: S

text: Main

text: eﬀects

text: coverage

text: percentage

text: (main.cov):

text: M

text: M

text: I(

text: T

text: T

text: Interaction

text: eﬀects

text: coverage

text: percentage

text: (inter.cov):

text: M

text: M

text: I(

text: S

text: S

text: Main

text: eﬀects

text: exact

text: selection

text: percentage

text: (main.exact):

text: M

text: M

text: I(

text: T

text: T

text: Interaction

text: eﬀects

text: exact

text: (inter.exact):

text: M

text: M

text: I(

text: (m)).

text: |S

text: |T

text: Model

text: size

text: (size):

text: M

text: M

text: (m)

text: (m)

text: Root

text: mean

text: squared

text: error

text: (RMSE):

text: M

text: M

text: p

text: (βˆ(m)

text: β

text: p

text: p

text: (βˆ(m)

text: j

text: j

text: k=j

text: j,k

text: β

text: j,k

text: N

text: Example

text: Set

text: (n,

text: p,

text: s,

text: t)

text: Generate

text: the

text: covariates

text: x

text: n

text: i.i.d.

text: Σ)

text: i

text: S

text: with

text: Σ

text: j

text: k

text: and

text: generate

text: the

text: response

text: y

text: by

text: model

text: with

text: the

text: jk


text: when p = o(n) (Choi et al., 2010). However, when p is much larger, these methods are not × feasible since their implementation requires storing and manipulating the entire O(p2) n design matrix and solving complex constrained optimization problems. The memory and computational cost can be extremely high and prohibitive. In this paper, we study regularization methods on model selection and estimation for QR and generalized quadratic regression (GQR) models under the marginality principle. (cid:29) The main focus is the case p n, which is a bottleneck for the existing regularization methods. We study theoretical properties of a two-stage regularization method based on the LASSO and propose a new eﬃcient algorithm, RAMP, which produces a hierarchy- preserving solution path. In contrast to existing regularization methods, these procedures × avoid storing O(p2) n design matrix and sidestep complex constraints and penalties, making them feasible to analyze data with many variables. In particular, our R package RAMP runs well on a desktop for data with n = 400 and p = 104 and it takes less than 30 seconds (with CPU 3.4 GHz Intel Core i7 and 32GB memory) to ﬁt the QR model and get the whole solution path. The main contribution of this paper is threefold. First, we establish a variable selection consistency result of the two-stage LASSO procedure for QR and oﬀer new insights on stage- wise selection methods. To our best knowledge, this is the ﬁrst selection consistency result for high dimensional QR. Second, the proposed algorithms are computationally eﬃcient and will make a valuable contribution to interaction selection tools in practice. Third, our methods are extended to interaction selection in GQR models, which are rarely studied in literature. (cid:62) × We deﬁne notations used in the paper. Let X = (x , ..., x ) be the n p design matrix 1 n (cid:62) of main eﬀects and y = (y , ..., y ) be the n-dimensional response vector. The linear term 1 n M { } I { ≤ ≤ ≤ } index set is = 1, 2, ..., p , and the order-2 index set is = (j, k) : 1 j k p . (cid:62) (cid:62) (cid:62) (cid:62) The regression coeﬃcient vector β = (β , βM, βI ) , where βM = (β , ..., β ) and βI = 0 1 p (cid:62) A ⊂ M A (β , β , ..., β ) . For a subset , use βA for the subvector of βM indexed in , and 1,1 1,2 p,p A XA for the submatrix of X whose columns are indexed in . In particular, X is the jth j column of X. We treat the subscripts (j, k) and (k, j) as identical, i.e., β = β . Let c , c , j,k k,j 1 2 (cid:113) ... and C (cid:80), C , ... be positive co(cid:80)nstants which are independent of the sample size(cid:80)n. They are 1 2 (cid:62) locally deﬁned and their values may vary in diﬀerent context. For a vector v = (v , ..., v ) , 1 p (cid:107) (cid:107) (cid:107) (cid:107) | | (cid:107) (cid:107) | | v = p v2 and v = p v . For a matrix A, deﬁne A ∞ = max A and j=1 j 1 j=1 j i j ij 3 ##LTLine##

text: when

text: p

text: o(n)

text: (Choi

text: et

text: al.,

text: However,

text: when

text: p

text: is

text: much

text: larger,

text: these

text: methods

text: are

text: not

text: feasible

text: since

text: their

text: implementation

text: requires

text: storing

text: and

text: manipulating

text: the

text: entire

text: n

text: design

text: matrix

text: and

text: solving

text: complex

text: constrained

text: optimization

text: problems.

text: The

text: memory

text: and

text: computational

text: cost

text: can

text: be

text: extremely

text: high

text: and

text: prohibitive.

text: In

text: this

text: paper,

text: we

text: study

text: regularization

text: methods

text: on

text: model

text: selection

text: and

text: estimation

text: for

text: QR

text: and

text: generalized

text: quadratic

text: regression

text: (GQR)

text: models

text: under

text: the

text: marginality

text: principle.

text: The

text: main

text: focus

text: is

text: the

text: case

text: p

text: n,

text: which

text: is

text: a

text: bottleneck

text: for

text: the

text: existing

text: regularization

text: methods.

text: We

text: study

text: theoretical

text: properties

text: of

text: a

text: two-stage

text: regularization

text: method

text: based

text: on

text: the

text: LASSO

text: and

text: propose

text: a

text: new

text: eﬃcient

text: algorithm,

text: RAMP,

text: which

text: produces

text: a

text: hierarchy-

text: preserving

text: solution

text: path.

text: In

text: contrast

text: to

text: existing

text: regularization

text: methods,

text: these

text: procedures

text: avoid

text: storing

text: n

text: design

text: matrix

text: and

text: sidestep

text: complex

text: constraints

text: and

text: penalties,

text: making

text: them

text: feasible

text: to

text: analyze

text: data

text: with

text: many

text: variables.

text: In

text: particular,

text: our

text: R

text: package

text: RAMP

text: runs

text: well

text: on

text: a

text: desktop

text: for

text: data

text: with

text: n

text: and

text: p

text: and

text: it

text: takes

text: less

text: than

text: seconds

text: (with

text: CPU

text: GHz

text: Intel

text: Core

text: and

text: memory)

text: to

text: ﬁt

text: the

text: QR

text: model

text: and

text: get

text: the

text: whole

text: solution

text: path.

text: The

text: main

text: contribution

text: of

text: this

text: paper

text: is

text: threefold.

text: First,

text: we

text: establish

text: a

text: variable

text: selection

text: consistency

text: result

text: of

text: the

text: two-stage

text: LASSO

text: procedure

text: for

text: QR

text: and

text: oﬀer

text: new

text: insights

text: on

text: stage-

text: wise

text: selection

text: methods.

text: To

text: our

text: best

text: knowledge,

text: this

text: is

text: the

text: ﬁrst

text: selection

text: consistency

text: result

text: for

text: high

text: dimensional

text: QR.

text: Second,

text: the

text: proposed

text: algorithms

text: are

text: computationally

text: eﬃcient

text: and

text: will

text: make

text: a

text: valuable

text: contribution

text: to

text: interaction

text: selection

text: tools

text: in

text: practice.

text: Third,

text: our

text: methods

text: are

text: extended

text: to

text: interaction

text: selection

text: in

text: GQR

text: models,

text: which

text: are

text: rarely

text: studied

text: in

text: literature.

text: We

text: deﬁne

text: notations

text: used

text: in

text: the

text: paper.

text: Let

text: X

text: (x

text: x

text: be

text: the

text: n

text: p

text: design

text: matrix

text: n

text: of

text: main

text: eﬀects

text: and

text: y

text: (y

text: y

text: be

text: the

text: n-dimensional

text: response

text: vector.

text: The

text: linear

text: term

text: n

text: M

text: I

text: index

text: set

text: is

text: p

text: and

text: the

text: index

text: set

text: is

text: (j,

text: k)

text: j

text: k

text: p

text: The

text: regression

text: coeﬃcient

text: vector

text: β

text: (β

text: βM,

text: βI

text: where

text: βM

text: (β

text: β

text: and

text: βI

text: p

text: A

text: M

text: A

text: (β

text: β

text: β

text: For

text: a

text: subset

text: use

text: βA

text: for

text: the

text: subvector

text: of

text: βM

text: indexed

text: in

text: and

text: p,p

text: A

text: XA

text: for

text: the

text: submatrix

text: of

text: X

text: whose

text: columns

text: are

text: indexed

text: in

text: In

text: particular,

text: X

text: is

text: the

text: jth

text: j

text: column

text: of

text: X.

text: We

text: treat

text: the

text: subscripts

text: (j,

text: k)

text: and

text: (k,

text: j)

text: as

text: identical,

text: i.e.,

text: β

text: β

text: Let

text: c

text: c

text: j,k

text: k,j

text: and

text: C

text: C

text: be

text: positive

text: which

text: are

text: independent

text: of

text: the

text: sample

text: They

text: are

text: locally

text: deﬁned

text: and

text: their

text: values

text: may

text: vary

text: in

text: diﬀerent

text: context.

text: For

text: a

text: vector

text: v

text: (v

text: v

text: p

text: v

text: p

text: and

text: v

text: p

text: v

text: For

text: a

text: matrix

text: A,

text: deﬁne

text: A

text: max

text: A

text: and

text: j

text: j

text: i

text: j

text: ij

text: ##LTLine##


text: An R package RAMP has been developed and is available from the CRAN website. 7 Appendix The main results are shown in Appendix A, and a related lemma is put in Appendix B. 7.1 Appendix A Proof of Theorem 1. We will apply the primal-dual witness (PDW) method and use (W 1), (W 2), etc. to denote the formula (1), (2),... in Wainwright (2009). Recall in our paper, the n-vector ω is the imaginary noise at Stage 1, which is the sum of the Gaussian (cid:62) (cid:62) (cid:62) noise ε and the interaction eﬀects (u βI, ..., u βI) , and hence it is not independent of the 1 n design matrix X. Part I: Verifying strict dual feasibility. The goal is to show that, with overwhelming probability, under condition (6), inequality | | ∈ S ∈ S Z < 1 holds for each j c, where Z is deﬁned in (W10). For every j c, conditional j j on XS, (W37) gives a decomposition Z = A + B where j j j (cid:26) (cid:18) (cid:19)(cid:27) (cid:62) (cid:62) − ω A = E XS(XS XS) 1zˇS + Π ⊥ j j XS λ n n − B = Σ S(ΣSS) 1zˇS, j j (cid:62) (cid:62) − − (cid:62) ∈ R ∼ N where E = X Σ S(ΣSS) 1XS n with E (0, [ΣS |S] ). j j j ij c jj Condition (C1) implies | | ≤ − max B 1 γ. ∈S j j c 23 ##LTLine##

text: An

text: R

text: package

text: RAMP

text: has

text: been

text: developed

text: and

text: is

text: available

text: from

text: the

text: CRAN

text: website.

text: Appendix

text: The

text: main

text: results

text: are

text: shown

text: in

text: Appendix

text: A,

text: and

text: a

text: related

text: lemma

text: is

text: put

text: in

text: Appendix

text: B.

text: Appendix

text: A

text: Proof

text: of

text: Theorem

text: We

text: will

text: apply

text: the

text: primal-dual

text: witness

text: (PDW)

text: method

text: and

text: use

text: (W

text: (W

text: etc.

text: to

text: denote

text: the

text: formula

text: in

text: Wainwright

text: Recall

text: in

text: our

text: paper,

text: the

text: n-vector

text: ω

text: is

text: the

text: imaginary

text: noise

text: at

text: Stage

text: which

text: is

text: the

text: sum

text: of

text: the

text: Gaussian

text: noise

text: ε

text: and

text: the

text: interaction

text: eﬀects

text: (u

text: βI,

text: u

text: βI)

text: and

text: hence

text: it

text: is

text: not

text: independent

text: of

text: the

text: n

text: design

text: matrix

text: X.

text: Part

text: I:

text: Verifying

text: strict

text: dual

text: feasibility.

text: The

text: goal

text: is

text: to

text: show

text: that,

text: with

text: overwhelming

text: probability,

text: under

text: condition

text: inequality

text: S

text: S

text: Z

text: holds

text: for

text: each

text: j

text: c,

text: where

text: Z

text: is

text: deﬁned

text: in

text: For

text: every

text: j

text: c,

text: conditional

text: j

text: j

text: on

text: XS,

text: gives

text: a

text: decomposition

text: Z

text: A

text: B

text: where

text: j

text: j

text: j

text: ω

text: A

text: E

text: XS(XS

text: XS)

text: Π

text: j

text: j

text: XS

text: λ

text: n

text: n

text: B

text: Σ

text: S(ΣSS)

text: j

text: j

text: R

text: N

text: where

text: E

text: X

text: Σ

text: S(ΣSS)

text: n

text: with

text: E

text: [ΣS

text: |S]

text: j

text: j

text: j

text: ij

text: c

text: jj

text: Condition

text: implies

text: max

text: B

text: γ.

text: ∈S

text: j

text: j

text: c

text: ##LTLine##


text: | | It is easy to check that (6) can guarantee that max ∈S Z < 1 holds with probability at j c j − − { − } least 1 c exp( c min s, log(p s), n1 ). 1 2 2 Now we show the suﬃciency of the alternative condition (8). In particular,we show (5) and (8) imply (6), which is equivalent to n − ρ (ΣS |S) 2(σ2 + τ2)C u c min > 2s log(p s) (1 + ). 1 + δ C γ2 λ2s min n Plugging in (5), we have n − ρ (ΣS |S) − ρ (ΣS |S) 2(σ2 + τ2)C u c u c min > 2s log(p s) + 2s log(p s) 1 + δ C γ2 C γ2 λ2s min − min n − ρ (ΣS |S) n log(p s) u c = 2s log(p s) + . (13) C γ2 φ log p min p Following the same argument after (W40) in Wainwright (2009), (13) is implied by (8) ≥ for φ 2. p Part II: Sign consistency. In order to show sign consistency, we need to show that (W13) holds. That is ∈ S sign(β + ∆ ) = sign(β ), for all j , (14) j j j where (cid:18) (cid:19) (cid:20) (cid:21) (cid:62) − (cid:62) XS XS 1 1 (cid:62) − ∆ = e XS ω λ sign(βS) . j j n n n From deﬁnition, we have 25 ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: It

text: is

text: easy

text: to

text: check

text: that

text: can

text: guarantee

text: that

text: max

text: ∈S

text: Z

text: holds

text: with

text: probability

text: at

text: j

text: c

text: j

text: least

text: c

text: exp(

text: c

text: min

text: s,

text: log(p

text: s),

text: Now

text: we

text: show

text: the

text: suﬃciency

text: of

text: the

text: alternative

text: condition

text: In

text: particular,we

text: show

text: and

text: imply

text: which

text: is

text: equivalent

text: to

text: n

text: ρ

text: (ΣS

text: |S)

text: u

text: c

text: min

text: log(p

text: s)

text: δ

text: C

text: γ2

text: min

text: n

text: Plugging

text: in

text: we

text: have

text: n

text: ρ

text: (ΣS

text: |S)

text: ρ

text: (ΣS

text: |S)

text: u

text: c

text: u

text: c

text: min

text: log(p

text: s)

text: log(p

text: s)

text: δ

text: C

text: γ2

text: C

text: γ2

text: min

text: min

text: n

text: ρ

text: (ΣS

text: |S)

text: n

text: log(p

text: s)

text: u

text: c

text: log(p

text: s)

text: C

text: γ2

text: φ

text: log

text: p

text: min

text: p

text: Following

text: the

text: same

text: argument

text: after

text: in

text: Wainwright

text: is

text: implied

text: by

text: for

text: φ

text: p

text: Part

text: II:

text: Sign

text: consistency.

text: In

text: order

text: to

text: show

text: sign

text: consistency,

text: we

text: need

text: to

text: show

text: that

text: holds.

text: That

text: is

text: S

text: sign(β

text: sign(β

text: for

text: all

text: j

text: j

text: j

text: j

text: where

text: XS

text: XS

text: e

text: XS

text: ω

text: λ

text: sign(βS)

text: j

text: j

text: n

text: n

text: n

text: From

text: deﬁnition,

text: we

text: have

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: | | ≤ ≤ max ∆ F + F F + (F + F ), ∈S j 1 2 1 2,1 2,2 j where (cid:13) (cid:13) (cid:18) (cid:19) (cid:13) (cid:13) (cid:13) (cid:13) (cid:13) (cid:13) (cid:13) (cid:62) − (cid:13) (cid:13) XS XS 1 (cid:13) (cid:18) (cid:19) F = λ(cid:13) sign((cid:13)βS) 1 (cid:13)n n (cid:13) (cid:13) (cid:13) ∞ (cid:13) (cid:62) − (cid:13) (cid:13) XS XS 1 1 (cid:62) (cid:13) (cid:18) (cid:19) F = (cid:13) XS ω(cid:13) 2 (cid:13) n n (cid:13) (cid:13) (cid:13) ∞ (cid:13) (cid:62) − (cid:13) (cid:13) XS XS 1 1 (cid:62) (cid:13) (cid:18) (cid:19) F = (cid:13) XS ε (cid:13) 2,1 (cid:13) n n (cid:13) (cid:13) (cid:13)∞ (cid:13) (cid:62) − (cid:13) XS XS 1 1 (cid:62) F = XS yI 2,2 n n ∞ (W41) and a correction version of (W42) give upper bounds of tail probability of F and 1 F , respectively. That is 2,1 (cid:18) (cid:19) (cid:13) (cid:13) (cid:13) (cid:13) (cid:13) (cid:13) −1 2 ≤ − { − } P F > c λ ΣSS2 4 exp( c min s, log(p s) ), (15) 1 3 n ∞ 2 (cid:32) (cid:115) (cid:33) ≥ σ2s ≤ − P F 20 4 exp( c s). (16) 2,1 1 C n min (cid:32)(cid:13) (cid:13) (cid:33) (cid:18) (cid:19) (cid:13) (cid:13) Now we work on the ad(cid:13)dition term F (cid:13). By (W60), (cid:13) 2(cid:13),2 (cid:13) (cid:13) − 1 (cid:62) 1 ≥ 9 ≤ − P XS XS 2 exp( n/2). n C min 2 26 ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: max

text: F

text: F

text: F

text: (F

text: F

text: ∈S

text: j

text: j

text: where

text: XS

text: XS

text: F

text: n

text: XS

text: XS

text: F

text: XS

text: n

text: n

text: XS

text: XS

text: F

text: XS

text: ε

text: n

text: n

text: XS

text: XS

text: F

text: XS

text: yI

text: n

text: n

text: and

text: a

text: correction

text: version

text: of

text: give

text: upper

text: bounds

text: of

text: tail

text: probability

text: of

text: F

text: and

text: F

text: respectively.

text: That

text: is

text: −1

text: P

text: F

text: c

text: λ

text: exp(

text: c

text: min

text: s,

text: log(p

text: s)

text: n

text: P

text: F

text: exp(

text: c

text: s).

text: C

text: n

text: min

text: Now

text: we

text: work

text: on

text: the

text: term

text: F

text: By

text: P

text: XS

text: XS

text: exp(

text: n

text: C

text: min

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: Two-stage regularization methods for interaction selection have been considered in Efron et al. (2004); Wu et al. (2009), among others. However, their theoretical properties are not clearly understood. In the following, we ﬁrst illustrate the general two-stage procedure for interaction selection. (cid:98) Two-stage Regularization Method: A { (cid:54) } ˆ Stage 1: Solve (2). Denote the selected model by = j : β = 0, j = 1, . . . , p . j Stage 2: Solve (cid:88) (cid:98) (cid:98) (cid:98) (cid:98) 1 (cid:107) − − − ◦ (cid:107) βˆ = argmin 2n y 1β0 XAβA XA2βA◦2 2 + (cid:98) Jλ(βα), β ∈A◦ α 2 (cid:98) (cid:98) At Stage 1, only main eﬀects are considered for selection, with all the order-2 terms being (cid:98) A A left out of the model. Denote the selected set by . At Stage 2, we expand by including A all the two-way interactions of those main eﬀects within and ﬁt the new model. To keep (cid:98) · the hierarchical structure, we do not penalize main eﬀects at Stage 2, i.e., set J ( ) = 0 for λ ∈ A j . In order to keep the hierarchy, it is also possible to use other methods (Zhao et al., 2009; Yuan et al., 2009; Choi et al., 2010; Bien et al., 2013) at Stage 2. One main advantage of this two-stage regularization procedure is its simple implementa- tion. Existing R packages lars and glmnet can be directly used to carry out the procedure. Stage 1 serves as a dimension reduction step prior to Stage 2, so the two-stage method avoids estimating O(p2) parameters altogether, making the procedure feasible for very large p. In spite of its computational advantages, theoretical properties of two-stage regulariza- tion methods are seldom studied in literature. A commonly raised concern is whether the important main eﬀects can be consistently identiﬁed at Stage 1, when all order-2 terms are left out of the model on purpose. Next, we focus on the two-stage LASSO method and inves- tigate its selection behavior at Stage 1. In particular, we establish the main-eﬀect selection consistency result of the two-stage LASSO for QR under some regularity conditions. The LASSO is a special case of (2) by using the (cid:96) penalty 1 5 ##LTLine##

text: Two-stage

text: regularization

text: methods

text: for

text: interaction

text: selection

text: have

text: been

text: considered

text: in

text: Efron

text: et

text: al.

text: Wu

text: et

text: al.

text: among

text: others.

text: However,

text: their

text: theoretical

text: properties

text: are

text: not

text: clearly

text: understood.

text: In

text: the

text: following,

text: we

text: ﬁrst

text: illustrate

text: the

text: general

text: two-stage

text: procedure

text: for

text: interaction

text: selection.

text: Two-stage

text: Regularization

text: Method:

text: A

text: ˆ

text: Stage

text: Solve

text: Denote

text: the

text: selected

text: model

text: by

text: j

text: β

text: j

text: p

text: j

text: Stage

text: Solve

text: βˆ

text: argmin

text: y

text: XAβA

text: Jλ(βα),

text: β

text: ∈A◦

text: α

text: At

text: Stage

text: only

text: main

text: eﬀects

text: are

text: considered

text: for

text: selection,

text: with

text: all

text: the

text: terms

text: being

text: A

text: A

text: left

text: out

text: of

text: the

text: model.

text: Denote

text: the

text: selected

text: set

text: by

text: At

text: Stage

text: we

text: expand

text: by

text: including

text: A

text: all

text: the

text: two-way

text: interactions

text: of

text: those

text: main

text: eﬀects

text: within

text: and

text: ﬁt

text: the

text: new

text: model.

text: To

text: keep

text: the

text: hierarchical

text: structure,

text: we

text: do

text: not

text: penalize

text: main

text: eﬀects

text: at

text: Stage

text: i.e.,

text: set

text: J

text: for

text: λ

text: A

text: j

text: In

text: order

text: to

text: keep

text: the

text: hierarchy,

text: it

text: is

text: also

text: possible

text: to

text: use

text: other

text: methods

text: (Zhao

text: et

text: al.,

text: Yuan

text: et

text: al.,

text: Choi

text: et

text: al.,

text: Bien

text: et

text: al.,

text: at

text: Stage

text: One

text: main

text: advantage

text: of

text: this

text: two-stage

text: regularization

text: procedure

text: is

text: its

text: simple

text: implementa-

text: tion.

text: Existing

text: R

text: packages

text: lars

text: and

text: glmnet

text: can

text: be

text: directly

text: used

text: to

text: carry

text: out

text: the

text: procedure.

text: Stage

text: serves

text: as

text: a

text: dimension

text: reduction

text: step

text: prior

text: to

text: Stage

text: so

text: the

text: two-stage

text: method

text: avoids

text: estimating

text: parameters

text: altogether,

text: making

text: the

text: procedure

text: feasible

text: for

text: very

text: large

text: p.

text: In

text: spite

text: of

text: its

text: computational

text: advantages,

text: theoretical

text: properties

text: of

text: two-stage

text: regulariza-

text: tion

text: methods

text: are

text: seldom

text: studied

text: in

text: literature.

text: A

text: commonly

text: raised

text: concern

text: is

text: whether

text: the

text: important

text: main

text: eﬀects

text: can

text: be

text: consistently

text: identiﬁed

text: at

text: Stage

text: when

text: all

text: terms

text: are

text: left

text: out

text: of

text: the

text: model

text: on

text: purpose.

text: Next,

text: we

text: focus

text: on

text: the

text: two-stage

text: LASSO

text: method

text: and

text: inves-

text: tigate

text: its

text: selection

text: behavior

text: at

text: Stage

text: In

text: particular,

text: we

text: establish

text: the

text: main-eﬀect

text: selection

text: consistency

text: result

text: of

text: the

text: two-stage

text: LASSO

text: for

text: QR

text: under

text: some

text: regularity

text: conditions.

text: The

text: LASSO

text: is

text: a

text: special

text: case

text: of

text: by

text: using

text: the

text: penalty

text: ##LTLine##


text: | | Then if β = min ∈S β > g(λ ), then sign(βˆ ) = sign(βM). min j j n L Furthermore, given (5), an alternative condition to (6) making the above results hold is (cid:48) n 1 + δ ρ (ΣS |S) − > − u c (8) 2s log(p s) 1 1 C γ2 φp min (cid:48) for some δ > 0. Remark 1. Conditions (C1)-(C2) are commonly used to show model selection consistency of the LASSO estimator in the literature. Conditions (6) and (7) are key requirements on dimensionality and minimal signal strength β , respectively. The normality assumption min is used here to facilitate proof and comparison to existing results in linear regression. In the supplementary material, we establish Theorem 2 which extends the consistency result to the non-Gaussian designs. Other possible extensions of theoretical results are discussed in Section . Remark 2. The result in Theorem 1 generalizes Theorem 3 in Wainwright (2009) that is established in the context of linear regression. Theorem 1 implies that the two-stage LASSO can identify important main eﬀects at Stage 1. The validity of the two-stage LASSO is T ⊂ S◦ then guaranteed as the index set of important interactions 2. That is, all important interaction eﬀects can be included at Stage 2. Given the result of Theorem 1, the interaction selection consistency result of Stage 2 can be obtained under some mild conditions on the ◦ matrix XS2, since the data dimensionality has been greatly reduced. One can also apply existing methods, e.g., Choi et al. (2010) at Stage 2, for which the selection consistency has been established. 8 ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: Then

text: if

text: β

text: min

text: ∈S

text: β

text: g(λ

text: then

text: sign(βˆ

text: sign(βM).

text: min

text: j

text: j

text: n

text: L

text: Furthermore,

text: given

text: an

text: alternative

text: condition

text: to

text: making

text: the

text: above

text: results

text: hold

text: is

text: n

text: δ

text: ρ

text: (ΣS

text: |S)

text: u

text: c

text: log(p

text: s)

text: C

text: γ2

text: φp

text: min

text: for

text: some

text: δ

text: Remark

text: Conditions

text: are

text: commonly

text: used

text: to

text: show

text: model

text: selection

text: consistency

text: of

text: the

text: LASSO

text: estimator

text: in

text: the

text: literature.

text: Conditions

text: and

text: are

text: key

text: requirements

text: on

text: dimensionality

text: and

text: minimal

text: signal

text: strength

text: β

text: respectively.

text: The

text: normality

text: assumption

text: min

text: is

text: used

text: here

text: to

text: facilitate

text: proof

text: and

text: comparison

text: to

text: existing

text: results

text: in

text: linear

text: regression.

text: In

text: the

text: supplementary

text: material,

text: we

text: establish

text: Theorem

text: which

text: extends

text: the

text: consistency

text: result

text: to

text: the

text: non-Gaussian

text: designs.

text: Other

text: possible

text: extensions

text: of

text: theoretical

text: results

text: are

text: discussed

text: in

text: Section

text: Remark

text: The

text: result

text: in

text: Theorem

text: generalizes

text: Theorem

text: in

text: Wainwright

text: that

text: is

text: established

text: in

text: the

text: context

text: of

text: linear

text: regression.

text: Theorem

text: implies

text: that

text: the

text: two-stage

text: LASSO

text: can

text: identify

text: important

text: main

text: eﬀects

text: at

text: Stage

text: The

text: validity

text: of

text: the

text: two-stage

text: LASSO

text: is

text: T

text: S◦

text: then

text: guaranteed

text: as

text: the

text: index

text: set

text: of

text: important

text: interactions

text: That

text: is,

text: all

text: important

text: interaction

text: eﬀects

text: can

text: be

text: included

text: at

text: Stage

text: Given

text: the

text: result

text: of

text: Theorem

text: the

text: interaction

text: selection

text: consistency

text: result

text: of

text: Stage

text: can

text: be

text: obtained

text: under

text: some

text: mild

text: conditions

text: on

text: the

text: matrix

text: since

text: the

text: data

text: dimensionality

text: has

text: been

text: greatly

text: reduced.

text: One

text: can

text: also

text: apply

text: existing

text: methods,

text: e.g.,

text: Choi

text: et

text: al.

text: at

text: Stage

text: for

text: which

text: the

text: selection

text: consistency

text: has

text: been

text: established.

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: )zH(f )zH(f )zH(f Four-winged ﬂapping ﬂyer 5 35 U(m s−1) 35 Pi(W) 9 35 η 0.06 8 30 2 30 30 7 0.05 6 25 25 25 0.04 1.5 5 0.03 20 20 4 20 1 3 0.02 15 15 15 2 0.01 10 0.5 10 1 10 0 (cid:47)/4 (cid:47)/2 3(cid:47)/4 (cid:47) 0 (cid:47)/4 (cid:47)/2 3(cid:47)/4 (cid:47) 0 (cid:47)/4 (cid:47)/2 3(cid:47)/4 (cid:47) ϕ ϕ ϕ Fig. 3 Cruising speed, consumed power and efﬁciency (see text) as a function of the forewing- hindwing phase lag and the ﬂapping frequency. Only the ﬁrst half o the phase-lag ϕ range where the forewing is leading was examined here. quency is related to the elastic properties of the wings. The second plot in Fig. 3 shows the consumed power P in the same parameter space. Here the main ob- i servation is that, while not surprisingly consumed power increases monotonically with increasing ﬂapping frequency, the effect of the phase lag on energy expendi- ture shown previously in Fig. 2 is clearly present regardless the ﬂapping frequency. Because of this effect, for different phase lags the ranges of frequency explored changed, giving for instance a maximum frequency for ϕ = 0 of around 30Hz while ∈ at ϕ [π/2,3π/4] the frequency could reach 35Hz. We use those two measure- ments to deﬁne the following expression of efﬁciency, considering that the aerody- namic thrust power is proportional to U3 (velocity times thrust force, the latter being ∼ U2): 1ρSU3 η = 2 (2) P i where S is the effective wing surface. Other deﬁnitions of efﬁciency using purely dynamical parameters (e.g. Kang et al., 2011) should give equivalent results to the expression 2 chosen here in terms of the measured consumed power P. It can be seen i that the optimum in terms of efﬁciency is shifted toward larger phase lags (around ≈ ϕ π/2) than the optimum in terms of maximum cruising speed. (ii) Wing kinematics. In addition to the performance measurements, the wing mo- tion was recorded using high-speed video in a ﬁxed frame. That is, with the mechan- ical insect not mounted on the merry-go-round but on a ﬁxed base. Figures 4 and 5 show typical time series for ϕ = 0 and 1.3π, respectively. A dark screen was used to mask the right side wings and have a clearer view of the kinematics. The main fea- ture that can be seen in these time series of snapshots is the large deformation of the wings during the ﬂapping cycle. Indeed, the wings bend over a length scale that is of ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTLine## ##LTLine##

text: )zH(f

text: )zH(f

text: )zH(f

text: Four-winged

text: ﬂapping

text: ﬂyer

text: U(m

text: Pi(W)

text: η

text: ϕ

text: ϕ

text: ϕ

text: Fig.

text: Cruising

text: speed,

text: consumed

text: power

text: and

text: efﬁciency

text: (see

text: text)

text: as

text: a

text: function

text: of

text: the

text: forewing-

text: hindwing

text: phase

text: lag

text: and

text: the

text: ﬂapping

text: frequency.

text: Only

text: the

text: ﬁrst

text: half

text: o

text: the

text: phase-lag

text: ϕ

text: range

text: where

text: the

text: forewing

text: is

text: leading

text: was

text: examined

text: here.

text: quency

text: is

text: related

text: to

text: the

text: elastic

text: properties

text: of

text: the

text: wings.

text: The

text: second

text: plot

text: in

text: Fig.

text: shows

text: the

text: consumed

text: power

text: P

text: in

text: the

text: same

text: parameter

text: space.

text: Here

text: the

text: main

text: ob-

text: i

text: servation

text: is

text: that,

text: while

text: not

text: surprisingly

text: consumed

text: power

text: increases

text: monotonically

text: with

text: increasing

text: ﬂapping

text: frequency,

text: the

text: effect

text: of

text: the

text: phase

text: lag

text: on

text: energy

text: expendi-

text: ture

text: shown

text: previously

text: in

text: Fig.

text: is

text: clearly

text: present

text: regardless

text: the

text: ﬂapping

text: frequency.

text: Because

text: of

text: this

text: effect,

text: for

text: different

text: phase

text: lags

text: the

text: ranges

text: of

text: frequency

text: explored

text: changed,

text: giving

text: for

text: instance

text: a

text: maximum

text: frequency

text: for

text: ϕ

text: of

text: around

text: while

text: at

text: ϕ

text: the

text: frequency

text: could

text: reach

text: We

text: use

text: those

text: two

text: measure-

text: ments

text: to

text: deﬁne

text: the

text: following

text: expression

text: of

text: efﬁciency,

text: considering

text: that

text: the

text: aerody-

text: namic

text: thrust

text: power

text: is

text: proportional

text: to

text: (velocity

text: times

text: thrust

text: force,

text: the

text: latter

text: being

text: η

text: P

text: i

text: where

text: S

text: is

text: the

text: effective

text: wing

text: surface.

text: Other

text: deﬁnitions

text: of

text: efﬁciency

text: using

text: purely

text: dynamical

text: parameters

text: (e.g.

text: Kang

text: et

text: al.,

text: should

text: give

text: equivalent

text: results

text: to

text: the

text: expression

text: chosen

text: here

text: in

text: terms

text: of

text: the

text: measured

text: consumed

text: power

text: P.

text: It

text: can

text: be

text: seen

text: i

text: that

text: the

text: optimum

text: in

text: terms

text: of

text: efﬁciency

text: is

text: shifted

text: toward

text: larger

text: phase

text: lags

text: (around

text: ϕ

text: than

text: the

text: optimum

text: in

text: terms

text: of

text: maximum

text: cruising

text: speed.

text: (ii)

text: Wing

text: kinematics.

text: In

text: addition

text: to

text: the

text: performance

text: measurements,

text: the

text: wing

text: mo-

text: tion

text: was

text: recorded

text: using

text: high-speed

text: video

text: in

text: a

text: ﬁxed

text: frame.

text: That

text: is,

text: with

text: the

text: mechan-

text: ical

text: insect

text: not

text: mounted

text: on

text: the

text: merry-go-round

text: but

text: on

text: a

text: ﬁxed

text: base.

text: Figures

text: and

text: show

text: typical

text: time

text: series

text: for

text: ϕ

text: and

text: respectively.

text: A

text: dark

text: screen

text: was

text: used

text: to

text: mask

text: the

text: right

text: side

text: wings

text: and

text: have

text: a

text: clearer

text: view

text: of

text: the

text: kinematics.

text: The

text: main

text: fea-

text: ture

text: that

text: can

text: be

text: seen

text: in

text: these

text: time

text: series

text: of

text: snapshots

text: is

text: the

text: large

text: deformation

text: of

text: the

text: wings

text: during

text: the

text: ﬂapping

text: cycle.

text: Indeed,

text: the

text: wings

text: bend

text: over

text: a

text: length

text: scale

text: that

text: is

text: of

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##


text: Four-winged ﬂapping ﬂyer 9 This will thus shift the location of the peak in the efﬁciency that corresponds to the aerodynamic power peak. In order to focus on the effect of the forewing-hindwing phase lag, a ﬁxed fre- quency can be chosen. In Fig. 2, for each of the two frequencies represented, the same trends are observed in the cruising speed and the consumed power and, as shown in the insets, the frequency dependence is well explained using the “elasto- inertial” dimensionless variables u and p deﬁned above. Now, with respect to the phase lag ϕ, two main points appear clearly: on the one hand, the fastest ﬂying per- formance is found around in-phase ﬂapping (i.e ϕ = 0), while around anti-phase ﬂapping (ϕ = π) this cruising speed is the lowest. We may note that the curve is not symmetric with respect to ϕ = 0, since it diminishes only slightly (one could consider a plateau) between ϕ = 0 and π/2, whereas it drops more rapidly when the hindwing leads (i.e. when going from ϕ = 0 = 2π towards ϕ = 3π/2). The con- sumed power curve on the contrary has two maxima (around ϕ = 0 and π) and two minima around (ϕ = π/2 and 3π/2). In-phase and anti-phase ﬂapping being the most expensive can be explained by a simple inertial argument since during these conﬁgurations the motor has to accommodate the acceleration/deceleration of both pairs of wings at the same time, contrary to the intermediate phase lags ϕ = π/2 and 3π/2 where the deceleration of one pair of wings occurs while the other pair is accelerating, hence redistributing the load on the motor. The shape of the cruising speed and consumed power curves determines the lo- cation of the maximum observed in the efﬁciency contours in Fig. 3 being around ϕ = π/2. A secondary maximum can be expected around ϕ = 3π/2 (a zone of the parameter space that was not fully explored in the present experiments). We can now compare the optimum phase-lags that lead to peaks of cruising speed and of ef- ﬁciency and comment on their physical origin: while the maximum cruising speed is observed for phase-lags between zero and π/4, the optimum phase-lags in terms of efﬁciency are around π/2. The former are ruled solely by the aerodynamics, where the performance of different kinematic patterns will have to be analysed considering for instance the interaction between hindwing and the vortex structures shed by the forewing. A wake capture process of this sort has been proposed by Kolomenskiy et al. (2013) as a possible explanation for the large propulsive force found at 0.75π in their 2D numerical simulations.1 Concerning the optimum phase lag in terms of efﬁciency requires on the other hand considering the power consumption, which is not only correlated to aerodynamics, but has a large contribution determined by solid inertia, as we have mentioned in reference to the power plot in Fig. 2b. The effect of the modulation of ϕ in terms of aerodynamics, as mentioned pre- viously, will be intrinsically related to the roles of the distance d between the two wing pairs and of the deformation kinematics. In this paper we have ﬁxed d and considered a single ﬂexibility in order to explore the possible roles of the wing de- formation in the propulsive performance. Considering our previous studies on the effect of ﬂexibility with a two wing ﬂyer (Thiria & Godoy-Diana, 2010; Ramana- narivo et al., 2011), where the trailing-edge-leading-edge phase lag γ was shown 1 Note that in Kolomenskiy et al. (2013) the phase lag is deﬁned with a negative sign with respect to ϕ used here so that 0.75π here corresponds to their 1.25π. ##LTLine##

text: Four-winged

text: ﬂapping

text: ﬂyer

text: This

text: will

text: thus

text: shift

text: the

text: location

text: of

text: the

text: peak

text: in

text: the

text: efﬁciency

text: that

text: corresponds

text: to

text: the

text: aerodynamic

text: power

text: peak.

text: In

text: order

text: to

text: focus

text: on

text: the

text: effect

text: of

text: the

text: forewing-hindwing

text: phase

text: lag,

text: a

text: ﬁxed

text: fre-

text: quency

text: can

text: be

text: chosen.

text: In

text: Fig.

text: for

text: each

text: of

text: the

text: two

text: frequencies

text: represented,

text: the

text: same

text: trends

text: are

text: observed

text: in

text: the

text: cruising

text: speed

text: and

text: the

text: consumed

text: power

text: and,

text: as

text: shown

text: in

text: the

text: insets,

text: the

text: frequency

text: dependence

text: is

text: well

text: explained

text: using

text: the

text: “elasto-

text: inertial”

text: dimensionless

text: variables

text: u

text: and

text: p

text: deﬁned

text: above.

text: Now,

text: with

text: respect

text: to

text: the

text: phase

text: lag

text: ϕ,

text: two

text: main

text: points

text: appear

text: clearly:

text: on

text: the

text: one

text: hand,

text: the

text: fastest

text: ﬂying

text: per-

text: formance

text: is

text: found

text: around

text: in-phase

text: ﬂapping

text: (i.e

text: ϕ

text: while

text: around

text: anti-phase

text: ﬂapping

text: (ϕ

text: π)

text: this

text: cruising

text: speed

text: is

text: the

text: lowest.

text: We

text: may

text: note

text: that

text: the

text: curve

text: is

text: not

text: symmetric

text: with

text: respect

text: to

text: ϕ

text: since

text: it

text: diminishes

text: only

text: slightly

text: (one

text: could

text: consider

text: a

text: plateau)

text: between

text: ϕ

text: and

text: whereas

text: it

text: drops

text: more

text: rapidly

text: when

text: the

text: hindwing

text: leads

text: (i.e.

text: when

text: going

text: from

text: ϕ

text: 2π

text: towards

text: ϕ

text: The

text: con-

text: sumed

text: power

text: curve

text: on

text: the

text: contrary

text: has

text: two

text: maxima

text: (around

text: ϕ

text: and

text: π)

text: and

text: two

text: minima

text: around

text: (ϕ

text: and

text: In-phase

text: and

text: anti-phase

text: ﬂapping

text: being

text: the

text: most

text: expensive

text: can

text: be

text: explained

text: by

text: a

text: simple

text: inertial

text: argument

text: since

text: during

text: these

text: conﬁgurations

text: the

text: motor

text: has

text: to

text: accommodate

text: the

text: acceleration/deceleration

text: of

text: both

text: pairs

text: of

text: wings

text: at

text: the

text: same

text: time,

text: contrary

text: to

text: the

text: intermediate

text: phase

text: lags

text: ϕ

text: and

text: where

text: the

text: deceleration

text: of

text: one

text: pair

text: of

text: wings

text: occurs

text: while

text: the

text: other

text: pair

text: is

text: accelerating,

text: hence

text: redistributing

text: the

text: load

text: on

text: the

text: motor.

text: The

text: shape

text: of

text: the

text: cruising

text: speed

text: and

text: consumed

text: power

text: curves

text: determines

text: the

text: lo-

text: cation

text: of

text: the

text: maximum

text: observed

text: in

text: the

text: efﬁciency

text: contours

text: in

text: Fig.

text: being

text: around

text: ϕ

text: A

text: secondary

text: maximum

text: can

text: be

text: expected

text: around

text: ϕ

text: (a

text: zone

text: of

text: the

text: parameter

text: space

text: that

text: was

text: not

text: fully

text: explored

text: in

text: the

text: present

text: experiments).

text: We

text: can

text: now

text: compare

text: the

text: optimum

text: phase-lags

text: that

text: lead

text: to

text: peaks

text: of

text: cruising

text: speed

text: and

text: of

text: ef-

text: ﬁciency

text: and

text: comment

text: on

text: their

text: physical

text: origin:

text: while

text: the

text: maximum

text: cruising

text: speed

text: is

text: observed

text: for

text: phase-lags

text: between

text: zero

text: and

text: the

text: optimum

text: phase-lags

text: in

text: terms

text: of

text: efﬁciency

text: are

text: around

text: The

text: former

text: are

text: ruled

text: solely

text: by

text: the

text: aerodynamics,

text: where

text: the

text: performance

text: of

text: different

text: kinematic

text: patterns

text: will

text: have

text: to

text: be

text: analysed

text: considering

text: for

text: instance

text: the

text: interaction

text: between

text: hindwing

text: and

text: the

text: vortex

text: structures

text: shed

text: by

text: the

text: forewing.

text: A

text: wake

text: capture

text: process

text: of

text: this

text: sort

text: has

text: been

text: proposed

text: by

text: Kolomenskiy

text: et

text: al.

text: as

text: a

text: possible

text: explanation

text: for

text: the

text: large

text: propulsive

text: force

text: found

text: at

text: in

text: their

text: numerical

text: Concerning

text: the

text: optimum

text: phase

text: lag

text: in

text: terms

text: of

text: efﬁciency

text: requires

text: on

text: the

text: other

text: hand

text: considering

text: the

text: power

text: consumption,

text: which

text: is

text: not

text: only

text: correlated

text: to

text: aerodynamics,

text: but

text: has

text: a

text: large

text: contribution

text: determined

text: by

text: solid

text: inertia,

text: as

text: we

text: have

text: mentioned

text: in

text: reference

text: to

text: the

text: power

text: plot

text: in

text: Fig.

text: The

text: effect

text: of

text: the

text: modulation

text: of

text: ϕ

text: in

text: terms

text: of

text: aerodynamics,

text: as

text: mentioned

text: pre-

text: viously,

text: will

text: be

text: intrinsically

text: related

text: to

text: the

text: roles

text: of

text: the

text: distance

text: d

text: between

text: the

text: two

text: wing

text: pairs

text: and

text: of

text: the

text: deformation

text: kinematics.

text: In

text: this

text: paper

text: we

text: have

text: ﬁxed

text: d

text: and

text: considered

text: a

text: single

text: ﬂexibility

text: in

text: order

text: to

text: explore

text: the

text: possible

text: roles

text: of

text: the

text: wing

text: de-

text: formation

text: in

text: the

text: propulsive

text: performance.

text: Considering

text: our

text: previous

text: studies

text: on

text: the

text: effect

text: of

text: ﬂexibility

text: with

text: a

text: two

text: wing

text: ﬂyer

text: (Thiria

text: Godoy-Diana,

text: Ramana-

text: narivo

text: et

text: al.,

text: where

text: the

text: trailing-edge-leading-edge

text: phase

text: lag

text: γ

text: was

text: shown

text: Note

text: that

text: in

text: Kolomenskiy

text: et

text: al.

text: the

text: phase

text: lag

text: is

text: deﬁned

text: with

text: a

text: negative

text: sign

text: with

text: respect

text: to

text: ϕ

text: used

text: here

text: so

text: that

text: here

text: corresponds

text: to

text: their

text: ##LTLine##


text: 10 Godoy-Diana et al. to be a crucial parameter to determine performance, we attempted here to monitor modiﬁcations in γ as a function of ϕ (see the analysis of wing kinematics in Figs. 6 and 7). We picked a range of ϕ (decreasing from 2π to 1.3π) where a clear de- terioration of the ﬂying performance is observed in ﬁgure 2 as the hindwing starts leading the forewing. The observations on the trailing-edge-leading-edge phase lag γ are however not conclusive. A slight decrease of γ is indeed observed while ϕ goes from 2π to 1.3π, which at these frequencies could explain a diminishing thrust performance, but the effect is too weak and the present results do not permit to give a thorough and quantitative conﬁrmation of the effect if it exists. 5 Conclusions We have shown that a four-winged ﬂapping ﬂyer in a cruising regime does present different optimal forewing-hindwing phase lags depending on whether one would want to tune the system for maximum cruising speed or minimum energy expenditure. These results are in accordance with previous studies in hovering con- ﬁgurations by Wang & Russell (2007); Usherwood & Lehmann (2008), hinting that the mechanisms described here should be robust elements to consider in any aero- dynamic model: one the one hand wing inertia is a major player in the power expen- diture, while on the other hand the thrust production is ruled by the aerodynamics around the ﬂexible wings. A full ﬂow ﬁeld reconstruction around the wings is certainly desirable to compare different points in the parameter space with different performances and clearly iden- tify the aerodynamic mechanisms at play. Experimentally this can be challenging, and numerical simulations such as a 3D version of the ﬂuid-structure simulations of Kolomenskiy et al. (2013) can be an interesting option to deﬁne robust aerodynamic models. Other issue that remains not fully understood is the effect of the forewing- hindwing separation d as a parameter independent of the forewing-hindwing phase lag which will have a different role depending on whether the system is hovering or cruising. This point will be of particular importance when going beyond the “steady” regimes of hovering or cruising and into the study of transient regimes. These bring indeed a vast set of open questions related to the unsteadiness —like for instance the multi-body dynamics of manoeuvring— and where the wings are to be analysed as part of a full system accelerating, performing sharp turns (e.g. Bergou et al., 2010) or taking off (e.g. Bimbard et al., 2013). Acknowledgements We thank all the people at the PMMH workshop for their help with the experimental setup, in particular D. Pradal who conceived and built most of its parts. We thank also X. Benoit-Gonin, M. Vilmay and G. Lemoult for the Lab- View interfacing, and S. Ramananarivo, V. Raspa, D. Kolomenskiy and G. Sped- ding for useful discussions. This work was supported by the French National Re- search Agency through project No. ANR-08-BLAN-0099 and by EADS Foundation through project ”Fluids and elasticity in biomimetic propulsion”. M. C. acknowl- edges ﬁnancial support from CONACyT and UNAM, Me´xico. P. J. acknowledges support from the Charpak Internship Program.

text: Godoy-Diana

text: et

text: al.

text: to

text: be

text: a

text: crucial

text: parameter

text: to

text: determine

text: performance,

text: we

text: attempted

text: here

text: to

text: monitor

text: modiﬁcations

text: in

text: γ

text: as

text: a

text: function

text: of

text: ϕ

text: (see

text: the

text: analysis

text: of

text: wing

text: kinematics

text: in

text: Figs.

text: and

text: We

text: picked

text: a

text: range

text: of

text: ϕ

text: (decreasing

text: from

text: 2π

text: to

text: where

text: a

text: clear

text: de-

text: terioration

text: of

text: the

text: ﬂying

text: performance

text: is

text: observed

text: in

text: ﬁgure

text: as

text: the

text: hindwing

text: starts

text: leading

text: the

text: forewing.

text: The

text: observations

text: on

text: the

text: trailing-edge-leading-edge

text: phase

text: lag

text: γ

text: are

text: however

text: not

text: conclusive.

text: A

text: slight

text: decrease

text: of

text: γ

text: is

text: indeed

text: observed

text: while

text: ϕ

text: goes

text: from

text: 2π

text: to

text: which

text: at

text: these

text: frequencies

text: could

text: explain

text: a

text: diminishing

text: thrust

text: performance,

text: but

text: the

text: effect

text: is

text: too

text: weak

text: and

text: the

text: present

text: results

text: do

text: not

text: permit

text: to

text: give

text: a

text: thorough

text: and

text: quantitative

text: conﬁrmation

text: of

text: the

text: effect

text: if

text: it

text: exists.

text: Conclusions

text: We

text: have

text: shown

text: that

text: a

text: four-winged

text: ﬂapping

text: ﬂyer

text: in

text: a

text: cruising

text: regime

text: does

text: present

text: different

text: optimal

text: forewing-hindwing

text: phase

text: lags

text: depending

text: on

text: whether

text: one

text: would

text: want

text: to

text: tune

text: the

text: system

text: for

text: maximum

text: cruising

text: speed

text: or

text: minimum

text: energy

text: expenditure.

text: These

text: results

text: are

text: in

text: accordance

text: with

text: previous

text: studies

text: in

text: hovering

text: con-

text: ﬁgurations

text: by

text: Wang

text: Russell

text: Usherwood

text: Lehmann

text: hinting

text: that

text: the

text: mechanisms

text: described

text: here

text: should

text: be

text: robust

text: elements

text: to

text: consider

text: in

text: any

text: aero-

text: dynamic

text: model:

text: one

text: the

text: one

text: hand

text: wing

text: inertia

text: is

text: a

text: major

text: player

text: in

text: the

text: power

text: expen-

text: diture,

text: while

text: on

text: the

text: other

text: hand

text: the

text: thrust

text: production

text: is

text: ruled

text: by

text: the

text: aerodynamics

text: around

text: the

text: ﬂexible

text: wings.

text: A

text: full

text: ﬂow

text: ﬁeld

text: reconstruction

text: around

text: the

text: wings

text: is

text: certainly

text: desirable

text: to

text: compare

text: different

text: points

text: in

text: the

text: parameter

text: space

text: with

text: different

text: performances

text: and

text: clearly

text: iden-

text: tify

text: the

text: aerodynamic

text: mechanisms

text: at

text: play.

text: Experimentally

text: this

text: can

text: be

text: challenging,

text: and

text: numerical

text: simulations

text: such

text: as

text: a

text: version

text: of

text: the

text: ﬂuid-structure

text: simulations

text: of

text: Kolomenskiy

text: et

text: al.

text: can

text: be

text: an

text: interesting

text: option

text: to

text: deﬁne

text: robust

text: aerodynamic

text: models.

text: Other

text: issue

text: that

text: remains

text: not

text: fully

text: understood

text: is

text: the

text: effect

text: of

text: the

text: forewing-

text: hindwing

text: separation

text: d

text: as

text: a

text: parameter

text: independent

text: of

text: the

text: forewing-hindwing

text: phase

text: lag

text: which

text: will

text: have

text: a

text: different

text: role

text: depending

text: on

text: whether

text: the

text: system

text: is

text: hovering

text: or

text: cruising.

text: This

text: point

text: will

text: be

text: of

text: particular

text: importance

text: when

text: going

text: beyond

text: the

text: “steady”

text: regimes

text: of

text: hovering

text: or

text: cruising

text: and

text: into

text: the

text: study

text: of

text: transient

text: regimes.

text: These

text: bring

text: indeed

text: a

text: vast

text: set

text: of

text: open

text: questions

text: related

text: to

text: the

text: unsteadiness

text: —like

text: for

text: instance

text: the

text: multi-body

text: dynamics

text: of

text: manoeuvring—

text: and

text: where

text: the

text: wings

text: are

text: to

text: be

text: analysed

text: as

text: part

text: of

text: a

text: full

text: system

text: accelerating,

text: performing

text: sharp

text: turns

text: (e.g.

text: Bergou

text: et

text: al.,

text: or

text: taking

text: off

text: (e.g.

text: Bimbard

text: et

text: al.,

text: Acknowledgements

text: We

text: thank

text: all

text: the

text: people

text: at

text: the

text: PMMH

text: workshop

text: for

text: their

text: help

text: with

text: the

text: experimental

text: setup,

text: in

text: particular

text: D.

text: Pradal

text: who

text: conceived

text: and

text: built

text: most

text: of

text: its

text: parts.

text: We

text: thank

text: also

text: X.

text: Benoit-Gonin,

text: M.

text: Vilmay

text: and

text: G.

text: Lemoult

text: for

text: the

text: Lab-

text: View

text: interfacing,

text: and

text: S.

text: Ramananarivo,

text: V.

text: Raspa,

text: D.

text: Kolomenskiy

text: and

text: G.

text: Sped-

text: ding

text: for

text: useful

text: discussions.

text: This

text: work

text: was

text: supported

text: by

text: the

text: French

text: National

text: Re-

text: search

text: Agency

text: through

text: project

text: No.

text: and

text: by

text: EADS

text: Foundation

text: through

text: project

text: ”Fluids

text: and

text: elasticity

text: in

text: biomimetic

text: propulsion”.

text: M.

text: C.

text: acknowl-

text: edges

text: ﬁnancial

text: support

text: from

text: CONACyT

text: and

text: UNAM,

text: Me´xico.

text: P.

text: J.

text: acknowledges

text: support

text: from

text: the

text: Charpak

text: Internship

text: Program.


text: ing angle and block the production of sterile neutrinos from oscillations. They can also lead to a rapid scattering-induced decoherent production of sterile neutrinos at later times before CMB. When ﬂavor equilibrium between active and sterile species is approached, it surprisingly leads to a decrease of N . eﬀ We also discussed how the conﬂict with cosmological neutrino mass bounds can be re- laxed in this scenario. We have found that, the more light sterile species we add, the less constrained they would be. If we take the latest Planck bounds [49], Ncmb < 3.7 and eﬀ meﬀ < 0.52eV, at least three sterile species are needed to evade such a constraint. ν ACKNOWLEDGMENTS The author is grateful to Rasmus Hansen for helpful discussions and sharing his code. This work is supported in part by National Research Foundation of Korea (NRF) Research Grant 2012R1A2A1A01006053. [1] LSND Collaboration , C. Athanassopoulos et al., Evidence for anti-muon-neutrino → anti-electron-neutrino oscillations from the LSND experiment at LAMPF, Phys.Rev.Lett. 77 (1996) 3082–3085 [nucl-ex/9605003]. [2] LSND Collaboration , A. Aguilar-Arevalo et al., Evidence for neutrino oscillations from the observation of anti-neutrino(electron) appearance in a anti-neutrino(muon) beam, Phys.Rev. D64 (2001) 112007 [hep-ex/0104049]. [3] G. Mention, M. Fechner, T. Lasserre, T. Mueller, D. Lhuillier, et al., The Reactor Antineutrino Anomaly, Phys.Rev. D83 (2011) 073006 [arXiv:1101.2755]. [4] J. Abdurashitov, V. Gavrin, S. Girin, V. Gorbachev, P. Gurkina, et al., Measurement of the response of a Ga solar neutrino experiment to neutrinos from an Ar-37 source, Phys.Rev. C73 (2006) 045805 [nucl-ex/0512041]. [5] MiniBooNE Collaboration , A. Aguilar-Arevalo et al., Event Excess in the MiniBooNE Search for ν¯µ → ν¯e Oscillations, Phys.Rev.Lett. 105 (2010) 181801 [arXiv:1007.1150]. [6] MiniBooNE Collaboration , A. Aguilar-Arevalo et al., Improved Search for ν¯ → ν¯ µ e Oscillations in the MiniBooNE Experiment, Phys.Rev.Lett. 110 (2013) 161801 [arXiv:1207.4809]. 14 ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: ing

text: angle

text: and

text: block

text: the

text: production

text: of

text: sterile

text: neutrinos

text: from

text: oscillations.

text: They

text: can

text: also

text: lead

text: to

text: a

text: rapid

text: scattering-induced

text: decoherent

text: production

text: of

text: sterile

text: neutrinos

text: at

text: later

text: times

text: before

text: CMB.

text: When

text: ﬂavor

text: equilibrium

text: between

text: active

text: and

text: sterile

text: species

text: is

text: approached,

text: it

text: surprisingly

text: leads

text: to

text: a

text: decrease

text: of

text: N

text: eﬀ

text: We

text: also

text: discussed

text: how

text: the

text: conﬂict

text: with

text: cosmological

text: neutrino

text: mass

text: bounds

text: can

text: be

text: re-

text: laxed

text: in

text: this

text: scenario.

text: We

text: have

text: found

text: that,

text: the

text: more

text: light

text: sterile

text: species

text: we

text: add,

text: the

text: less

text: constrained

text: they

text: would

text: be.

text: If

text: we

text: take

text: the

text: latest

text: Planck

text: bounds

text: Ncmb

text: and

text: eﬀ

text: meﬀ

text: at

text: least

text: three

text: sterile

text: species

text: are

text: needed

text: to

text: evade

text: such

text: a

text: constraint.

text: ν

text: ACKNOWLEDGMENTS

text: The

text: author

text: is

text: grateful

text: to

text: Rasmus

text: Hansen

text: for

text: helpful

text: discussions

text: and

text: sharing

text: his

text: code.

text: This

text: work

text: is

text: supported

text: in

text: part

text: by

text: National

text: Research

text: Foundation

text: of

text: Korea

text: (NRF)

text: Research

text: Grant

text: LSND

text: Collaboration

text: C.

text: Athanassopoulos

text: et

text: al.,

text: Evidence

text: for

text: anti-muon-neutrino

text: anti-electron-neutrino

text: oscillations

text: from

text: the

text: LSND

text: experiment

text: at

text: LAMPF,

text: Phys.Rev.Lett.

text: LSND

text: Collaboration

text: A.

text: Aguilar-Arevalo

text: et

text: al.,

text: Evidence

text: for

text: neutrino

text: oscillations

text: from

text: the

text: observation

text: of

text: anti-neutrino(electron)

text: appearance

text: in

text: a

text: anti-neutrino(muon)

text: beam,

text: Phys.Rev.

text: G.

text: Mention,

text: M.

text: Fechner,

text: T.

text: Lasserre,

text: T.

text: Mueller,

text: D.

text: Lhuillier,

text: et

text: al.,

text: The

text: Reactor

text: Antineutrino

text: Anomaly,

text: Phys.Rev.

text: J.

text: Abdurashitov,

text: V.

text: Gavrin,

text: S.

text: Girin,

text: V.

text: Gorbachev,

text: P.

text: Gurkina,

text: et

text: al.,

text: Measurement

text: of

text: the

text: response

text: of

text: a

text: Ga

text: solar

text: neutrino

text: experiment

text: to

text: neutrinos

text: from

text: an

text: source,

text: Phys.Rev.

text: MiniBooNE

text: Collaboration

text: A.

text: Aguilar-Arevalo

text: et

text: al.,

text: Event

text: Excess

text: in

text: the

text: MiniBooNE

text: Search

text: for

text: ν¯µ

text: ν¯e

text: Oscillations,

text: Phys.Rev.Lett.

text: MiniBooNE

text: Collaboration

text: A.

text: Aguilar-Arevalo

text: et

text: al.,

text: Improved

text: Search

text: for

text: ν¯

text: ν¯

text: µ

text: e

text: Oscillations

text: in

text: the

text: MiniBooNE

text: Experiment,

text: Phys.Rev.Lett.

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##