text: (cid:99) (cid:48) for all j, all x , x , and all x− . In this setting, the following lemma, proved via Hoeffding’s inequality, j j j gives a bound on the KL ’s: j (cid:48) L1 e−m1m, a 2. If the condition (13) holds unifo(cid:99)rmly for all j a(cid:112)nd all xj, xj, x−j, then with probability at least p ≤(cid:18)nδ2 (cid:19) · max KL + 2 n log(p) δ. j=1,...,p j 2 (cid:99) √ In other words, if Q satisﬁes (13) for some δ = o 1 , then with high probability every KL will be j j nlog(p) small. By Theorem 1, then, the FDR for model-X knockoffs in this setting is controlled near the target level q. 3.3.2 Gaussian knockoffs For a second example, suppose that the distribution of the feature vector X is mean zero and has covariance − Θ(cid:101) 1, where Θ is so(cid:101)me unknown precision matrix. (We assume zero mean for simplicity, but these results can of course be generalized to an arbitrary m(cid:101)ean.) Suppose that we have estimated Θ with some approximation Θ, and let Θj and Θj denote the jth columns o−f these matrices. Our results below will assume that t(cid:101)he error in estimating each column of Θ is small, i.e. Θ Θ is small for all j. j j Following Candès et al. [4, eqn. (3.2)], the Gaussian knockoff construction consists of drawing X from the (cid:0) (cid:1) distribution (cid:101) (cid:101) (cid:101) | ∼ N − − X ∗ X ∗ (I DΘ)X ∗, (2D DΘD) , (14) i, i, p p i, (cid:101) (cid:101) (cid:101) { } − inde(cid:23)pendently for each i, wher(cid:22)e D =− diag dj is a nonnegative diagonal matrix chosen(cid:101)to sa·t|isfy 2D DΘD 0, or equivalently, D 2Θ 1. Another formulation is that the distribution P | ( X) for X (cid:0) (cid:1) X X conditional on X is obtained by se(cid:101)tting (cid:101) (cid:101) | ∼ N − − X X (I DΘ)X, 2D DΘD . (15) p p (cid:101) (cid:101) If the true prec(cid:20)is(cid:18)ion ma(cid:19)tr(cid:21)ix of(cid:18)X w(cid:19)ere given (cid:18)by(cid:18)Θ (as(cid:19)su(cid:19)med(cid:32)to be positive deﬁnite), the(cid:33)n we can calculate that the joint distribution of the pair (X, X) has ﬁrst and second mome(cid:101)nts given(cid:101)by (cid:101) (cid:101) (cid:101) (cid:101) − − − E X 0 X Θ 1 Θ 1 D = , Var = − − − . X 0(cid:101) X Θ 1 D Θ 1 In other words, for every j, X and X are exchangeable if we only look at the ﬁrst and sec(cid:101)ond moments of the j j joint distribution. (cid:101) (cid:101) − If the true distribution of X is in fact Gaussian, again with mean zero and covariance Θ 1, then a stronger (cid:101) claim follows—the joint dAistr⊆ibution of (X, X) is then multivariate Gaussian and therefor(cid:101)e (X, X(cid:101))swap(A) =d (X, X) for every subset [p]. In other words, the knockoff construction determined by P | satisﬁes N − X X pairwise exchangeability, as deﬁned in (3), with respect to the distribution Q = (0, Θ 1). To frame this X p property in terms of conditionals, Qj,·|rather tha(cid:32)n an e(cid:32)stimated jo(cid:33)int dist(cid:33)ribution, QX, we can calculate the estimated conditional distributions Q ( X− ) as (cid:101) j j (cid:101) (cid:101) − | ∼ N (cid:62) Θ− 1 X X− X− j,j , , (16) (cid:101) j (cid:101)j j(cid:101) Θ Θ jj jj ∈ R − where Θ− p 1 is the column Θ w(cid:101) ith entry Θ removed. j,j j jj As noted in Section , we may want to work with estimated precision matrices, which are not positive semideﬁnite (PSD). The rationale is that if Θ is ﬁtted by regressing each X on the remaining features X− to j j 11 ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: for

text: all

text: j,

text: all

text: x

text: x

text: and

text: all

text: x−

text: In

text: this

text: setting,

text: the

text: following

text: lemma,

text: proved

text: via

text: Hoeffding’s

text: inequality,

text: j

text: j

text: j

text: gives

text: a

text: bound

text: on

text: the

text: KL

text: ’s:

text: j

text: a

text: If

text: the

text: condition

text: holds

text: for

text: all

text: j

text: all

text: xj,

text: xj,

text: x−j,

text: then

text: with

text: probability

text: at

text: least

text: p

text: max

text: KL

text: n

text: log(p)

text: δ.

text: j

text: In

text: other

text: words,

text: if

text: Q

text: satisﬁes

text: for

text: some

text: δ

text: o

text: then

text: with

text: high

text: probability

text: every

text: KL

text: will

text: be

text: j

text: j

text: nlog(p)

text: small.

text: By

text: Theorem

text: then,

text: the

text: FDR

text: for

text: model-X

text: knockoffs

text: in

text: this

text: setting

text: is

text: controlled

text: near

text: the

text: target

text: level

text: q.

text: Gaussian

text: knockoffs

text: For

text: a

text: second

text: example,

text: suppose

text: that

text: the

text: distribution

text: of

text: the

text: feature

text: vector

text: X

text: is

text: mean

text: zero

text: and

text: has

text: covariance

text: where

text: Θ

text: is

text: unknown

text: precision

text: matrix.

text: (We

text: assume

text: zero

text: mean

text: for

text: simplicity,

text: but

text: these

text: results

text: can

text: of

text: course

text: be

text: generalized

text: to

text: an

text: arbitrary

text: Suppose

text: that

text: we

text: have

text: estimated

text: Θ

text: with

text: some

text: approximation

text: Θ,

text: and

text: let

text: Θj

text: and

text: Θj

text: denote

text: the

text: jth

text: columns

text: o−f

text: these

text: matrices.

text: Our

text: results

text: below

text: will

text: assume

text: that

text: error

text: in

text: estimating

text: each

text: column

text: of

text: Θ

text: is

text: small,

text: i.e.

text: Θ

text: Θ

text: is

text: small

text: for

text: all

text: j.

text: j

text: j

text: Following

text: Candès

text: et

text: al.

text: eqn.

text: the

text: Gaussian

text: knockoff

text: construction

text: consists

text: of

text: drawing

text: X

text: from

text: the

text: distribution

text: N

text: X

text: X

text: (I

text: DΘ)X

text: DΘD)

text: i,

text: i,

text: p

text: p

text: i,

text: for

text: each

text: i,

text: D

text: diag

text: dj

text: is

text: a

text: nonnegative

text: diagonal

text: matrix

text: sa·t|isfy

text: DΘD

text: or

text: equivalently,

text: D

text: 2Θ

text: Another

text: formulation

text: is

text: that

text: the

text: distribution

text: P

text: X)

text: for

text: X

text: X

text: X

text: conditional

text: on

text: X

text: is

text: obtained

text: by

text: N

text: X

text: X

text: (I

text: DΘ)X,

text: DΘD

text: p

text: p

text: If

text: the

text: true

text: given

text: be

text: positive

text: deﬁnite),

text: we

text: can

text: calculate

text: that

text: the

text: joint

text: distribution

text: of

text: the

text: pair

text: (X,

text: X)

text: has

text: ﬁrst

text: and

text: second

text: E

text: X

text: X

text: Θ

text: Θ

text: D

text: Var

text: X

text: X

text: Θ

text: D

text: Θ

text: In

text: other

text: words,

text: for

text: every

text: j,

text: X

text: and

text: X

text: are

text: exchangeable

text: if

text: we

text: only

text: look

text: at

text: the

text: ﬁrst

text: and

text: moments

text: of

text: the

text: j

text: j

text: joint

text: distribution.

text: If

text: the

text: true

text: distribution

text: of

text: X

text: is

text: in

text: fact

text: Gaussian,

text: again

text: with

text: mean

text: zero

text: and

text: covariance

text: Θ

text: then

text: a

text: stronger

text: claim

text: follows—the

text: joint

text: dAistr⊆ibution

text: of

text: (X,

text: X)

text: is

text: then

text: multivariate

text: Gaussian

text: and

text: (X,

text: =d

text: (X,

text: X)

text: for

text: every

text: subset

text: [p].

text: In

text: other

text: words,

text: the

text: knockoff

text: construction

text: determined

text: by

text: P

text: satisﬁes

text: N

text: X

text: X

text: pairwise

text: exchangeability,

text: as

text: deﬁned

text: in

text: with

text: respect

text: to

text: the

text: distribution

text: Q

text: Θ

text: To

text: frame

text: this

text: X

text: p

text: property

text: in

text: terms

text: of

text: conditionals,

text: Qj,·|rather

text: an

text: QX,

text: we

text: can

text: calculate

text: the

text: estimated

text: conditional

text: distributions

text: Q

text: X−

text: as

text: j

text: j

text: N

text: Θ−

text: X

text: X−

text: X−

text: j,j

text: j

text: Θ

text: Θ

text: jj

text: jj

text: R

text: where

text: Θ−

text: p

text: is

text: the

text: column

text: Θ

text: ith

text: entry

text: Θ

text: removed.

text: j,j

text: j

text: jj

text: As

text: noted

text: in

text: Section

text: we

text: may

text: want

text: to

text: work

text: with

text: estimated

text: precision

text: matrices,

text: which

text: are

text: not

text: positive

text: semideﬁnite

text: (PSD).

text: The

text: rationale

text: is

text: that

text: if

text: Θ

text: is

text: ﬁtted

text: by

text: regressing

text: each

text: X

text: on

text: the

text: remaining

text: features

text: X−

text: to

text: j

text: j

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: (cid:101) (cid:98) × (cid:101) (cid:98) for some (cid:15), c > 0 when (X, X) is drawn from P P | . Then there exists a conditional distribution P | , S X ∈XRX× × R S Y X⊆ and a testing procedure that maps data (X, Y) n p n to a selected set of features (X, Y) { } 1, . . . , p , such that: • × If the data points (X ∗, Y ) are i.i.d. draws from the distribution Q P | , where Q is any dis- i, i (cid:0) (cid:1) X Y X X tribution whose jth conditional is Q (i.e. our estim(cid:98) ated conditional distribution Q for feature X is j j j correct), then S FDR = q. • × On the other hand, if the data points (X ∗, Y ) are i.i.d. draws from the distribution P P | (i.e. our i,(cid:0) (cid:1)i (cid:0) (cid:1) X Y X estimated conditional distribution Q is no(cid:98)t correct, as the true conditional distribution is P ), then j j S ≥ − − FDR q 1 + c(1 e (cid:15)) . (cid:99) (cid:0) (cid:1) (cid:98) ≈ ≥ For the last case (where P is the true distribution), if c 1 (i.e. KL (cid:15) with high probability) then S (cid:39) − − X ≈ − − ≈ ≈j FDR q(2 e (cid:15)); when (cid:15) 0 is small, we have 2 e (cid:15) 1 + (cid:15) e(cid:15), thus matching the upper bound in Theorem 1. 4 Proof of Theorem 1 Whereas all proofs of FDR control for the knockoff methods thus far have relied on martingale arguments (see [1, 2, 4]), here we will prove our main theorem using a novel leave-one-out argument. Before we begin, we would like to draw a loose analogy. To prove FDR controlling properties of the Benjamini–Hochberg procedure under independence of the p-values, Storey et al. [15] developed a very elegant martingale argument. Other proof techniques, however, operate by removing or leaving out one hypothesis (or one p-value); see Ferreira and Zwinderman [6] for example. At a very high level, our own methods are partially inspired by the latter approach. (cid:110) (cid:111) (cid:80) Fix (cid:15) > 0 and for any threshold t > 0, deﬁne (cid:99) (cid:80) ≥ ≤ R (t) := j∈H0 1 Wj { t, K≤Lj− }(cid:15) . (cid:15) 1 + j∈H0 1 Wj t (cid:110) (cid:111) T(cid:12)(cid:12) hen, for t(cid:98)he knockoff+(cid:99)ﬁlter wi(cid:12)(cid:12)th thr(cid:80)eshold T+, we can writ(cid:99)e (cid:80) {j : j ∈ S ∩|SH(cid:98)|0∨and KLj ≤ (cid:15)} = j∈(cid:80)H0∨1 Wj {≥ T+≥, KLj}(cid:80)≤ (cid:15) (cid:110) (cid:99) (cid:111) 1 (cid:80)1 j 1 Wj T+ (cid:80) { ≤ − } ≥ ≤ = 1 +∨ j 1 {Wj ≥ T+} · j∈H0 1 Wj{ T≤+,−KLj } (cid:15) ≤ q · R (T ), 1 j 1 Wj T+ 1 + j 1 Wj T+ (cid:15) + wk(cid:12)(cid:12)nhoecrkeotfhfeﬁ(cid:98)llatsetrs(treapthheorltd(cid:99)hsanbykndoeﬁckn(cid:12)(cid:12)oitfifo+n),otfhRe(cid:80)n(cid:15)wanedutsheethceonthstrreuschtoiol(cid:80)dnTo0f tahnedkon(cid:110)botcakinoff+ ﬁlte(cid:99)r. If we(cid:111)instead use the (cid:80) (cid:80) {j : j ∈ S q∩−H1 +0 a|nS(cid:98)d| KLj ≤ (cid:15)} = q1−+1 + j 1j{1W{Wj ≤j ≥−TT00}} · j∈1H+0 1 jW1j{W≥ jT≤0, K−LTj0}≤ (cid:15) ≤ q·R(cid:15)(T0), where the last step holds by deﬁnition of R and the construction of the knockoff ﬁlter. Either way, then, it is E ≤ (cid:15) sufﬁcient to prove that [R (T)] e(cid:15), where T is either T or T . (cid:15) + 0 13 ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

label: paragraph

text: for

text: some

text: c

text: when

text: (X,

text: X)

text: is

text: drawn

text: from

text: P

text: P

text: Then

text: there

text: exists

text: a

text: conditional

text: distribution

text: P

text: S

text: X

text: ∈XRX×

text: R

text: S

text: Y

text: X⊆

text: and

text: a

text: testing

text: procedure

text: that

text: maps

text: data

text: (X,

text: Y)

text: n

text: p

text: n

text: to

text: a

text: selected

text: set

text: of

text: features

text: (X,

text: Y)

text: p

text: such

text: that:

label: list

text: If

text: the

text: data

text: points

text: (X

text: Y

text: are

text: i.i.d.

text: draws

text: from

text: the

text: distribution

text: Q

text: P

text: where

text: Q

text: is

text: any

text: dis-

text: i,

text: i

text: X

text: Y

text: X

text: X

text: tribution

text: whose

text: jth

text: conditional

text: is

text: Q

text: (i.e.

text: our

text: ated

text: conditional

text: distribution

text: Q

text: for

text: feature

text: X

text: is

text: j

text: j

text: j

text: correct),

text: then

text: S

text: FDR

text: q.

text: On

text: the

text: other

text: hand,

text: if

text: the

text: data

text: points

text: (X

text: Y

text: are

text: i.i.d.

text: draws

text: from

text: the

text: distribution

text: P

text: P

text: (i.e.

text: our

text: X

text: Y

text: X

text: estimated

text: conditional

text: distribution

text: Q

text: is

text: correct,

text: as

text: the

text: true

text: conditional

text: distribution

text: is

text: P

text: then

text: j

text: j

text: S

text: FDR

text: q

text: e

text: For

text: the

text: last

text: case

text: (where

text: P

text: is

text: the

text: true

text: distribution),

text: if

text: c

text: (i.e.

text: KL

text: with

text: high

text: probability)

text: then

text: S

text: X

text: ≈j

text: FDR

text: e

text: when

text: is

text: small,

text: we

text: have

text: e

text: thus

text: matching

text: the

text: upper

text: bound

text: in

text: Theorem

text: Proof

text: of

text: Theorem

text: Whereas

text: all

text: proofs

text: of

text: FDR

text: control

text: for

text: the

text: knockoff

text: methods

text: thus

text: far

text: have

text: relied

text: on

text: martingale

text: arguments

text: (see

text: here

text: we

text: will

text: prove

text: our

text: main

text: theorem

text: using

text: a

text: novel

text: leave-one-out

text: argument.

text: Before

text: we

text: begin,

text: we

text: would

text: like

text: to

text: draw

text: a

text: loose

text: analogy.

text: To

text: prove

text: FDR

text: controlling

text: properties

text: of

text: the

text: Benjamini–Hochberg

text: procedure

text: under

text: independence

text: of

text: the

text: p-values,

text: Storey

text: et

text: al.

text: developed

text: a

text: very

text: elegant

text: martingale

text: argument.

text: Other

text: proof

text: techniques,

text: however,

text: operate

text: by

text: removing

text: or

text: leaving

text: out

text: one

text: hypothesis

text: (or

text: one

text: p-value);

text: see

text: Ferreira

text: and

text: Zwinderman

text: for

text: example.

text: At

text: a

text: very

text: high

text: level,

text: our

text: own

text: methods

text: are

text: partially

text: inspired

text: by

text: the

text: latter

text: approach.

text: Fix

text: and

text: for

text: any

text: threshold

text: t

text: deﬁne

text: R

text: (t)

text: Wj

text: t,

text: K≤Lj−

text: Wj

text: t

text: hen,

text: for

text: T+,

text: we

text: can

text: {j

text: j

text: S

text: KLj

text: Wj

text: T+≥,

text: j

text: Wj

text: T+

text: j

text: {Wj

text: T+}

text: Wj{

text: T≤+,−KLj

text: q

text: R

text: (T

text: j

text: Wj

text: T+

text: j

text: Wj

text: T+

text: If

text: use

text: the

text: {j

text: j

text: S

text: KLj

text: j

text: ≤j

text: where

text: the

text: last

text: step

text: holds

text: by

text: deﬁnition

text: of

text: R

text: and

text: the

text: construction

text: of

text: the

text: knockoff

text: ﬁlter.

text: Either

text: way,

text: then,

text: it

text: is

text: E

text: sufﬁcient

text: to

text: prove

text: that

text: [R

text: (T)]

text: where

text: T

text: is

text: either

text: T

text: or

text: T

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: (cid:101) (cid:101) Now we prove the general result, Lemma 6. Fix any j. Suppose that we condition on X− , X− , and on the { } { } j j unordered pair Xij, Xij = aij, bij f(cid:40)or each i—that is, after observing the unlabeled pair, we arbitrarily label them as a and b. Write a = (a , . . . , a ) and same(cid:101)for b . Let C = 0 if a = b , and otherwise let j 1j nj j ij ij ij (cid:101) +1, if (X , X ) = (a , b ), C := − ij ij ij ij ij 1, if (X , X ) = (b , a ). ij ij ij ij (cid:32) (cid:33) Then we h(cid:88)ave (cid:101) (cid:99) (cid:101) | · | P (X X − ) Q (X X −(cid:18) ) (cid:19) KLj = log Qj (Xij | Xi,−j ) · Pj(cid:88)(Xij | Xi,−j) (cid:88) (cid:99) i j ij i, j j ij i, j | · | P (a X − ) Q (b X − ) = C log j ij | i, j · j ij | i, j =: C KL . ij Q (a X − ) P (b X − ) ij ij j ij i, j j ij i, j i i (cid:110) (cid:12) (cid:54) (cid:111) (cid:110) (cid:12) (cid:111) By Lemma 1, for(cid:12)(cid:12)each i with ai(cid:101)j = bij we have (cid:101) (cid:12)(cid:12) (cid:101) (cid:110) (cid:12) (cid:111) (cid:110) (cid:12) (cid:111) P (cid:12) (cid:101) P (cid:101) (cid:12) (cid:101) C = +1 (cid:12) a , b , X− , X− (X , X ) = (a , b ) (cid:12) a , b , X− , X− ij j j j j ij ij ij ij j j j j = P − P C = 1 a , b , X− , X− (X , X ) = (b , a ) a , b , X− , X− ij j j j j ij ij ij ij j j j j (cid:99) | | P (a X − )Q (b X − ) = Qj (aij | Xi,−j )Pj(bij | Xi,−(cid:101)j) = eKLij. (25) j ij i, j j ij i, j (cid:99) Note th(cid:101)at this binary outcome is independent for each i. From this point on we treat X−j, X−j, aj, bj as ﬁxed (where a = (a , . . . , a ) and same for b ), and only the C ’s as random. Since KL depends only on j 1j nj j ij ij X|C−ijj|,≤X−1jb,ya(cid:99)jd,ebﬁjn(itii.oe(cid:104).n(cid:99),own eth(cid:12)(cid:12)(cid:12)seeevathriaat(cid:101)bles that w(cid:105)e are(cid:112)condition(cid:115)in(cid:88)g on(cid:99)), and i(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)s theref(cid:101)ore treatedas ﬁxed, while P − E ≥ ≤ 1 KL KL X− , X− , a , b 2 log(p) (KL )2 X− , X− , a , b j j j j j j ij (cid:99)j j j j p2 i (cid:54) by Hoeffding’s inequality. Next we work with the condition(cid:12)al expectati(cid:12)on of KL . For any i with a = b , we use (25) to calculate (cid:12)(cid:12)(cid:12) (cid:104) (cid:12)(cid:12)(cid:12) (cid:101) (cid:105)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) (cid:99) (cid:12)(cid:12)(cid:12) (cid:99) j ij ij (cid:12) − (cid:12) | | E C X− , X− , a , b == eK(cid:99)Lij 1 ≤ KLij . ij j j j j eKLij + 1 2 (cid:12) (cid:12) (cid:12) (cid:104) (cid:12) (cid:105)(cid:12) (cid:12)(cid:88) (cid:104) (cid:12) (cid:105) (cid:12) (cid:88) Then (cid:12)(cid:12) (cid:99) (cid:12)(cid:12) (cid:101) (cid:12)(cid:12) (cid:12)(cid:12) (cid:12)(cid:12) (cid:101) (cid:99) (cid:12)(cid:12) (cid:99) (cid:12) (cid:12) E E · ≤ 1 KL X− , X− , a , b = C X− , X− , a , b KL (KL )2. j j j j j ij j j j j ij 2 ij i i  (cid:12)  Therefore, combining everyt(cid:88)hing, (cid:112) (cid:115)(cid:88) (cid:12)(cid:12)  (cid:99) (cid:99) (cid:99) (cid:12) (cid:101)  (cid:12)(cid:12)  P ≥ 1 ≤ 1 KL (KL )2 + 2 log(p) (KL )2 X− , X− , a , b . j 2 ij (cid:80) (cid:99) ij j j j j p2 i i Now, under the event E(cid:26)δ (cid:99)we must have i(KLij)2(cid:112)≤ nδ2, and(cid:12)(cid:12)(cid:12)(cid:12) so we (cid:101)can write (cid:27) P · {E } ≥ nδ2 ≤ 1 KLj 1 δ 2 + 2δ n log(p) X−j, X−j, aj, bj p2 . 20 ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: Now

text: we

text: prove

text: the

text: general

text: result,

text: Lemma

text: Fix

text: any

text: j.

text: Suppose

text: that

text: we

text: condition

text: on

text: X−

text: X−

text: and

text: on

text: the

text: j

text: j

text: unordered

text: pair

text: Xij,

text: Xij

text: aij,

text: bij

text: each

text: i—that

text: is,

text: after

text: observing

text: the

text: unlabeled

text: pair,

text: we

text: arbitrarily

text: label

text: them

text: as

text: a

text: and

text: b.

text: Write

text: a

text: (a

text: a

text: and

text: b

text: Let

text: C

text: if

text: a

text: b

text: and

text: otherwise

text: let

text: j

text: nj

text: j

text: ij

text: ij

text: ij

text: if

text: (X

text: X

text: (a

text: b

text: C

text: ij

text: ij

text: ij

text: ij

text: ij

text: if

text: (X

text: X

text: (b

text: a

text: ij

text: ij

text: ij

text: ij

text: Then

text: we

text: P

text: (X

text: X

text: Q

text: (X

text: X

text: KLj

text: log

text: Qj

text: (Xij

text: Xi,−j

text: Xi,−j)

text: i

text: j

text: ij

text: i,

text: j

text: j

text: ij

text: i,

text: j

text: P

text: (a

text: X

text: Q

text: (b

text: X

text: C

text: log

text: j

text: ij

text: i,

text: j

text: j

text: ij

text: i,

text: j

text: C

text: KL

text: ij

text: Q

text: (a

text: X

text: P

text: (b

text: X

text: ij

text: ij

text: j

text: ij

text: i,

text: j

text: j

text: ij

text: i,

text: j

text: i

text: i

text: By

text: Lemma

text: i

text: with

text: bij

text: we

text: have

text: P

text: P

text: C

text: a

text: b

text: X−

text: X−

text: (X

text: X

text: (a

text: b

text: a

text: b

text: X−

text: X−

text: ij

text: j

text: j

text: j

text: j

text: ij

text: ij

text: ij

text: ij

text: j

text: j

text: j

text: j

text: P

text: P

text: C

text: a

text: b

text: X−

text: X−

text: (X

text: X

text: (b

text: a

text: a

text: b

text: X−

text: X−

text: ij

text: j

text: j

text: j

text: j

text: ij

text: ij

text: ij

text: ij

text: j

text: j

text: j

text: j

text: P

text: (a

text: X

text: )Q

text: (b

text: X

text: Qj

text: (aij

text: Xi,−j

text: )Pj(bij

text: eKLij.

text: j

text: ij

text: i,

text: j

text: j

text: ij

text: i,

text: j

text: Note

text: this

text: binary

text: outcome

text: is

text: independent

text: for

text: each

text: i.

text: From

text: this

text: point

text: on

text: we

text: treat

text: X−j,

text: X−j,

text: aj,

text: bj

text: as

text: ﬁxed

text: (where

text: a

text: (a

text: a

text: and

text: same

text: for

text: b

text: and

text: only

text: the

text: C

text: ’s

text: as

text: random.

text: Since

text: KL

text: depends

text: only

text: on

text: j

text: nj

text: j

text: ij

text: ij

text: that

text: and

text: treatedas

text: ﬁxed,

text: while

text: P

text: E

text: KL

text: KL

text: X−

text: X−

text: a

text: b

text: log(p)

text: (KL

text: X−

text: X−

text: a

text: b

text: j

text: j

text: j

text: j

text: j

text: j

text: ij

text: j

text: j

text: j

text: i

text: by

text: Hoeffding’s

text: inequality.

text: Next

text: we

text: work

text: with

text: the

text: of

text: KL

text: For

text: any

text: i

text: with

text: a

text: b

text: we

text: use

text: to

text: calculate

text: j

text: ij

text: ij

text: E

text: C

text: X−

text: X−

text: a

text: b

text: KLij

text: ij

text: j

text: j

text: j

text: j

text: eKLij

text: Then

text: E

text: E

text: KL

text: X−

text: X−

text: a

text: b

text: C

text: X−

text: X−

text: a

text: b

text: KL

text: (KL

text: j

text: j

text: j

text: j

text: j

text: ij

text: j

text: j

text: j

text: j

text: ij

text: ij

text: i

text: i

text: Therefore,

text: combining

text: P

text: KL

text: (KL

text: log(p)

text: (KL

text: X−

text: X−

text: a

text: b

text: j

text: ij

text: ij

text: j

text: j

text: j

text: j

text: i

text: i

text: Now,

text: under

text: the

text: event

text: must

text: have

text: so

text: we

text: write

text: P

text: {E

text: KLj

text: δ

text: 2δ

text: n

text: log(p)

text: X−j,

text: X−j,

text: aj,

text: bj

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: [10] N. Li and M. Stephens. Modeling linkage disequilibrium and identifying recombination hotspots using single-nucleotide polymorphism data. Genetics, 165(4):2213–2233, 2003. [11] J. Marchini and B. Howie. Genotype imputation for genome-wide association studies. Nat. Rev. Genet., 11(7):499–511, 2010. [12] Z. S. Qin, T. Niu, and J. S. Liu. Partition-ligation-expectation-maximization algorithm for haplotype inference with single-nucleotide polymorphisms. Am. J. Hum. Genet., 71(5):1242–1247, Nov 2002. [13] M. Sesia, C. Sabatti, and E. J. Candès. Gene Hunting with Knockoffs for Hidden Markov Models. ArXiv e-prints, June 2017. [14] M. Stephens, N. J. Smith, and P. Donnelly. A new statistical method for haplotype reconstruction from population data. Am. J. Hum. Genet., 68(4):978–989, Apr 2001. [15] J. D. Storey, J. E. Taylor, and D. Siegmund. Strong control, conservative point estimation and simulta- neous conservative consistency of false discovery rates: a uniﬁed approach. J. R. Stat. Soc. Ser. B Stat. Methodol., 66(1):187–205, 2004. [16] K. Zhang, M. Deng, T. Chen, M. S. Waterman, and F. Sun. A dynamic programming algorithm for haplotype block partitioning. Proc. Natl. Acad. Sci. U.S.A., 99(11):7335–7339, May 2002. 24

label: reference

text: N.

text: Li

text: and

text: M.

text: Stephens.

text: Modeling

text: linkage

text: disequilibrium

text: and

text: identifying

text: recombination

text: hotspots

text: using

text: single-nucleotide

text: polymorphism

text: data.

text: Genetics,

text: J.

text: Marchini

text: and

text: B.

text: Howie.

text: Genotype

text: imputation

text: for

text: genome-wide

text: association

text: studies.

text: Nat.

text: Rev.

text: Genet.,

text: Z.

text: S.

text: Qin,

text: T.

text: Niu,

text: and

text: J.

text: S.

text: Liu.

text: algorithm

text: for

text: haplotype

text: inference

text: with

text: single-nucleotide

text: polymorphisms.

text: Am.

text: J.

text: Hum.

text: Genet.,

text: Nov

text: M.

text: Sesia,

text: C.

text: Sabatti,

text: and

text: E.

text: J.

text: Candès.

text: Gene

text: Hunting

text: with

text: Knockoffs

text: for

text: Hidden

text: Markov

text: Models.

text: ArXiv

text: e-prints,

text: June

text: M.

text: Stephens,

text: N.

text: J.

text: Smith,

text: and

text: P.

text: Donnelly.

text: A

text: new

text: statistical

text: method

text: for

text: haplotype

text: reconstruction

text: from

text: population

text: data.

text: Am.

text: J.

text: Hum.

text: Genet.,

text: Apr

text: J.

text: D.

text: Storey,

text: J.

text: E.

text: Taylor,

text: and

text: D.

text: Siegmund.

text: Strong

text: control,

text: conservative

text: point

text: estimation

text: and

text: simulta-

text: neous

text: conservative

text: consistency

text: of

text: false

text: discovery

text: rates:

text: a

text: uniﬁed

text: approach.

text: J.

text: R.

text: Stat.

text: Soc.

text: Ser.

text: B

text: Stat.

text: Methodol.,

text: K.

text: Zhang,

text: M.

text: Deng,

text: T.

text: Chen,

text: M.

text: S.

text: Waterman,

text: and

text: F.

text: Sun.

text: A

text: dynamic

text: programming

text: algorithm

text: for

text: haplotype

text: block

text: partitioning.

text: Proc.

text: Natl.

text: Acad.

text: Sci.

text: U.S.A.,

text: May


text: user’s activity from September 15th 2008, launch day of StackOverﬂow, to September 14th 2012. A. Analysis of used data We build a program which internal structure is shown in Fig. 4. Information which is contained in xml ﬁles can be represented in a form of a table. Therefore we performed conversion from xml to csv format because it can be manged by programing tools that we used for creating DIB-RM model. We wrote a parser which optimized to generate output results. It operates only with required ﬁelds without converting all ﬁle to aboutcsv. The next step is creating internal structure of data from csv ﬁles that provides the DIB-RM fast access to information of interactions. By interactions we consider Figure 4: DIB-RM structure. both posts and comments because they show activity of a user and his/her contribution to the system. Post is a general concept of content which users produce. It can generated 8.630.000 posts, 16.067.000 comments and received be of two types: question or answer. In this paper we 33.269.000 votes. do not distinguish types of interactions and assign the same reputation value to them. A typical post tuple is < B. Activity period factor PostId, CreationDate, PostTypeId, ParentId, UserId >, The ﬁrst step of our experimentation is to understand a typical tuple of comment is < the importance of the t constant (activity period) essential CommentId, CreationDate, UserId >. for computing value of a”cumulative” part of interaction. We 1) PostId, CommentId - positive integer which represent performed a set of computations changing the t constant a unique identiﬁer of entity. (in days) obtaining ratings difference averages and standard 2) CreationDate - date and time when post or comment was deviation values shown in table I and in table II. DIB- created. RM model has three factors which can be changed: t , α a 3) UserId - positive integer which represents unique iden- (cumulative factor), β (forgetting factor). Computations are tiﬁer of the user who is author. provided with ﬁxed α = 1 and β = 0.99 for both parameters Those two tuples have similar domains so we can reputation and historical reputation. store them together. Sorting of interactions dataset by The results which are provided in tables I and II show that (UserId, CreationDate) key pair will give historical se- if ta (activity period) increases, metric value also increases quence for each user. We do not add votes as interactions for both parameters. It comes from the nature of the Stack- to DIB-RM because the purpose is to compare it with Stack- Overﬂow model which calculates reputation by adding value Overﬂow model which is based on voting system. of a new vote to the sum of previous ones and does not Vote entities are required to make simulation of decrease over time. So when ta increases, reputation value StackOverﬂow model. We created a program which is starts to decrease after a longer period of time, users have fully based on rules of calculating users’ reputation wider window to interact and increase cumulative part of in StackOverﬂow. Votes’ tuple has the structure < interactions’ reputation value. That means reputation keeps V oteId, CreatinDate, V oteTypeId, PostId, UserId >. almost the same or increases to the high value of interaction because cumulative part also decreases less often. It makes 1) VoteId - positive integer which represent unique identi- reputation lose a dynamic property and become static as the ﬁer of vote. StackOverﬂow model. 2) PostId - positive integer which represent unique identi- Historical reputation does not decrease as StackOverﬂow ﬁer of post. Vote is related to this post. reputation also and contains values of all interactions. That 3) VoteTypeId - positive integer which represent type of is why historical reputation approximates better than the vote. It can have a value in the range from 0 to 9. StackOverﬂow’s model reputation value. Values of metric in 4) CreationDate - date and time when vote was created. table II are between 0.88 and 0.882. That means increasing of Each post has a UserId attribute and we can connect a vote t does not have signiﬁcant inﬂuence on metric results. a with its’ recipient and change his reputation. Four graphs are depicted in Fig. 5 for different t (activity a Total amount of users that we used for computations is period) values. They show reputation changes for two users 15.000. Between the minimum and maximum StackOverﬂow’s over time. The red line belongs to user with id = 300, the blue reputations we identiﬁed 10 equal intervals and extracted line to user with id = 235. Comparison of these lines shows 1.500 users from each group. This method allows to have that high value of t increases the distance between reputation a a representative set of users. During 4 years those users values. At ﬁrst sub-graph user 235 has four times has greater ##LTFigure##

text: user’s

text: activity

text: from

text: September

text: launch

text: day

text: of

text: StackOverﬂow,

text: to

text: September

text: A.

label: section

text: Analysis

text: of

text: used

text: data

text: We

text: build

text: a

text: program

text: which

text: internal

text: structure

text: is

text: shown

text: in

text: Fig.

text: Information

text: which

text: is

text: contained

text: in

text: xml

text: ﬁles

text: can

text: be

text: represented

text: in

text: a

text: form

text: of

text: a

text: table.

text: Therefore

text: we

text: performed

text: conversion

text: from

text: xml

text: to

text: csv

text: format

text: because

text: it

text: can

text: be

text: manged

text: by

text: programing

text: tools

text: that

text: we

text: used

text: for

text: creating

text: DIB-RM

text: model.

text: We

text: wrote

text: a

text: parser

text: which

text: optimized

text: to

text: generate

text: output

text: results.

text: It

text: operates

text: only

text: with

text: required

text: ﬁelds

text: without

text: converting

text: all

text: ﬁle

text: to

text: aboutcsv.

text: The

text: next

text: step

text: is

text: creating

text: internal

text: structure

text: of

text: data

text: from

text: csv

text: ﬁles

text: that

text: provides

text: the

text: DIB-RM

text: fast

text: access

text: to

text: information

text: of

text: interactions.

text: By

text: interactions

text: we

text: consider

text: Figure

text: DIB-RM

text: structure.

text: both

text: posts

text: and

text: comments

text: because

text: they

text: show

text: activity

text: of

text: a

text: user

text: and

text: his/her

text: contribution

text: to

text: the

text: system.

text: Post

text: is

text: a

text: general

text: concept

text: of

text: content

text: which

text: users

text: produce.

text: It

text: can

text: generated

text: posts,

text: comments

text: and

text: received

text: be

text: of

text: two

text: types:

text: question

text: or

text: answer.

text: In

text: this

text: paper

text: we

text: votes.

text: do

text: not

text: distinguish

text: types

text: of

text: interactions

text: and

text: assign

text: the

text: same

text: reputation

text: value

text: to

text: them.

text: A

text: typical

text: post

text: tuple

text: is

text: B.

text: Activity

text: period

text: factor

text: PostId,

text: CreationDate,

text: PostTypeId,

text: ParentId,

text: UserId

text: The

text: ﬁrst

text: step

text: of

text: our

text: experimentation

text: is

text: to

text: understand

text: a

text: typical

text: tuple

text: of

text: comment

text: is

text: the

text: importance

text: of

text: the

text: t

text: constant

text: (activity

text: period)

text: essential

text: CommentId,

text: CreationDate,

text: UserId

text: for

text: computing

text: value

text: of

text: a”cumulative”

text: part

text: of

text: interaction.

text: We

text: PostId,

text: CommentId

text: positive

text: integer

text: which

text: represent

text: performed

text: a

text: set

text: of

text: computations

text: changing

text: the

text: t

text: constant

text: a

text: unique

text: identiﬁer

text: of

text: entity.

text: (in

text: days)

text: obtaining

text: ratings

text: difference

text: averages

text: and

text: standard

text: CreationDate

text: date

text: and

text: time

text: when

text: post

text: or

text: comment

text: was

text: deviation

text: values

text: shown

text: in

text: table

text: I

text: and

text: in

text: table

text: II.

text: DIB-

text: created.

text: RM

text: model

text: has

text: three

text: factors

text: which

text: can

text: be

text: changed:

text: t

text: α

text: a

text: UserId

text: positive

text: integer

text: which

text: represents

text: unique

text: iden-

text: (cumulative

text: factor),

text: β

text: (forgetting

text: factor).

text: Computations

text: are

text: tiﬁer

text: of

text: the

text: user

text: who

text: is

text: author.

text: provided

text: with

text: ﬁxed

text: α

text: and

text: β

text: for

text: both

text: parameters

text: Those

text: two

text: tuples

text: have

text: similar

text: domains

text: so

text: we

text: can

text: reputation

text: and

text: historical

text: reputation.

text: store

text: them

text: together.

text: Sorting

text: of

text: interactions

text: dataset

text: by

text: The

text: results

text: which

text: are

text: provided

text: in

text: tables

text: I

text: and

text: II

text: show

text: that

text: (UserId,

text: CreationDate)

text: key

text: pair

text: will

text: give

text: historical

text: se-

text: if

text: ta

text: (activity

text: period)

text: increases,

text: metric

text: value

text: also

text: increases

text: quence

text: for

text: each

text: user.

text: We

text: do

text: not

text: add

text: votes

text: as

text: interactions

text: for

text: both

text: parameters.

text: It

text: comes

text: from

text: the

text: nature

text: of

text: the

text: Stack-

text: to

text: DIB-RM

text: because

text: the

text: purpose

text: is

text: to

text: compare

text: it

text: with

text: Stack-

text: Overﬂow

text: model

text: which

text: calculates

text: reputation

text: by

text: adding

text: value

text: Overﬂow

text: model

text: which

text: is

text: based

text: on

text: voting

text: system.

text: of

text: a

text: new

text: vote

text: to

text: the

text: sum

text: of

text: previous

text: ones

text: and

text: does

text: not

text: Vote

text: entities

text: are

text: required

text: to

text: make

text: simulation

text: of

text: decrease

text: over

text: time.

text: So

text: when

text: ta

text: increases,

text: reputation

text: value

text: StackOverﬂow

text: model.

text: We

text: created

text: a

text: program

text: which

text: is

text: starts

text: to

text: decrease

text: after

text: a

text: longer

text: period

text: of

text: time,

text: users

text: have

text: fully

text: based

text: on

text: rules

text: of

text: calculating

text: users’

text: reputation

text: wider

text: window

text: to

text: interact

text: and

text: increase

text: cumulative

text: part

text: of

text: in

text: StackOverﬂow.

text: Votes’

text: tuple

text: has

text: the

text: structure

text: interactions’

text: reputation

text: value.

text: That

text: means

text: reputation

text: keeps

text: V

text: oteId,

text: CreatinDate,

text: V

text: oteTypeId,

text: PostId,

text: UserId

text: almost

text: the

text: same

text: or

text: increases

text: to

text: the

text: high

text: value

text: of

text: interaction

text: because

text: cumulative

text: part

text: also

text: decreases

text: less

text: often.

text: It

text: makes

text: VoteId

text: positive

text: integer

text: which

text: represent

text: unique

text: identi-

text: reputation

text: lose

text: a

text: dynamic

text: property

text: and

text: become

text: static

text: as

text: the

text: ﬁer

text: of

text: vote.

text: StackOverﬂow

text: model.

text: PostId

text: positive

text: integer

text: which

text: represent

text: unique

text: identi-

text: Historical

text: reputation

text: does

text: not

text: decrease

text: as

text: StackOverﬂow

text: ﬁer

text: of

text: post.

text: Vote

text: is

text: related

text: to

text: this

text: post.

text: reputation

text: also

text: and

text: contains

text: values

text: of

text: all

text: interactions.

text: That

text: VoteTypeId

text: positive

text: integer

text: which

text: represent

text: type

text: of

text: is

text: why

text: historical

text: reputation

text: approximates

text: better

text: than

text: the

text: vote.

text: It

text: can

text: have

text: a

text: value

text: in

text: the

text: range

text: from

text: to

text: StackOverﬂow’s

text: model

text: reputation

text: value.

text: Values

text: of

text: metric

text: in

text: CreationDate

text: date

text: and

text: time

text: when

text: vote

text: was

text: created.

text: table

text: II

text: are

text: between

text: and

text: That

text: means

text: increasing

text: of

text: Each

text: post

text: has

text: a

text: UserId

text: attribute

text: and

text: we

text: can

text: connect

text: a

text: vote

text: t

text: does

text: not

text: have

text: signiﬁcant

text: inﬂuence

text: on

text: metric

text: results.

text: a

text: with

text: its’

text: recipient

text: and

text: change

text: his

text: reputation.

text: Four

text: graphs

text: are

text: depicted

text: in

text: Fig.

text: for

text: different

text: t

text: (activity

text: a

text: Total

text: amount

text: of

text: users

text: that

text: we

text: used

text: for

text: computations

text: is

text: period)

text: values.

text: They

text: show

text: reputation

text: changes

text: for

text: two

text: users

text: Between

text: the

text: minimum

text: and

text: maximum

text: StackOverﬂow’s

text: over

text: time.

text: The

text: red

text: line

text: belongs

text: to

text: user

text: with

text: id

text: the

text: blue

text: reputations

text: we

text: identiﬁed

text: equal

text: intervals

text: and

text: extracted

text: line

text: to

text: user

text: with

text: id

text: Comparison

text: of

text: these

text: lines

text: shows

text: users

text: from

text: each

text: group.

text: This

text: method

text: allows

text: to

text: have

text: that

text: high

text: value

text: of

text: t

text: increases

text: the

text: distance

text: between

text: reputation

text: a

text: a

text: representative

text: set

text: of

text: users.

text: During

text: years

text: those

text: users

text: values.

text: At

text: ﬁrst

text: sub-graph

text: user

text: has

text: four

text: times

text: has

text: greater

text: ##LTFigure##


text: # α µH σH [13] Lik Mui, Mojdeh Mohtashemi, and Ari Halberstadt. A computational 1 1 0,8805 0,0026 model of trust and reputation. In System Sciences, 2002. HICSS. Proceedings of the 35th Annual Hawaii International Conference on, 2 2 0,8806 0,0025 pages 2431–2439. IEEE, 2002. [14] Monika Rakoczy, Amel Bouzeghoub, Katarzyna Wegrzyn-Wolska, and 3 4 0,8808 0,0030 Alda Gancarski Lopes. Users views on others–analysis of confused 4 8 0,8808 0,0023 relation-based terms in social network. In OTM Confederated Inter- national Conferences” On the Move to Meaningful Internet Systems”, pages 155–174. Springer, 2016. Table VI: Table of historical reputation metric results for [15] Denise M Rousseau, Sim B Sitkin, Ronald S Burt, and Colin Camerer. different α values Not so different after all: A cross-discipline view of trust. Academy of management review, 23(3):393–404, 1998. [16] Jordi Sabater and Carles Sierra. Reputation and social network analysis in multi-agent systems. In Proceedings of the ﬁrst international joint and hypothesis by comparing ratings of users that are created conference on Autonomous agents and multiagent systems: Part 1, pages by DIB-RM and StackOverﬂow model. We used two values for 475–482. ACM, 2002. [17] Wanita Sherchan, Surya Nepal, and Cecile Paris. A survey of trust in creating ratings: reputation and historical reputation. Historical social networks. ACM Comput. Surv., 45(4):47:1–47:33, August 2013. reputation value gave better results, around 88% similarity [18] Megan Tschannen-Moran and Wayne K Hoy. A multidisciplinary between DIB-RM and StackOverﬂow ratings. Results of eval- analysis of the nature, meaning, and measurement of trust. Review of Educational Research, 70(4):547–593, 2000. uation show that this value is resistant to factors’ changes, so [19] Yanchao Zhang and Yuguang Fang. A ﬁne-grained reputation system for it allow to adopt model to some environments by selecting reliable service selection in peer-to-peer networks. IEEE Transactions different values of factors without decreasing of metric value. on Parallel and Distributed Systems, 18(8), 2007. [20] Runfang Zhou and Kai Hwang. Powertrust: A robust and scalable We believe that this factors can be used as an evidence of reputation system for trusted peer-to-peer computing. IEEE Transactions users’ trustworthiness in combination with more traditional on parallel and distributed systems, 18(4), 2007. ones. Our further works will be addressed to determining environments in which context dynamic factors can be used as a strong evidence of trustworthiness. REFERENCES [1] Sibel Adali, Robert Escriva, Mark K Goldberg, Mykola Hayvanovych, Malik Magdon-Ismail, Boleslaw K Szymanski, William A Wallace, and Gregory Williams. Measuring behavioral trust in social networks. In Intelligence and Security Informatics (ISI), 2010 IEEE International Conference on, pages 150–152. IEEE, 2010. [2] Elizabeth Chang, Patricia Thomson, Tharam Dillon, and Farookh Hus- sain. The fuzzy and dynamic nature of trust. Trust, privacy, and security in digital business, pages 161–174, 2005. [3] Anupam Das and Mohammad Mahfuzul Islam. Securedtrust: a dynamic trust computation model for secured communication in multiagent systems. IEEE Transactions on Dependable and Secure Computing, 9(2):261–274, 2012. [4] Jennifer Golbeck. Generating predictive movie recommendations from trust in social networks. Trust Management, pages 93–104, 2006. [5] Sana Hamdi. Computational models of trust and reputation in online social networks. PhD thesis, Universite´ Paris-Saclay, 2016. [6] Wenjun Jiang and Guojun Wang. Swtrust: Generating trusted graph for trust evaluation in online social networks. In Trust, Security and Privacy in Computing and Communications (TrustCom), 2011 IEEE 10th International Conference on, pages 320–327. IEEE, 2011. [7] Sepandar D Kamvar, Mario T Schlosser, and Hector Garcia-Molina. The eigentrust algorithm for reputation management in p2p networks. In Proceedings of the 12th international conference on World Wide Web, pages 640–651. ACM, 2003. [8] Ugur Kuter and Jennifer Golbeck. Sunny: A new algorithm for trust inference in social networks using probabilistic conﬁdence models. In AAAI, volume 7, pages 1377–1382, 2007. [9] Haifeng Liu, Ee-Peng Lim, Hady W Lauw, Minh-Tam Le, Aixin Sun, Jaideep Srivastava, and Young Kim. Predicting trusts among users of online communities: an epinions case study. In Proceedings of the 9th ACM conference on Electronic commerce, pages 310–319. ACM, 2008. [10] Luca Longo, Pierpaolo Dondio, and Stephen Barrett. Temporal factors to evaluate trustworthiness of virtual identities. In Security and Privacy in Communications Networks and the Workshops, 2007. SecureComm 2007. Third International Conference on, pages 11–19. IEEE, 2007. [11] Luca Longo, Pierpaolo Dondio, Bresciani Riccardo, Andrew Butterﬁeld, and Stephen Barrett. Enabling adaptation in trust computations. Com- putationworld, 2009. [12] Manuel Mazzara, Luca Biselli, Pier Paolo Greco, Nicola Dragoni, An- tonio Marraffa, Nafees Qamar, and Simona de Nicola. Social networks and collective intelligence: a return to the agora. IGI Global, 2013. ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

label: table

text: α

text: µH

text: σH

text: Lik

text: Mui,

text: Mojdeh

text: Mohtashemi,

text: and

text: Ari

text: Halberstadt.

text: A

text: computational

text: model

text: of

text: trust

text: and

text: reputation.

text: In

text: System

text: Sciences,

text: HICSS.

text: Proceedings

text: of

text: the

text: Annual

text: Hawaii

text: International

text: Conference

text: on,

text: pages

text: IEEE,

text: Monika

text: Rakoczy,

text: Amel

text: Bouzeghoub,

text: Katarzyna

text: Wegrzyn-Wolska,

text: and

text: Alda

text: Gancarski

text: Lopes.

text: Users

text: views

text: on

text: others–analysis

text: of

text: confused

text: relation-based

text: terms

text: in

text: social

text: network.

text: In

text: OTM

text: Confederated

text: Inter-

text: national

text: Conferences”

text: On

text: the

text: Move

text: to

text: Meaningful

text: Internet

text: Systems”,

text: pages

text: Springer,

text: Table

text: VI:

text: Table

label: caption

text: of

text: historical

text: reputation

text: metric

text: results

text: for

text: Denise

text: M

text: Rousseau,

text: Sim

text: B

text: Sitkin,

text: Ronald

text: S

text: Burt,

text: and

text: Colin

text: Camerer.

text: different

text: α

text: values

text: Not

text: so

text: different

text: after

text: all:

text: A

text: cross-discipline

text: view

text: of

text: trust.

text: Academy

text: of

text: management

text: review,

text: Jordi

text: Sabater

text: and

text: Carles

text: Sierra.

text: Reputation

text: and

text: social

text: network

text: analysis

text: in

text: multi-agent

text: systems.

text: In

text: Proceedings

text: of

text: the

text: ﬁrst

text: international

text: joint

text: and

text: hypothesis

text: by

text: comparing

text: ratings

text: of

text: users

text: that

text: are

text: created

text: conference

text: on

text: Autonomous

text: agents

text: and

text: multiagent

text: systems:

text: Part

text: pages

text: by

text: DIB-RM

text: and

text: StackOverﬂow

text: model.

text: We

text: used

text: two

text: values

text: for

text: ACM,

text: Wanita

text: Sherchan,

text: Surya

text: Nepal,

text: and

text: Cecile

text: Paris.

text: A

text: survey

text: of

text: trust

text: in

text: creating

text: ratings:

text: reputation

text: and

text: historical

text: reputation.

text: Historical

text: social

text: networks.

text: ACM

text: Comput.

text: Surv.,

text: August

text: reputation

text: value

text: gave

text: better

text: results,

text: around

text: similarity

text: Megan

text: Tschannen-Moran

text: and

text: Wayne

text: K

text: Hoy.

text: A

text: multidisciplinary

text: between

text: DIB-RM

text: and

text: StackOverﬂow

text: ratings.

text: Results

text: of

text: eval-

text: analysis

text: of

text: the

text: nature,

text: meaning,

text: and

text: measurement

text: of

text: trust.

text: Review

text: of

text: Educational

text: Research,

text: uation

text: show

text: that

text: this

text: value

text: is

text: resistant

text: to

text: factors’

text: changes,

text: so

text: Yanchao

text: Zhang

text: and

text: Yuguang

text: Fang.

text: A

text: ﬁne-grained

text: reputation

text: system

text: for

text: it

text: allow

text: to

text: adopt

text: model

text: to

text: some

text: environments

text: by

text: selecting

text: reliable

text: service

text: selection

text: in

text: peer-to-peer

text: networks.

text: IEEE

text: Transactions

text: different

text: values

text: of

text: factors

text: without

text: decreasing

text: of

text: metric

text: value.

text: on

text: Parallel

text: and

text: Distributed

text: Systems,

text: Runfang

text: Zhou

text: and

text: Kai

text: Hwang.

text: Powertrust:

text: A

text: robust

text: and

text: scalable

text: We

text: believe

text: that

text: this

text: factors

text: can

text: be

text: used

text: as

text: an

text: evidence

text: of

text: reputation

text: system

text: for

text: trusted

text: peer-to-peer

text: computing.

text: IEEE

text: Transactions

text: users’

text: trustworthiness

text: in

text: combination

text: with

text: more

text: traditional

text: on

text: parallel

text: and

text: distributed

text: systems,

text: ones.

text: Our

text: further

text: works

text: will

text: be

text: addressed

text: to

text: determining

text: environments

text: in

text: which

text: context

text: dynamic

text: factors

text: can

text: be

text: used

text: as

text: a

text: strong

text: evidence

text: of

text: trustworthiness.

text: REFERENCES

text: Sibel

text: Adali,

text: Robert

text: Escriva,

text: Mark

text: K

text: Goldberg,

text: Mykola

text: Hayvanovych,

text: Malik

text: Magdon-Ismail,

text: Boleslaw

text: K

text: Szymanski,

text: William

text: A

text: Wallace,

text: and

text: Gregory

text: Williams.

text: Measuring

text: behavioral

text: trust

text: in

text: social

text: networks.

text: In

text: Intelligence

text: and

text: Security

text: Informatics

text: (ISI),

text: IEEE

text: International

text: Conference

text: on,

text: pages

text: IEEE,

text: Elizabeth

text: Chang,

text: Patricia

text: Thomson,

text: Tharam

text: Dillon,

text: and

text: Farookh

text: Hus-

text: sain.

text: The

text: fuzzy

text: and

text: dynamic

text: nature

text: of

text: trust.

text: Trust,

text: privacy,

text: and

text: security

text: in

text: digital

text: business,

text: pages

text: Anupam

text: Das

text: and

text: Mohammad

text: Mahfuzul

text: Islam.

text: Securedtrust:

text: a

text: dynamic

text: trust

text: computation

text: model

text: for

text: secured

text: communication

text: in

text: multiagent

text: systems.

text: IEEE

text: Transactions

text: on

text: Dependable

text: and

text: Secure

text: Computing,

text: Jennifer

text: Golbeck.

text: Generating

text: predictive

text: movie

text: recommendations

text: from

text: trust

text: in

text: social

text: networks.

text: Trust

text: Management,

text: pages

text: Sana

text: Hamdi.

text: Computational

text: models

text: of

text: trust

text: and

text: reputation

text: in

text: online

text: social

text: networks.

text: PhD

text: thesis,

text: Universite´

text: Paris-Saclay,

text: Wenjun

text: Jiang

text: and

text: Guojun

text: Wang.

text: Swtrust:

text: Generating

text: trusted

text: graph

text: for

text: trust

text: evaluation

text: in

text: online

text: social

text: networks.

text: In

text: Trust,

text: Security

text: and

text: Privacy

text: in

text: Computing

text: and

text: Communications

text: (TrustCom),

text: IEEE

text: International

text: Conference

text: on,

text: pages

text: IEEE,

text: Sepandar

text: D

text: Kamvar,

text: Mario

text: T

text: Schlosser,

text: and

text: Hector

text: Garcia-Molina.

text: The

text: eigentrust

text: algorithm

text: for

text: reputation

text: management

text: in

text: networks.

text: In

text: Proceedings

text: of

text: the

text: international

text: conference

text: on

text: World

text: Wide

text: Web,

text: pages

text: ACM,

text: Ugur

text: Kuter

text: and

text: Jennifer

text: Golbeck.

text: Sunny:

text: A

text: new

text: algorithm

text: for

text: trust

text: inference

text: in

text: social

text: networks

text: using

text: probabilistic

text: conﬁdence

text: models.

text: In

text: AAAI,

text: volume

text: pages

text: Haifeng

text: Liu,

text: Ee-Peng

text: Lim,

text: Hady

text: W

text: Lauw,

text: Minh-Tam

text: Le,

text: Aixin

text: Sun,

text: Jaideep

text: Srivastava,

text: and

text: Young

text: Kim.

text: Predicting

text: trusts

text: among

text: users

text: of

text: online

text: communities:

text: an

text: epinions

text: case

text: study.

text: In

text: Proceedings

text: of

text: the

text: ACM

text: conference

text: on

text: Electronic

text: commerce,

text: pages

text: ACM,

text: Luca

text: Longo,

text: Pierpaolo

text: Dondio,

text: and

text: Stephen

text: Barrett.

text: Temporal

text: factors

text: to

text: evaluate

text: trustworthiness

text: of

text: virtual

text: identities.

text: In

text: Security

text: and

text: Privacy

text: in

text: Communications

text: Networks

text: and

text: the

text: Workshops,

text: SecureComm

text: Third

text: International

text: Conference

text: on,

text: pages

text: IEEE,

text: Luca

text: Longo,

text: Pierpaolo

text: Dondio,

text: Bresciani

text: Riccardo,

text: Andrew

text: Butterﬁeld,

text: and

text: Stephen

text: Barrett.

text: Enabling

text: adaptation

text: in

text: trust

text: computations.

text: Com-

text: putationworld,

text: Manuel

text: Mazzara,

text: Luca

text: Biselli,

text: Pier

text: Paolo

text: Greco,

text: Nicola

text: Dragoni,

text: An-

text: tonio

text: Marraffa,

text: Nafees

text: Qamar,

text: and

text: Simona

text: de

text: Nicola.

text: Social

text: networks

text: and

text: collective

text: intelligence:

text: a

text: return

text: to

text: the

text: agora.

text: IGI

text: Global,

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: 6 • The ego vehicle and the preceding vehicle should be in the same lane. If the left or right turn light was on, which indicates the driver would make a lane change or overtaking behavior, then the event was ended. • If one other vehicle was merging between the ego vehicle and the preceding vehicle, the car-following event was (a) ended. (b) (c) • When vehicle speed was less than 5 m/s (i.e., 18 km/h), the event ended. In addition, if the relative range was less than 10 m, the event ended, ensuring that no Stop-&-Go case was included. • The duration for a singular car-following event should be larger than 50 s. • Any driver with less than 500 car-following events was eliminated, which was able to guarantee the collected data were enough to capture the underlying driving styles [46]. Fig. 5. One of the experimental vehicles with data-collection equipment. (a) An experimental vehicle; (b) Mobileye; (c) Data acquisition systems. Based on the above criteria, 49 drivers were selected out from IV. EXPERIMENT AND DATA COLLECTION 56 drivers. The number of car-following events for each driver was about 1,480 and each event lasted about 86.93 seconds We used naturalistic driving data collected from the Uni- on average. The extracted data consisted of the variables as versity of Michigan Safety Pilot Model Deployment (SPMD) follows: program [45]. It recorded the naturalistic driving of 2,842 equipped vehicles in Ann Arbor, Michigan, for over three ∈ years. As of April 2016, 34.9 million miles of driving data 1) Vehicle speed, vE [5, 45] [m/s]; were logged, making the SPMD database one of the largest 2) Relative speed, ∆v∈; public N-FOT ones ever. In the SPMD database, 98 sedans 3) Relative range, L [10, 120] [m]; were equipped with data acquisition systems (DASs) and 4) Time to collision, TT∈C, calculated from (1) MobilEye (Fig. 5). The vehicle was pre-equipped from the 5) Throttle opening, Th [0, 100]; factory with sensors to measure the speed, lane marks, relative 6) Braking action, Br = 1 or 0. speed and distance between the ego vehicle and the preceding The braking actions Br were recorded with the rules: Br = 1 vehicle, and road curvature. The data of relative speed, relative when the driver put his/her foot on the brake pedal; otherwise range, and road curvature were collected from the MobileEye. Br = 0 when the driver removed his/her foot from the brake The ego vehicle speed, brake pedal position, and throttle pedal. opening were recorded from the CAN-bus of each vehicle. A. Driver Participants and Data Record C. General Data Analysis In this work, 56 drivers were included. They all hold a Overall, we collected 72,166 car-following events from the valid driving license. Each driver performed casual daily trips 49 drivers. Fig. 6 shows the number of car-following events without any restrictions or requirements on their trips. The and the percentage of data with braking actions in all events drivers were not restricted to a particular route and were given of each driver. We found that the percentage of those pushing no restrictions, including the duration of routes. While driving, the brake pedal during the car-following task was about 4.5% the on-board PC recorded driving data at a frequency of 10 on average, as shown by the green dashed line in Fig. 6. Some Hz. The DASs were not shown to the drivers and the process drivers tended to prefer frequent braking actions, (e.g., driver of recording data were hidden from the drivers, guaranteeing #42 with 12.27% braking actions); others tended to brake less that the drivers were not disturbed by the DASs and that the often (e.g., driver #8 with 1.43% braking actions). When we recorded data were naturalistic. While driving, many kinds of preprocessed data, the driver’s movement time – the time to driving behaviors, such as lane changing, overtaking, and car- lift the foot off the accelerator pedal, move it laterally to the following, were recorded. brake pedal, or vice versa – was not considered because the performance time was very small (about 0.15 – 0.30 seconds B. Data Extraction [47]), compared to the total driving time and the time of In order to extract the car-following data from the database, keeping their foot on the brake pedal. We also found that we deﬁned the car-following event as follows: drivers would not always put their foot on the brake pedal and gas pedal, but the duration was very small, compared • The relative distance L between the ego vehicle and to the total car-following time (with a percentage of about × − preceding vehicle should be less than 120 m [24]. If the 1.873 10 4 in our experiments). For each driver, in total, relative distance was L > 120 m, then we ended the we obtained 95.6 minutes of driving data on average with event. keeping their foot on the brake pedal. ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure##

text: The

text: ego

text: vehicle

text: and

text: the

text: preceding

text: vehicle

text: should

text: be

text: in

text: the

text: same

text: lane.

text: If

text: the

text: left

text: or

text: right

text: turn

text: light

text: was

text: on,

text: which

text: indicates

text: the

text: driver

text: would

text: make

text: a

text: lane

text: change

text: or

text: overtaking

text: behavior,

text: then

text: the

text: event

text: was

text: ended.

text: If

text: one

text: other

text: vehicle

text: was

text: merging

text: between

text: the

text: ego

text: vehicle

text: and

text: the

text: preceding

text: vehicle,

text: the

text: car-following

text: event

text: was

text: (a)

font: default

text: ended.

text: (b)

text: (c)

text: When

text: vehicle

text: speed

text: was

text: less

text: than

text: m/s

text: (i.e.,

text: km/h),

text: the

text: event

text: ended.

text: In

text: addition,

text: if

text: the

text: relative

text: range

text: was

text: less

text: than

text: m,

text: the

text: event

text: ended,

text: ensuring

text: that

text: no

text: Stop-&-Go

text: case

text: was

text: included.

text: The

text: duration

text: for

text: a

text: singular

text: car-following

text: event

text: should

text: be

text: larger

text: than

text: s.

text: Any

text: driver

text: with

text: less

text: than

text: car-following

text: events

text: was

text: eliminated,

text: which

text: was

text: able

text: to

text: guarantee

text: the

text: collected

text: data

text: were

text: enough

text: to

text: capture

text: the

text: underlying

text: driving

text: styles

text: Fig.

text: One

text: of

text: the

text: experimental

text: vehicles

text: with

text: data-collection

text: equipment.

text: (a)

text: An

text: experimental

text: vehicle;

text: (b)

text: Mobileye;

text: (c)

text: Data

text: acquisition

text: systems.

text: Based

text: on

text: the

text: above

text: criteria,

text: drivers

text: were

text: selected

text: out

text: from

text: IV.

text: EXPERIMENT

text: AND

text: DATA

text: COLLECTION

text: drivers.

text: The

text: number

text: of

text: car-following

text: events

text: for

text: each

text: driver

text: was

text: about

text: and

text: each

text: event

text: lasted

text: about

text: seconds

text: We

text: used

text: naturalistic

text: driving

text: data

text: collected

text: from

text: the

text: Uni-

text: on

text: average.

text: The

text: extracted

text: data

text: consisted

text: of

text: the

text: variables

text: as

text: versity

text: of

text: Michigan

text: Safety

text: Pilot

text: Model

text: Deployment

text: (SPMD)

text: follows:

text: program

text: It

text: recorded

text: the

text: naturalistic

text: driving

text: of

text: equipped

text: vehicles

text: in

text: Ann

text: Arbor,

text: Michigan,

text: for

text: over

text: three

text: years.

text: As

text: of

text: April

text: million

text: miles

text: of

text: driving

text: data

text: Vehicle

text: speed,

text: vE

text: [m/s];

text: were

text: logged,

text: making

text: the

text: SPMD

text: database

text: one

text: of

text: the

text: largest

text: Relative

text: speed,

text: ∆v∈;

text: public

text: N-FOT

text: ones

text: ever.

text: In

text: the

text: SPMD

text: database,

text: sedans

text: Relative

text: range,

text: L

text: [m];

text: were

text: equipped

text: with

text: data

text: acquisition

text: systems

text: (DASs)

text: and

text: Time

text: to

text: collision,

text: TT∈C,

text: calculated

text: from

text: MobilEye

text: (Fig.

text: The

text: vehicle

text: was

text: pre-equipped

text: from

text: the

text: Throttle

text: opening,

text: Th

text: factory

text: with

text: sensors

text: to

text: measure

text: the

text: speed,

text: lane

text: marks,

text: relative

text: Braking

text: action,

text: Br

text: or

text: speed

text: and

text: distance

text: between

text: the

text: ego

text: vehicle

text: and

text: the

text: preceding

text: The

text: braking

text: actions

text: Br

text: were

text: recorded

text: with

text: the

text: rules:

text: Br

text: vehicle,

text: and

text: road

text: curvature.

text: The

text: data

text: of

text: relative

text: speed,

text: relative

text: when

text: the

text: driver

text: put

text: his/her

text: foot

text: on

text: the

text: brake

text: pedal;

text: otherwise

text: range,

text: and

text: road

text: curvature

text: were

text: collected

text: from

text: the

text: MobileEye.

text: Br

text: when

text: the

text: driver

text: removed

text: his/her

text: foot

text: from

text: the

text: brake

text: The

text: ego

text: vehicle

text: speed,

text: brake

text: pedal

text: position,

text: and

text: throttle

text: pedal.

text: opening

text: were

text: recorded

text: from

text: the

text: CAN-bus

text: of

text: each

text: vehicle.

text: A.

text: Driver

text: Participants

text: and

text: Data

text: Record

text: C.

text: General

text: Data

text: Analysis

text: In

text: this

text: work,

text: drivers

text: were

text: included.

text: They

text: all

text: hold

text: a

text: Overall,

text: we

text: collected

text: car-following

text: events

text: from

text: the

text: valid

text: driving

text: license.

text: Each

text: driver

text: performed

text: casual

text: daily

text: trips

text: drivers.

text: Fig.

text: shows

text: the

text: number

text: of

text: car-following

text: events

text: without

text: any

text: restrictions

text: or

text: requirements

text: on

text: their

text: trips.

text: The

text: and

text: the

text: percentage

text: of

text: data

text: with

text: braking

text: actions

text: in

text: all

text: events

text: drivers

text: were

text: not

text: restricted

text: to

text: a

text: particular

text: route

text: and

text: were

text: given

text: of

text: each

text: driver.

text: We

text: found

text: that

text: the

text: percentage

text: of

text: those

text: pushing

text: no

text: restrictions,

text: including

text: the

text: duration

text: of

text: routes.

text: While

text: driving,

text: the

text: brake

text: pedal

text: during

text: the

text: car-following

text: task

text: was

text: about

text: the

text: on-board

text: PC

text: recorded

text: driving

text: data

text: at

text: a

text: frequency

text: of

text: on

text: average,

text: as

text: shown

text: by

text: the

text: green

text: dashed

text: line

text: in

text: Fig.

text: Some

text: Hz.

text: The

text: DASs

text: were

text: not

text: shown

text: to

text: the

text: drivers

text: and

text: the

text: process

text: drivers

text: tended

text: to

text: prefer

text: frequent

text: braking

text: actions,

text: (e.g.,

text: driver

text: of

text: recording

text: data

text: were

text: hidden

text: from

text: the

text: drivers,

text: guaranteeing

text: with

text: braking

text: actions);

text: others

text: tended

text: to

text: brake

text: less

text: that

text: the

text: drivers

text: were

text: not

text: disturbed

text: by

text: the

text: DASs

text: and

text: that

text: the

text: often

text: (e.g.,

text: driver

text: with

text: braking

text: actions).

text: When

text: we

text: recorded

text: data

text: were

text: naturalistic.

text: While

text: driving,

text: many

text: kinds

text: of

text: preprocessed

text: data,

text: the

text: driver’s

text: movement

text: time

text: the

text: time

text: to

text: driving

text: behaviors,

text: such

text: as

text: lane

text: changing,

text: overtaking,

text: and

text: car-

text: lift

text: the

text: foot

text: off

text: the

text: accelerator

text: pedal,

text: move

text: it

text: laterally

text: to

text: the

text: following,

text: were

text: recorded.

text: brake

text: pedal,

text: or

text: vice

text: versa

text: was

text: not

text: considered

text: because

text: the

text: performance

text: time

text: was

text: very

text: small

text: (about

text: seconds

text: B.

text: Data

text: Extraction

text: compared

text: to

text: the

text: total

text: driving

text: time

text: and

text: the

text: time

text: of

text: In

text: order

text: to

text: extract

text: the

text: car-following

text: data

text: from

text: the

text: database,

text: keeping

text: their

text: foot

text: on

text: the

text: brake

text: pedal.

text: We

text: also

text: found

text: that

text: we

text: deﬁned

text: the

text: car-following

text: event

text: as

text: follows:

text: drivers

text: would

text: not

text: always

text: put

text: their

text: foot

text: on

text: the

text: brake

text: pedal

text: and

text: gas

text: pedal,

text: but

text: the

text: duration

text: was

text: very

text: small,

text: compared

text: The

text: relative

text: distance

text: L

text: between

text: the

text: ego

text: vehicle

text: and

text: to

text: the

text: total

text: car-following

text: time

text: (with

text: a

text: percentage

text: of

text: about

text: preceding

text: vehicle

text: should

text: be

text: less

text: than

text: m

text: If

text: the

text: in

text: our

text: experiments).

text: For

text: each

text: driver,

text: in

text: total,

text: relative

text: distance

text: was

text: L

text: m,

text: then

text: we

text: ended

text: the

text: we

text: obtained

text: minutes

text: of

text: driving

text: data

text: on

text: average

text: with

text: event.

text: keeping

text: their

text: foot

text: on

text: the

text: brake

text: pedal.

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##