text: 3 is Λ itself, which has no inputs, and outputs a bipartite quantum system. Now, consider taking the quantum Bell scenario imple- mentation, and modifying it as follows: Instead of letting Bob choose his setting y freely, copy Alice’s classical out- put a and wire the copy into Bob’s classical input. This creates a new scenario, characterized by the conditional | probabilities p(ab x), since y is no longer freely chosen. | | Of course, p(ab x) = p(ab x, y = a) so we could describe | the probabilities p(ab x) that would characterize the new scenario without ever needing to actually perform the FIG. 2. The Signalling-Between-Outputs scenario considered hypothetical modiﬁcation, so long as we have a priori in Refs. [8, 18]. This scenario relaxes the no-communication | assumption between the variables A and B, thus modifying knowledge of p(ab xy). the usual Bell scenario depicted in Fig. 1(a) by the addition But this hypothetical scenario which we have imag- → of an edge A B. ined constructing by modifying the quantum Bell sce- nario is identically the quantum Instrumental scenario per Fig. 1(b). Furthermore, the same is true for the clas- sical and GPT variants of the scenarios, as we can simply genuinely quantum, nonlocal data may admit a purely take the third device Λ to be a source of either classical or classical explanation. This eﬀect is at the basis, for in- GPT states ρ. This leads us to the following fundamental stance, of the infamous detection loophole in Bell ex- statement periments [2]. It is thus not obvious a priori that the Instrumental scenario should admit a separation between ∈ T | (cid:48) | (cid:48) ∈ T classical, quantum and GPT correlations; it might be p Instr iﬀ p(ab x) = p (ab x, y=a) where p Bell another example of an uninteresting DAG, which simply (3) happens not to be identiﬁed by the HLP criterion. T The Instrumental scenario can also be understood as where is a placeholder for a correlation set, such as C Q G a Bell scenario with relaxed causality constraints. Such classical , quantum , or GPT . relaxations of the Bell scenario have been considered pre- In this sense, correlations in the Instrumental scenario viously. For instance, the no-communication assumption are essentially postselections of Bell scenario correlations: between the variables A and B in the Bell scenario has An experimenter might perform many runs of the Bell been relaxed in Refs. [8, 18], leading to the Signalling- scenario experiment, but then postselect to examine only Between-Outputs scenario represented in Fig. 2. Other those experimental runs when y=a. This postselected works have considered modiﬁed Bell scenarios wherein data will exhibit Instrumental scenario statistics, and T one no longer assumes that the measurement inputs X moreover, every correlation can arise via this sort Instr and Y could have been set-up freely [19, 20]. The In- of postselection on Bell scenario statistics2. strumental scenario represents simultaneous relaxation of To make this relationship between DAGs more concrete, both the measurement-freedom and no-communication as- note that compatibility with the Instrumental scenario sumptions: not only may the outcome of the measurement is deﬁned nearly identically to compatibility with the performed at B depend directly on the distant outcome Bell scenario, except all refere(cid:80)nces to the variable Y get A, but furthermore the measurement setting Y is not overwritten with references to the variable A: (cid:0) (cid:1) chosen freely but is instead ﬁxed entirely by the value A ∈ C | | | output by the distant measurement device. p iﬀ p(ab x) = p(λ)p(a xλ)p(b aλ) , Instr λ T ∈ Q | ⊗ We remark that testing for membership in , by p iﬀ p(ab x) = tr ρ E | F | , T Instr ∈ G Instr | | a x| b a admitting an extension to Bell per Eq.(3), generalizes to p Instr iﬀ p(ab x) = (ea|x (eb|a Ψ) . any well deﬁned correlation set in the Bell scenario. For example, one might consider relaxations of the quantum Even though the Quantum- and GPT-Bell-Scenario sets set corresponding to levels of the Navascu´es-Pironio-Ac´ın are in general distinct from the Classical-Bell-Scenario (NPA) hierarchy [21, 22], including the set known as Al- set, it may be that their postselections (deﬁning the cor- most Quantum Correlations [23], or tests for compatibil- responding Instrumental-Scenario correlation sets) all co- ity with a restricted Hilbert space dimension [24–26]. All incide. Indeed, it is well known that postselection of those correlation membership tests for the Bell scenario can be applied to the Instrumental scenario by simply introducing existential quantiﬁers: Does an extension of | ≡ | | (cid:54) T p(ab x) p(ab x, y=a) exist to p(ab x, y=a) such that the 2 The rTecognition that every Instr correlation must be compatible relevant condition for membership in T is satisﬁed? with is an example of a device-independent causal infer- Bell Bell ence technique which we call Interruption. Relating pairs of This modiﬁcation is especially easy for testing NPA-level DAGs in this manner is explored more generally in a forthcoming compatibility, as those semideﬁnite tests already natively manuscript, currently in preparation [17]. support unspeciﬁed probabilities. ##LTFigure## ##LTLine##

text: is

text: Λ

text: itself,

text: which

text: has

text: no

text: inputs,

text: and

text: outputs

text: a

text: bipartite

text: quantum

text: system.

text: Now,

text: consider

text: taking

text: the

text: quantum

text: Bell

text: scenario

text: imple-

text: mentation,

text: and

text: modifying

text: it

text: as

text: follows:

text: Instead

text: of

text: letting

text: Bob

text: choose

text: his

text: setting

text: y

text: freely,

text: copy

text: Alice’s

text: classical

text: out-

text: put

text: a

text: and

text: wire

text: the

text: copy

text: into

text: Bob’s

text: classical

text: input.

text: This

text: creates

text: a

text: new

text: scenario,

text: characterized

text: by

text: the

text: conditional

text: probabilities

text: p(ab

text: x),

text: since

text: y

text: is

text: no

text: longer

text: freely

text: chosen.

text: Of

text: course,

text: p(ab

text: x)

text: p(ab

text: x,

text: y

text: a)

text: so

text: we

text: could

text: describe

text: the

text: probabilities

text: p(ab

text: x)

text: that

text: would

text: characterize

text: the

text: new

text: scenario

text: without

text: ever

text: needing

text: to

text: actually

text: perform

text: the

text: FIG.

text: The

text: Signalling-Between-Outputs

text: scenario

text: considered

text: hypothetical

text: modiﬁcation,

text: so

text: long

text: as

text: we

text: have

text: a

text: priori

text: in

text: Refs.

text: This

text: scenario

text: relaxes

text: the

text: no-communication

text: assumption

text: between

text: the

text: variables

text: A

text: and

text: B,

text: thus

text: modifying

text: knowledge

text: of

text: p(ab

text: xy).

text: the

text: usual

text: Bell

text: scenario

text: depicted

text: in

text: Fig.

text: by

text: the

text: addition

text: But

text: this

text: hypothetical

text: scenario

text: which

text: we

text: have

text: imag-

text: of

text: an

text: edge

text: A

text: B.

text: ined

text: constructing

text: by

text: modifying

text: the

text: quantum

text: Bell

text: sce-

text: nario

text: is

text: identically

text: the

text: quantum

text: Instrumental

text: scenario

text: per

text: Fig.

text: Furthermore,

text: the

text: same

text: is

text: true

text: for

text: the

text: clas-

text: sical

text: and

text: GPT

text: variants

text: of

text: the

text: scenarios,

text: as

text: we

text: can

text: simply

text: genuinely

text: quantum,

text: nonlocal

text: data

text: may

text: admit

text: a

text: purely

text: take

text: the

text: third

text: device

text: Λ

text: to

text: be

text: a

text: source

text: of

text: either

text: classical

text: or

text: classical

text: explanation.

text: This

text: eﬀect

text: is

text: at

text: the

text: basis,

text: for

text: in-

text: GPT

text: states

text: ρ.

text: This

text: leads

text: us

text: to

text: the

text: following

text: fundamental

text: stance,

text: of

text: the

text: infamous

text: detection

text: loophole

text: in

text: Bell

text: ex-

text: statement

text: periments

text: It

text: is

text: thus

text: not

text: obvious

text: a

text: priori

text: that

text: the

text: Instrumental

text: scenario

text: should

text: admit

text: a

text: separation

text: between

text: T

text: T

text: classical,

text: quantum

text: and

text: GPT

text: correlations;

text: it

text: might

text: be

text: p

text: Instr

text: iﬀ

text: p(ab

text: x)

text: p

text: (ab

text: x,

text: y=a)

text: where

text: p

text: Bell

text: another

text: example

text: of

text: an

text: uninteresting

text: DAG,

text: which

text: simply

text: happens

text: not

text: to

text: be

text: identiﬁed

text: by

text: the

text: HLP

text: criterion.

text: T

text: The

text: Instrumental

text: scenario

text: can

text: also

text: be

text: understood

text: as

text: where

text: is

text: a

text: placeholder

text: for

text: a

text: correlation

text: set,

text: such

text: as

text: C

text: Q

text: G

text: a

text: Bell

text: scenario

text: with

text: relaxed

text: causality

text: constraints.

text: Such

text: classical

text: quantum

text: or

text: GPT

text: relaxations

text: of

text: the

text: Bell

text: scenario

text: have

text: been

text: considered

text: pre-

text: In

text: this

text: sense,

text: correlations

text: in

text: the

text: Instrumental

text: scenario

text: viously.

text: For

text: instance,

text: the

text: no-communication

text: assumption

text: are

text: essentially

text: postselections

text: of

text: Bell

text: scenario

text: correlations:

text: between

text: the

text: variables

text: A

text: and

text: B

text: in

text: the

text: Bell

text: scenario

text: has

text: An

text: experimenter

text: might

text: perform

text: many

text: runs

text: of

text: the

text: Bell

text: been

text: relaxed

text: in

text: Refs.

text: leading

text: to

text: the

text: Signalling-

text: scenario

text: experiment,

text: but

text: then

text: postselect

text: to

text: examine

text: only

text: Between-Outputs

text: scenario

text: represented

text: in

text: Fig.

text: Other

text: those

text: experimental

text: runs

text: when

text: y=a.

text: This

text: postselected

text: works

text: have

text: considered

text: modiﬁed

text: Bell

text: scenarios

text: wherein

text: data

text: will

text: exhibit

text: Instrumental

text: scenario

text: statistics,

text: and

text: T

text: one

text: no

text: longer

text: assumes

text: that

text: the

text: measurement

text: inputs

text: X

text: moreover,

text: every

text: correlation

text: can

text: arise

text: via

text: this

text: sort

text: Instr

text: and

text: Y

text: could

text: have

text: been

text: set-up

text: freely

text: The

text: In-

text: of

text: postselection

text: on

text: Bell

text: scenario

text: strumental

text: scenario

text: represents

text: simultaneous

text: relaxation

text: of

text: To

text: make

text: this

text: relationship

text: between

text: DAGs

text: more

text: concrete,

text: both

text: the

text: measurement-freedom

text: and

text: no-communication

text: as-

text: note

text: that

text: compatibility

text: with

text: the

text: Instrumental

text: scenario

text: sumptions:

text: not

text: only

text: may

text: the

text: outcome

text: of

text: the

text: measurement

text: is

text: deﬁned

text: nearly

text: identically

text: to

text: compatibility

text: with

text: the

text: performed

text: at

text: B

text: depend

text: directly

text: on

text: the

text: distant

text: outcome

text: Bell

text: scenario,

text: except

text: all

text: to

text: the

text: variable

text: Y

text: get

text: A,

text: but

text: furthermore

text: the

text: measurement

text: setting

text: Y

text: is

text: not

text: overwritten

text: with

text: references

text: to

text: the

text: variable

text: A:

text: chosen

text: freely

text: but

text: is

text: instead

text: ﬁxed

text: entirely

text: by

text: the

text: value

text: A

text: C

text: output

text: by

text: the

text: distant

text: measurement

text: device.

text: p

text: iﬀ

text: p(ab

text: x)

text: p(λ)p(a

text: xλ)p(b

text: aλ)

text: Instr

text: λ

text: T

text: Q

text: We

text: remark

text: that

text: testing

text: for

text: membership

text: in

text: by

text: p

text: iﬀ

text: p(ab

text: x)

text: tr

text: ρ

text: E

text: F

text: T

text: Instr

text: G

text: Instr

text: a

text: x|

text: b

text: a

text: admitting

text: an

text: extension

text: to

text: Bell

text: per

text: generalizes

text: to

text: p

text: Instr

text: iﬀ

text: p(ab

text: x)

text: (ea|x

text: (eb|a

text: Ψ)

text: any

text: well

text: deﬁned

text: correlation

text: set

text: in

text: the

text: Bell

text: scenario.

text: For

text: example,

text: one

text: might

text: consider

text: relaxations

text: of

text: the

text: quantum

text: Even

text: though

text: the

text: Quantum-

text: and

text: GPT-Bell-Scenario

text: sets

text: set

text: corresponding

text: to

text: levels

text: of

text: the

text: Navascu´es-Pironio-Ac´ın

text: are

text: in

text: general

text: distinct

text: from

text: the

text: Classical-Bell-Scenario

text: (NPA)

text: hierarchy

text: including

text: the

text: set

text: known

text: as

text: Al-

text: set,

text: it

text: may

text: be

text: that

text: their

text: postselections

text: (deﬁning

text: the

text: cor-

text: most

text: Quantum

text: Correlations

text: or

text: tests

text: for

text: compatibil-

text: responding

text: Instrumental-Scenario

text: correlation

text: sets)

text: all

text: co-

text: ity

text: with

text: a

text: restricted

text: Hilbert

text: space

text: dimension

text: All

text: incide.

text: Indeed,

text: it

text: is

text: well

text: known

text: that

text: postselection

text: of

text: those

text: correlation

text: membership

text: tests

text: for

text: the

text: Bell

text: scenario

text: can

text: be

text: applied

text: to

text: the

text: Instrumental

text: scenario

text: by

text: simply

text: introducing

text: existential

text: quantiﬁers:

text: Does

text: an

text: extension

text: of

text: T

text: p(ab

text: x)

text: p(ab

text: x,

text: y=a)

text: exist

text: to

text: p(ab

text: x,

text: y=a)

text: such

text: that

text: the

text: The

text: rTecognition

text: that

text: every

text: Instr

text: correlation

text: must

text: be

text: compatible

text: relevant

text: condition

text: for

text: membership

text: in

text: T

text: is

text: satisﬁed?

text: with

text: is

text: an

text: example

text: of

text: a

text: device-independent

text: causal

text: infer-

text: Bell

text: Bell

text: ence

text: technique

text: which

text: we

text: call

text: Interruption.

text: Relating

text: pairs

text: of

text: This

text: modiﬁcation

text: is

text: especially

text: easy

text: for

text: testing

text: NPA-level

text: DAGs

text: in

text: this

text: manner

text: is

text: explored

text: more

text: generally

text: in

text: a

text: forthcoming

text: compatibility,

text: as

text: those

text: semideﬁnite

text: tests

text: already

text: natively

text: manuscript,

text: currently

text: in

text: preparation

text: support

text: unspeciﬁed

text: probabilities.

text: ##LTFigure##

text: ##LTLine##


text: Magnitude Phase subnetwork subnetwork Phase Linear, 321 Prediction Enhanced Audio L2 normalization 11DD R 1K R eKDK=es= =5siC5d5i,do ,u,1 unCa15avl=5 3 lb 31B6blo65llooc3cck6kk,, + S=[1, ...0.5, 1, …0.5, ...1] x15 Linear, 642 1D Conv Block: AP=UASv=eUrapgsea mPpoloeling Concatenate over channels x6 11DD 1R RKKD eK=e= sC=s55id5io,,d u,nC1 ua1v5a=l5 3l1Bb 36b0ll6ool2occ4ckkk,,, + x10 11DD R1 RK KDeKe==s =Cs5i5d5i,o,dS u, 1n Cua=15va=l15 3 lb1B 36bl56loloo3ccc6kkk,, 11DD R1 RKS KDeKe===s =Cs5[i5d15i,o,d u,, 12n Cua15,va=l15 3 lb1B, 362bl56lo,loo31ccc6]kkk,, x5 σ S = 1 ISTFT DS Conv1D (K,C,S) Concatenate over channelsx US/AP BN Linear, 1536 Linear, 512 Linear, 512 ReLU 3D Resnet Noisy magnitude Spectrogram Magnitude Noisy Phase Prediction Spectrogram STFT Speaker Video Noisy Audio Figure 2: Audio-visual enhancement network. BN: Batch Normalization, C: number of Channels; K: kernel width; S: strides – fractional ones denote transposed convolutions. The network consists of a magnitude and a phase sub-network. The basic building unit is a temporal convolutional block with pre-activation [35] shown on the left. Identity skip connections are added after every convolution layer (and speed up training). All convolutional layers have 1536 channels in the magnitude sub-network and 1024 in the phase subnetwork. Depth-wise separable convolution layers [36] are used, which consist of a separate convolution along the time dimension for every channel, followed by a position-wise projection on the channel dimensions (equivalent to a convolution with kernel width 1). ground truth. The phase prediction is obtained by maximizing aration criteria described in [40]. We use the implementation the cosine similarity between the phase prediction and ground provided by the PEASS toolkit [41]. The metrics are derived by truth, scaled by the ground truth magnitudes. The overall opti- decomposing the noise present in the enhanced signal into com- L || − ∗|| − ∗(cid:12) ∗ misation objective is = Mˆ M M < Φˆ, Φ >. ponents accounting for different deformations and measuring 1 3. Experiments energy ratios between them. In particular the Signal to Interfer- ence Ratio (SIR) measures how well the unwanted signals have 3.1. Datasets been suppressed, the Signal to Artefacts Ratio (SAR) accounts The model is trained on two datasets: the ﬁrst is the for the introduction of artefacts by the enhancement process, BBC-Oxford Multi-View Lip Reading Sentences (MV-LRS) and the Signal to Distortion Ratio (SDR) is an overall quality dataset [32, 37], which contains thousands of sentences from measure, taking both of the above into account. BBC programs such as Doctors and EastEnders; the second is Additionally, we use an ASR system to test for the intel- VoxCeleb2 [38], which contains over a million utterances spo- ligibility of the enhanced speech. For this, we use the Google ken by over 6,000 different speakers. Speech Recognition interface, and report the Word Error Rates The only assumption we make on the datasets is that the (WER) on the clean, mixed and generated audio samples. audio and video are properly synchronized. Where this is not 3.4. Training the case (e.g. TV broadcast), we use the pipeline described in We pre-train the spatio-temporal visual front-end on a [39] to perform and track active speakers and synchronize the word-level lip reading task, following [31]. Like the authors, video and the audio. we ﬁrst train on the LRW dataset, consisting of approximately 3.2. Experimental setup 500K single-word utterances over a vocabulary of 500 words. The MV-LRS dataset is divided into training and test sets This dataset covers only near-frontal poses so we further train by broadcast date, in order to ensure that there is no overlap- on an internal multi-view dataset of a similar size. To accel- ping video between the sets. The dataset covers a large number erate the training process, we pre-compute and save the visual of speakers, which encourages the trained model to be speaker features for all the videos in our dataset. We also compute and agnostic. However, since no identity labels are provided with save the magnitude and phase spectrograms for both the clean the dataset, there may be some overlapping speakers between and noise audio. the sets. The ground truth transcriptions are provided with the The training takes place in 3 phases: ﬁrst, we train the mag- dataset, which allows us to perform quantitative tests on the in- nitude prediction sub-network, following a curriculum which telligibility of the generated audio. starts with high SNR inputs (i.e. only one additional speaker) The VoxCeleb2 dataset lacks the text transcriptions, how- and then progressively move to more challenging examples with ever the dataset is divided into training and test sets by iden- a greater number of speakers; second, we freeze the magnitude tity, which allows us to test the model explicitly for speaker- sub-network and train only the phase network; ﬁnally, we ﬁne- independent performance. tune the whole network end-to-end. We examine scenarios where we add 1 to 4 extra interfer- To generate training examples we ﬁrst select a reference ence speakers on the clean signal, therefore we generate signals with 2 to 5 speakers in total. It should be noted that the task pair of visual and audio features (vr, ar) by randomly sampling a 60-frame clean segment, making sure that the audio and visual of separating the voice of multiple speakers with equal average features correspond and are correctly aligned. We then sample “loudness” is more challenging than separating the speech sig- ∈ nal from ambient background noise. N noise spectrograms xn, n [1, N], and mix them with the 3.3. Evaluation Protocol reference spectrogram in the frequency domain by summing up We evaluate the enhancement performance of the model in the complex spectra, obtaining the mixed spectrogram a . This m terms of perceptual speech quality using the blind source sep- is a natural way to augment our training data since a different ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: Magnitude

label: paragraph

font: default

text: Phase

text: subnetwork

text: subnetwork

text: Phase

text: Linear,

text: Prediction

text: Enhanced

text: Audio

text: normalization

text: R

text: R

text: eKDK=es=

text: lb

text: Linear,

text: Conv

text: Block:

text: AP=UASv=eUrapgsea

text: mPpoloeling

text: Concatenate

text: over

text: channels

text: RKKD

text: eK=e=

text: RK

text: KDeKe==s

text: u,

text: RKS

text: KDeKe===s

text: u,,

text: σ

text: S

text: ISTFT

text: DS

text: (K,C,S)

text: Concatenate

text: over

text: channelsx

text: US/AP

text: BN

text: Linear,

text: Linear,

text: Linear,

text: ReLU

text: Resnet

text: Noisy

text: magnitude

text: Spectrogram

text: Magnitude

text: Noisy

text: Phase

text: Prediction

text: Spectrogram

text: STFT

text: Speaker

text: Video

text: Noisy

text: Audio

text: Figure

text: Audio-visual

text: enhancement

text: network.

text: BN:

text: Batch

text: Normalization,

text: C:

text: number

text: of

text: Channels;

text: K:

text: kernel

text: width;

text: S:

text: strides

text: fractional

text: ones

text: denote

text: transposed

text: convolutions.

text: The

text: network

text: consists

text: of

text: a

text: magnitude

text: and

text: a

text: phase

text: sub-network.

text: The

text: basic

text: building

text: unit

text: is

text: a

text: temporal

text: convolutional

text: block

text: with

text: pre-activation

text: shown

text: on

text: the

text: left.

text: Identity

text: skip

text: connections

text: are

text: added

text: after

text: every

text: convolution

text: layer

text: (and

text: speed

text: up

text: training).

text: All

text: convolutional

text: layers

text: have

text: channels

text: in

text: the

text: magnitude

text: sub-network

text: and

text: in

text: the

text: phase

text: subnetwork.

text: Depth-wise

text: separable

text: convolution

text: layers

text: are

text: used,

text: which

text: consist

text: of

text: a

text: separate

text: convolution

text: along

text: the

text: time

text: dimension

text: for

text: every

text: channel,

text: followed

text: by

text: a

text: position-wise

text: projection

text: on

text: the

text: channel

text: dimensions

text: (equivalent

text: to

text: a

text: convolution

text: with

text: kernel

text: width

text: ground

text: truth.

text: The

text: phase

text: prediction

text: is

text: obtained

text: by

text: maximizing

text: aration

text: criteria

text: described

text: in

text: We

text: use

text: the

text: implementation

text: the

text: cosine

text: similarity

text: between

text: the

text: phase

text: prediction

text: and

text: ground

text: provided

text: by

text: the

text: PEASS

text: toolkit

text: The

text: metrics

text: are

text: derived

text: by

text: truth,

text: scaled

text: by

text: the

text: ground

text: truth

text: magnitudes.

text: The

text: overall

text: opti-

text: decomposing

text: the

text: noise

text: present

text: in

text: the

text: enhanced

text: signal

text: into

text: com-

text: L

text: misation

text: objective

text: is

text: Mˆ

text: M

text: M

text: Φˆ,

text: Φ

text: ponents

text: accounting

text: for

text: different

text: deformations

text: and

text: measuring

text: Experiments

text: energy

text: ratios

text: between

text: them.

text: In

text: particular

text: the

text: Signal

text: to

text: Interfer-

text: ence

text: Ratio

text: (SIR)

text: measures

text: how

text: well

text: the

text: unwanted

text: signals

text: have

text: Datasets

text: been

text: suppressed,

text: the

text: Signal

text: to

text: Artefacts

text: Ratio

text: (SAR)

text: accounts

text: The

text: model

text: is

text: trained

text: on

text: two

text: datasets:

text: the

text: ﬁrst

text: is

text: the

text: for

text: the

text: introduction

text: of

text: artefacts

text: by

text: the

text: enhancement

text: process,

text: BBC-Oxford

text: Multi-View

text: Lip

text: Reading

text: Sentences

text: (MV-LRS)

text: and

text: the

text: Signal

text: to

text: Distortion

text: Ratio

text: (SDR)

text: is

text: an

text: overall

text: quality

text: dataset

text: which

text: contains

text: thousands

text: of

text: sentences

text: from

text: measure,

text: taking

text: both

text: of

text: the

text: above

text: into

text: account.

text: BBC

text: programs

text: such

text: as

text: Doctors

text: and

text: EastEnders;

text: the

text: second

text: is

text: Additionally,

text: we

text: use

text: an

text: ASR

text: system

text: to

text: test

text: for

text: the

text: intel-

text: which

text: contains

text: over

text: a

text: million

text: utterances

text: spo-

text: ligibility

text: of

text: the

text: enhanced

text: speech.

text: For

text: this,

text: we

text: use

text: the

text: Google

text: ken

text: by

text: over

text: different

text: speakers.

text: Speech

text: Recognition

text: interface,

text: and

text: report

text: the

text: Word

text: Error

text: Rates

text: The

text: only

text: assumption

text: we

text: make

text: on

text: the

text: datasets

text: is

text: that

text: the

text: (WER)

text: on

text: the

text: clean,

text: mixed

text: and

text: generated

text: audio

text: samples.

text: audio

text: and

text: video

text: are

text: properly

text: synchronized.

text: Where

text: this

text: is

text: not

text: Training

text: the

text: case

text: (e.g.

text: TV

text: broadcast),

text: we

text: use

text: the

text: pipeline

text: described

text: in

text: We

text: pre-train

text: the

text: spatio-temporal

text: visual

text: front-end

text: on

text: a

text: to

text: perform

text: and

text: track

text: active

text: speakers

text: and

text: synchronize

text: the

text: word-level

text: lip

text: reading

text: task,

text: following

text: Like

text: the

text: authors,

text: video

text: and

text: the

text: audio.

text: we

text: ﬁrst

text: train

text: on

text: the

text: LRW

text: dataset,

text: consisting

text: of

text: approximately

text: Experimental

text: setup

text: single-word

text: utterances

text: over

text: a

text: vocabulary

text: of

text: words.

text: The

text: MV-LRS

text: dataset

text: is

text: divided

text: into

text: training

text: and

text: test

text: sets

text: This

text: dataset

text: covers

text: only

text: near-frontal

text: poses

text: so

text: we

text: further

text: train

text: by

text: broadcast

text: date,

text: in

text: order

text: to

text: ensure

text: that

text: there

text: is

text: no

text: overlap-

text: on

text: an

text: internal

text: multi-view

text: dataset

text: of

text: a

text: similar

text: size.

text: To

text: accel-

text: ping

text: video

text: between

text: the

text: sets.

text: The

text: dataset

text: covers

text: a

text: large

text: number

text: erate

text: the

text: training

text: process,

text: we

text: pre-compute

text: and

text: save

text: the

text: visual

text: of

text: speakers,

text: which

text: encourages

text: the

text: trained

text: model

text: to

text: be

text: speaker

text: features

text: for

text: all

text: the

text: videos

text: in

text: our

text: dataset.

text: We

text: also

text: compute

text: and

text: agnostic.

text: However,

text: since

text: no

text: identity

text: labels

text: are

text: provided

text: with

text: save

text: the

text: magnitude

text: and

text: phase

text: spectrograms

text: for

text: both

text: the

text: clean

text: the

text: dataset,

text: there

text: may

text: be

text: some

text: overlapping

text: speakers

text: between

text: and

text: noise

text: audio.

text: the

text: sets.

text: The

text: ground

text: truth

text: transcriptions

text: are

text: provided

text: with

text: the

text: The

text: training

text: takes

text: place

text: in

text: phases:

text: ﬁrst,

text: we

text: train

text: the

text: mag-

text: dataset,

text: which

text: allows

text: us

text: to

text: perform

text: quantitative

text: tests

text: on

text: the

text: in-

text: nitude

text: prediction

text: sub-network,

text: following

text: a

text: curriculum

text: which

text: telligibility

text: of

text: the

text: generated

text: audio.

text: starts

text: with

text: high

text: SNR

text: inputs

text: (i.e.

text: only

text: one

text: additional

text: speaker)

text: The

text: dataset

text: lacks

text: the

text: text

text: transcriptions,

text: how-

text: and

text: then

text: progressively

text: move

text: to

text: more

text: challenging

text: examples

text: with

text: ever

text: the

text: dataset

text: is

text: divided

text: into

text: training

text: and

text: test

text: sets

text: by

text: iden-

text: a

text: greater

text: number

text: of

text: speakers;

text: second,

text: we

text: freeze

text: the

text: magnitude

text: tity,

text: which

text: allows

text: us

text: to

text: test

text: the

text: model

text: explicitly

text: for

text: speaker-

text: sub-network

text: and

text: train

text: only

text: the

text: phase

text: network;

text: ﬁnally,

text: we

text: ﬁne-

text: independent

text: performance.

text: tune

text: the

text: whole

text: network

text: end-to-end.

text: We

text: examine

text: scenarios

text: where

text: we

text: add

text: to

text: extra

text: interfer-

text: To

text: generate

text: training

text: examples

text: we

text: ﬁrst

text: select

text: a

text: reference

text: ence

text: speakers

text: on

text: the

text: clean

text: signal,

text: therefore

text: we

text: generate

text: signals

text: with

text: to

text: speakers

text: in

text: total.

text: It

text: should

text: be

text: noted

text: that

text: the

text: task

text: pair

text: of

text: visual

text: and

text: audio

text: features

text: (vr,

text: ar)

text: by

text: randomly

text: sampling

text: a

text: clean

text: segment,

text: making

text: sure

text: that

text: the

text: audio

text: and

text: visual

text: of

text: separating

text: the

text: voice

text: of

text: multiple

text: speakers

text: with

text: equal

text: average

text: features

text: correspond

text: and

text: are

text: correctly

text: aligned.

text: We

text: then

text: sample

text: “loudness”

text: is

text: more

text: challenging

text: than

text: separating

text: the

text: speech

text: sig-

text: nal

text: from

text: ambient

text: background

text: noise.

text: N

text: noise

text: spectrograms

text: xn,

text: n

text: N],

text: and

text: mix

text: them

text: with

text: the

text: Evaluation

text: Protocol

text: reference

text: spectrogram

text: in

text: the

text: frequency

text: domain

text: by

text: summing

text: up

text: We

text: evaluate

text: the

text: enhancement

text: performance

text: of

text: the

text: model

text: in

text: the

text: complex

text: spectra,

text: obtaining

text: the

text: mixed

text: spectrogram

text: a

text: This

text: m

text: terms

text: of

text: perceptual

text: speech

text: quality

text: using

text: the

text: blind

text: source

text: sep-

text: is

text: a

text: natural

text: way

text: to

text: augment

text: our

text: training

text: data

text: since

text: a

text: different

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: We use the norm–induced length of each domain to deﬁne a length structure [20, Ch. 2] on M from which we derive F R − R · F · a distance metric. To that end, for γ ∈ Γ, let {r }k be a Ξ(t, x) =( j(cid:48)(t, (x)) D (x)· F j(t, x)) Dgj,j(cid:48)(x) | i i=0 Dg (cid:48)(x) (t, x) countable partition of [0, 1] such that γ is piecewise– R j,j j differentiable for all i ∈ {0, 1, . . . , k −[r1i,}ri.+W1)ith the length + D (x) (11) of γ deﬁned as (cid:88) ≥ ∈ G G ∩ R− D for all t 0 and all x (cid:48) = 1( (cid:48)). j,j j j k | L(γ) = L(γ ), (6) IV. MAIN RESULT [ri,ri+1) i=1 The main contribution of this paper is to provide local the function d : M × M → R≥ deﬁned by conditions under which the distance between any pair of 0 trajectories in a hybrid system (as measured by the intrinsic metric deﬁned in (7)) is globally bounded by an exponential d(x, y) = inf L(γ) (7) envelope. These conditions are made precise in Theorem 1 ∈ γ Γ(x,y) and Corollary 1. In the case when the system satisﬁes a M continuous contraction condition within each domain of the is a distance metric on compatible with the quotient hybrid system as well as a discrete nonexpansion condition topology [14, Thm. 13]. through the reset map between domains, this exponential The ﬁnal required technical assumption is closely related envelope is decreasing in time so that the intrinsic distance to continuity of φ with respect to initial conditions x, as between trajectories decreases exponentially in time, i.e., the claimed in Proposition 1 below (some proofs omitted due to (cid:96) system is contractive. page constraints). Example 1. Consider a hybrid system with two domains in R D D D Assumption 7 (forward–invariance of -connected paths). the positive orthant of the plane so that = with ≥ R D D { ∈ R | ≥ ≥ } L R For all t τ and all piecewise–differentiable -connected = = x 2 x 0 and x 0 , and further take ∈ L R − 1 −2 paths γ Γ, φ(t, τ, γ(r)), interpreted as a function of r, is g (x) = x 1 and g (x) = 1 x so that the system is R R,L 1 L,R D 1 D a piecewise–differentiable -connected path. in the left (resp., right) domain (resp., ) when x < 1 L R R 1 Proposition 1. Under Assumptions 1–7, the ﬂ≥ow φ(t, τ, x) (arnedspx˙.,=x1(cid:20)F>(1x)). =AsAsu(cid:21)mxefothrejr∈ese{tLm, Rap} wiitshthe identity map varies continuously with respect to x for all t τ such that j j − (cid:54)∈ G φ(t , τ, x) . − a 0 ∈ { } It is well–known [21] that Assumptions 1–7 together ensure A = j − , a , b > 0 for j L, R . (12) j 0 b j j that the ﬂow φ is differentiable almost everywhere and, j moreover, its derivative can be computed by solving a jump– D D All executions initialized in ﬂow to and converge to linear–time–varying differential equation as in the following L the origin. Equip both domains with the standard Euclidean Proposition. | | | | | | 2–norm so that x = x = x and consider two L R 2 Proposition 2. Under Assumptions 1–7, given an initial time executions x(t) = φ(t, 0, ξ), z(t) = φ(t, 0, ζ) with initial ≥ R ∈ ∈ D | − | τ 0 and a piecewise–differentiable ≥-connected path γ c| ond|itions ξ, ζ . Th−en d(x(t), z(t)) = x(t) z(t) 2 = Γ, let ψ(t, r) = φ(t, τ, γ(r)) for all t τ and deﬁne e(t) 2 for e(t) = x(t) z(t)∈. WDhen both exec∈ut{ions a}re in the same domain so that x, z for some j L, R , the j error dynamics obey the dynamics of that domain. It there- w(t, r) = Drψ(t, r) (8) fore follows that D d(x, z) ≤ max{−a , −b }d(x, z) so that t j j {− − } the distance decreases at exponential rate max a , b . j j whenever the derivative exists. Then w(τ−, r) = D γ(r) Now suppose x and z are in different domain∈s aDt some and w(·, r) satisﬁes a linear–time–varying differential erqua- tim∈eDt and, without loss of generality, assume x L and tion z R. Writing (cid:20) (cid:21) (cid:20) (cid:21) − D w(t, r) = D F(t, ψ(t, r))w(t, r), ψ(t−, r) ∈ D\G, (9) x = 1 (cid:15)L , z = 1 + (cid:15)R (13) t x x x + δ 2 2 ∈ R with jumps for some (cid:15) , (cid:15) > 0 and δ , we have L R w(t, r) = Ξ(t, ψ(t−, r))w(t−, r), ψ(t−, r) ∈ G, (10) Dt(d(x, z)2) = Dt((x − z)T (x − z)) (14) − − = 2((cid:15) + (cid:15) )(a a ) + 2δx (b b ) L R L R 2 1 2 where Ξ(t, x) is a saltation matrix given by + H.O.T. (15) ##LTLine##

text: We

text: use

text: the

text: norm–induced

text: length

text: of

text: each

text: domain

text: to

text: deﬁne

text: a

text: length

text: structure

text: Ch.

text: on

text: M

text: from

text: which

text: we

text: derive

text: F

label: equation

text: R

text: R

text: F

text: a

text: distance

text: metric.

text: To

text: that

text: end,

text: for

text: γ

text: Γ,

text: let

text: {r

text: }k

text: be

text: a

text: Ξ(t,

text: x)

text: (x))

text: D

text: (x)·

text: F

text: j(t,

text: x))

text: i

text: Dg

text: (t,

text: x)

text: countable

text: partition

text: of

text: such

text: that

text: γ

text: is

text: piecewise–

text: R

text: j,j

text: j

text: differentiable

text: for

text: all

text: i

text: k

text: the

text: length

text: D

text: (x)

text: of

text: γ

text: deﬁned

text: as

text: G

text: G

text: R−

text: D

text: for

text: all

text: t

text: and

text: all

text: x

text: j,j

text: j

text: j

text: k

text: L(γ)

text: L(γ

text: IV.

text: MAIN

text: RESULT

text: The

text: main

text: contribution

text: of

text: this

text: paper

text: is

text: to

text: provide

text: local

text: the

text: function

text: d

text: M

text: M

text: R≥

text: deﬁned

text: by

text: conditions

text: under

text: which

text: the

text: distance

text: between

text: any

text: pair

text: of

text: trajectories

text: in

text: a

text: hybrid

text: system

text: (as

text: measured

text: by

text: the

text: intrinsic

text: metric

text: deﬁned

text: in

text: is

text: globally

text: bounded

text: by

text: an

text: exponential

text: d(x,

text: y)

text: inf

text: L(γ)

text: envelope.

text: These

text: conditions

text: are

text: made

text: precise

text: in

text: Theorem

text: γ

text: Γ(x,y)

text: and

text: Corollary

text: In

text: the

text: case

text: when

text: the

text: system

text: satisﬁes

text: a

text: M

text: continuous

text: contraction

text: condition

text: within

text: each

text: domain

text: of

text: the

text: is

text: a

text: distance

text: metric

text: on

text: compatible

text: with

text: the

text: quotient

text: hybrid

text: system

text: as

text: well

text: as

text: a

text: discrete

text: nonexpansion

text: condition

text: topology

text: Thm.

text: through

text: the

text: reset

text: map

text: between

text: domains,

text: this

text: exponential

text: The

text: ﬁnal

text: required

text: technical

text: assumption

text: is

text: closely

text: related

text: envelope

text: is

text: decreasing

text: in

text: time

text: so

text: that

text: the

text: intrinsic

text: distance

text: to

text: continuity

text: of

text: φ

text: with

text: respect

text: to

text: initial

text: conditions

text: x,

text: as

text: between

text: trajectories

text: decreases

text: exponentially

text: in

text: time,

text: i.e.,

text: the

text: claimed

text: in

text: Proposition

text: below

text: (some

text: proofs

text: omitted

text: due

text: to

text: system

text: is

text: contractive.

text: page

text: constraints).

text: Example

text: Consider

text: a

text: hybrid

text: system

text: with

text: two

text: domains

text: in

text: R

text: D

text: D

text: D

text: Assumption

text: (forward–invariance

text: of

text: -connected

text: paths).

text: the

text: positive

text: orthant

text: of

text: the

text: plane

text: so

text: that

text: with

text: R

text: D

text: D

text: R

text: L

text: R

text: For

text: all

text: t

text: τ

text: and

text: all

text: piecewise–differentiable

text: -connected

text: x

text: x

text: and

text: x

text: and

text: further

text: take

text: L

text: R

text: −2

text: paths

text: γ

text: Γ,

text: φ(t,

text: τ,

text: γ(r)),

text: interpreted

text: as

text: a

text: function

text: of

text: r,

text: is

text: g

text: (x)

text: x

text: and

text: g

text: (x)

text: x

text: so

text: that

text: the

text: system

text: is

text: R

text: R,L

text: L,R

text: D

text: D

text: a

text: piecewise–differentiable

text: -connected

text: path.

text: in

text: the

text: left

text: (resp.,

text: right)

text: domain

text: (resp.,

text: when

text: x

text: L

text: R

text: R

text: Proposition

text: Under

text: Assumptions

text: the

text: ﬂ≥ow

text: φ(t,

text: τ,

text: x)

text: Rap}

text: wiitshthe

text: identity

text: map

text: varies

text: continuously

text: with

text: respect

text: to

text: x

text: for

text: all

text: t

text: τ

text: such

text: that

text: j

text: j

text: G

text: φ(t

text: τ,

text: x)

text: a

text: It

text: is

text: well–known

text: that

text: Assumptions

text: together

text: ensure

text: A

text: j

text: a

text: b

text: for

text: j

text: L,

text: R

text: j

text: b

text: j

text: j

text: that

text: the

text: ﬂow

text: φ

text: is

text: differentiable

text: almost

text: everywhere

text: and,

text: j

text: moreover,

text: its

text: derivative

text: can

text: be

text: computed

text: by

text: solving

text: a

text: jump–

text: D

text: D

text: All

text: executions

text: initialized

text: in

text: ﬂow

text: to

text: and

text: converge

text: to

text: linear–time–varying

text: differential

text: equation

text: as

text: in

text: the

text: following

text: L

text: the

text: origin.

text: Equip

text: both

text: domains

text: with

text: the

text: standard

text: Euclidean

text: Proposition.

text: so

text: that

text: x

text: x

text: x

text: and

text: consider

text: two

text: L

text: R

text: Proposition

text: Under

text: Assumptions

text: given

text: an

text: initial

text: time

text: executions

text: x(t)

text: φ(t,

text: ξ),

text: z(t)

text: φ(t,

text: ζ)

text: with

text: initial

text: R

text: D

text: τ

text: and

text: a

text: piecewise–differentiable

text: ≥-connected

text: path

text: γ

text: c|

text: ond|itions

text: ξ,

text: ζ

text: Th−en

text: d(x(t),

text: z(t))

text: x(t)

text: z(t)

text: Γ,

text: let

text: ψ(t,

text: r)

text: φ(t,

text: τ,

text: γ(r))

text: for

text: all

text: t

text: τ

text: and

text: deﬁne

text: e(t)

text: for

text: e(t)

text: x(t)

text: z(t)∈.

text: WDhen

text: both

text: exec∈ut{ions

text: a}re

text: in

text: the

text: same

text: domain

text: so

text: that

text: x,

text: z

text: for

text: some

text: j

text: L,

text: R

text: the

text: j

text: error

text: dynamics

text: obey

text: the

text: dynamics

text: of

text: that

text: domain.

text: It

text: there-

text: w(t,

text: r)

text: Drψ(t,

text: r)

text: fore

text: follows

text: that

text: D

text: d(x,

text: z)

text: max{−a

text: −b

text: }d(x,

text: z)

text: so

text: that

text: t

text: j

text: j

text: the

text: distance

text: decreases

text: at

text: exponential

text: rate

text: max

text: a

text: b

text: j

text: j

text: whenever

text: the

text: derivative

text: exists.

text: Then

text: w(τ−,

text: r)

text: D

text: γ(r)

text: Now

text: suppose

text: x

text: and

text: z

text: are

text: in

text: different

text: domain∈s

text: aDt

text: some

text: and

text: w(·,

text: r)

text: satisﬁes

text: a

text: linear–time–varying

text: differential

text: erqua-

text: tim∈eDt

text: and,

text: without

text: loss

text: of

text: generality,

text: assume

text: x

text: L

text: and

text: tion

text: z

text: R.

text: Writing

text: D

text: w(t,

text: r)

text: D

text: F(t,

text: ψ(t,

text: r))w(t,

text: r),

text: ψ(t−,

text: r)

text: D\G,

text: x

text: z

text: t

text: x

text: x

text: x

text: δ

text: R

text: with

text: jumps

text: for

text: some

text: and

text: δ

text: we

text: have

text: L

text: R

text: w(t,

text: r)

text: Ξ(t,

text: ψ(t−,

text: r))w(t−,

text: r),

text: ψ(t−,

text: r)

text: G,

text: Dt(d(x,

text: Dt((x

text: z)T

text: (x

text: z))

text: )(a

text: a

text: (b

text: b

text: L

text: R

text: L

text: R

text: where

text: Ξ(t,

text: x)

text: is

text: a

text: saltation

text: matrix

text: given

text: by

text: H.O.T.

text: ##LTLine##


text: Text2Colors: Guiding Image Colorization through Text-Driven Palette Generation Wonwoong Cho*1, Hyojin Bahng*1, David K. Park*1, Seungjoo Yoo*1 *Equal contribution, random order Ziming Wu2, Xiaojuan Ma2, and Jaegul Choo1 1Korea University { } tyflehd21,hjj552,heykeetae,seungjooyoo,jchoo @korea.ac.kr 2Hong Kong University of Science and Technology zwual@connect.ust.hk, mxj@cse.ust.hk Abstract. In this paper, we propose a novel approach to generate multi- ple color palettes that reﬂect the semantics of input text and then colorize a given grayscale image according to the generated color palette. In con- trast to existing approaches, our model can understand rich text, whether it is a single word, a phrase, or a sentence, and generate multiple possible palettes from it. To achieve this task, we introduce our manually curated dataset called Palette-and-Text (PAT), which consists of 10,183 pairs of text and its corresponding color palette. Our proposed model consists of two conditional generative adversarial networks: the text-to-palette generation networks and the palette-based colorization networks. The former employs a sequence-to-sequence model with an attention module to capture the semantics of the text input and produce relevant color palettes. The latter utilizes a U-Net architecture to colorize a grayscale image using the generated color palette. Our evaluation results show that people preferred our generated palettes over ground truth palettes and that our model can eﬀectively reﬂect the given palette when colorizing an image. Keywords: Color Palette Generation, Image Colorization, Conditional Generative Adversarial Networks 1 Introduction Humans can associate certain words with certain colors. The real question is, can machines eﬀectively learn the relationship between colors and texts? Previ- ous approaches have attempted to match a single word with a single color [1–4]. Since multi-color palettes are highly expressive in conveying semantics [5], other studies have further tried to link single words with multi-color palettes [6, 7]. However, these methods only take single words as input and can only recom- mend color palettes in pre-existing datasets. Compared to previous works, our model can generate a whole new palette of diverse colors when given rich text

label: title

text: Guiding

text: Image

text: Colorization

text: through

text: Text-Driven

text: Palette

text: Generation

text: Wonwoong

label: author

text: Hyojin

text: David

text: K.

text: Seungjoo

text: *Equal

text: contribution,

text: random

text: order

text: Ziming

text: Xiaojuan

text: and

text: Jaegul

text: University

label: footer

text: @korea.ac.kr

text: Kong

text: University

text: of

text: Science

text: and

text: Technology

text: zwual@connect.ust.hk,

text: mxj@cse.ust.hk

text: Abstract.

text: In

label: abstract

text: this

text: paper,

text: we

text: propose

text: a

text: novel

text: approach

text: to

text: generate

text: multi-

text: ple

text: color

text: palettes

text: that

text: reﬂect

text: the

text: semantics

text: of

text: input

text: text

text: and

text: then

text: colorize

text: a

text: given

text: grayscale

text: image

text: according

text: to

text: the

text: generated

text: color

text: palette.

text: In

text: con-

text: trast

text: to

text: existing

text: approaches,

text: our

text: model

text: can

text: understand

text: rich

text: text,

text: whether

text: it

text: is

text: a

text: single

text: word,

text: a

text: phrase,

text: or

text: a

text: sentence,

text: and

text: generate

text: multiple

text: possible

text: palettes

text: from

text: it.

text: To

text: achieve

text: this

text: task,

text: we

text: introduce

text: our

text: manually

text: curated

text: dataset

text: called

text: Palette-and-Text

text: (PAT),

text: which

text: consists

text: of

text: pairs

text: of

text: text

text: and

text: its

text: corresponding

text: color

text: palette.

text: Our

text: proposed

text: model

text: consists

text: of

text: two

text: conditional

text: generative

text: adversarial

text: networks:

text: the

text: text-to-palette

text: generation

text: networks

text: and

text: the

text: palette-based

text: colorization

text: networks.

text: The

text: former

text: employs

text: a

text: sequence-to-sequence

text: model

text: with

text: an

text: attention

text: module

text: to

text: capture

text: the

text: semantics

text: of

text: the

text: text

text: input

text: and

text: produce

text: relevant

text: color

text: palettes.

text: The

text: latter

text: utilizes

text: a

text: U-Net

text: architecture

text: to

text: colorize

text: a

text: grayscale

text: image

text: using

text: the

text: generated

text: color

text: palette.

text: Our

text: evaluation

text: results

text: show

text: that

text: people

text: preferred

text: our

text: generated

text: palettes

text: over

text: ground

text: truth

text: palettes

text: and

text: that

text: our

text: model

text: can

text: eﬀectively

text: reﬂect

text: the

text: given

text: palette

text: when

text: colorizing

text: an

text: image.

text: Keywords:

text: Color

text: Palette

text: Generation,

text: Image

text: Colorization,

text: Conditional

text: Generative

text: Adversarial

text: Networks

text: Introduction

text: Humans

text: can

text: associate

text: certain

text: words

text: with

text: certain

text: colors.

text: The

text: real

text: question

text: is,

text: can

text: machines

text: eﬀectively

text: learn

text: the

text: relationship

text: between

text: colors

text: and

text: texts?

text: Previ-

text: ous

text: approaches

text: have

text: attempted

text: to

text: match

text: a

text: single

text: word

text: with

text: a

text: single

text: color

text: Since

text: multi-color

text: palettes

text: are

text: highly

text: expressive

text: in

text: conveying

text: semantics

text: other

text: studies

text: have

text: further

text: tried

text: to

text: link

text: single

text: words

text: with

text: multi-color

text: palettes

text: However,

text: these

text: methods

text: only

text: take

text: single

text: words

text: as

text: input

text: and

text: can

text: only

text: recom-

text: mend

text: color

text: palettes

text: in

text: pre-existing

text: datasets.

text: Compared

text: to

text: previous

text: works,

text: our

text: model

text: can

text: generate

text: a

text: whole

text: new

text: palette

text: of

text: diverse

text: colors

text: when

text: given

text: rich

text: text


text: 2 Fig. 1. Our model can produce a diverse selection of palettes for text inputs such as ‘success.’ Users can choose which palette will be used for the ﬁnal colorization output. input, including both single-word and multiple-word descriptions. A diversity of a color palette refers to how diﬀerent the ﬁve colors included in the palette are. In this paper, we propose a novel method to generate multiple color palettes that convey the semantics of rich text and then colorize a given grayscale im- age according to the generated color palette. Perception of color is inherently multimodal [8], meaning that a particular text input can be mapped to multiple possible color palettes. To incorporate such multimodality into our model, our palette generation networks are designed to generate multiple palettes from a single text input. We further apply our generated color palette to the coloriza- tion task. Motivated from previous user-guided colorizations that utilize color hints given by users [9, 10], we design our colorization networks to utilize the palette during the colorization process. Our evaluation demonstrates that the colorized outputs do not only reﬂect the colors in the palette but also convey the semantics of the text input. The contribution of this paper includes: (1) We propose a novel deep neural network architecture that can generate mul- tiple color palettes based on natural-language text input. (2) Our model is able to use the generated palette to produce plausible coloriza- tions of a grayscale image. (3) We introduce our manually curated dataset called Palette-and-Text (PAT), which includes 10,183 pairs of a multi-word text and a multi-color palette. 1 2 Related Work Meanings associated with a color are both innate and learned [11]. For in- stance, red can make us instinctively feel alert, and white is sometimes culturally learned to be associated with purity [11]. Since color has a strong association with high-level semantic concepts [12], producing palettes from text input is useful in aiding artists and designers [5] and allows automatic colorization from 1 Dataset and codes are publicly available at https://github.com/awesome- davian/Text2Colors/ ##LTFigure## ##LTLine##

text: Fig.

text: Our

label: caption

text: model

text: can

text: produce

text: a

text: diverse

text: selection

text: of

text: palettes

text: for

text: text

text: inputs

text: such

text: as

text: ‘success.’

text: Users

text: can

text: choose

text: which

text: palette

text: will

text: be

text: used

text: for

text: the

text: ﬁnal

text: colorization

text: output.

text: input,

text: including

text: both

text: single-word

text: and

text: multiple-word

text: descriptions.

text: A

text: diversity

text: of

text: a

text: color

text: palette

text: refers

text: to

text: how

text: diﬀerent

text: the

text: ﬁve

text: colors

text: included

text: in

text: the

text: palette

text: are.

text: In

text: this

text: paper,

text: we

text: propose

text: a

text: novel

text: method

text: to

text: generate

text: multiple

text: color

text: palettes

text: that

text: convey

text: the

text: semantics

text: of

text: rich

text: text

text: and

text: then

text: colorize

text: a

text: given

text: grayscale

text: im-

text: age

text: according

text: to

text: the

text: generated

text: color

text: palette.

text: Perception

text: of

text: color

text: is

text: inherently

text: multimodal

text: meaning

text: that

text: a

text: particular

text: text

text: input

text: can

text: be

text: mapped

text: to

text: multiple

text: possible

text: color

text: palettes.

text: To

text: incorporate

text: such

text: multimodality

text: into

text: our

text: model,

text: our

text: palette

text: generation

text: networks

text: are

text: designed

text: to

text: generate

text: multiple

text: palettes

text: from

text: a

text: single

text: text

text: input.

text: We

text: further

text: apply

text: our

text: generated

text: color

text: palette

text: to

text: the

text: coloriza-

text: tion

text: task.

text: Motivated

text: from

text: previous

text: user-guided

text: colorizations

text: that

text: utilize

text: color

text: hints

text: given

text: by

text: users

text: we

text: design

text: our

text: colorization

text: networks

text: to

text: utilize

text: the

text: palette

text: during

text: the

text: colorization

text: process.

text: Our

text: evaluation

text: demonstrates

text: that

text: the

text: colorized

text: outputs

text: do

text: not

text: only

text: reﬂect

text: the

text: colors

text: in

text: the

text: palette

text: but

text: also

text: convey

text: the

text: semantics

text: of

text: the

text: text

text: input.

text: The

text: contribution

text: of

text: this

text: paper

text: includes:

text: We

text: propose

text: a

text: novel

text: deep

text: neural

text: network

text: architecture

text: that

text: can

text: generate

text: mul-

text: tiple

text: color

text: palettes

text: based

text: on

text: natural-language

text: text

text: input.

text: Our

text: model

text: is

text: able

text: to

text: use

text: the

text: generated

text: palette

text: to

text: produce

text: plausible

text: coloriza-

text: tions

text: of

text: a

text: grayscale

text: image.

text: We

text: introduce

text: our

text: manually

text: curated

text: dataset

text: called

text: Palette-and-Text

text: (PAT),

text: which

text: includes

text: pairs

text: of

text: a

text: multi-word

text: text

text: and

text: a

text: multi-color

text: palette.

text: Related

text: Work

text: Meanings

text: associated

text: with

text: a

text: color

text: are

text: both

text: innate

text: and

text: learned

text: For

text: in-

text: stance,

text: red

text: can

text: make

text: us

text: instinctively

text: feel

text: alert,

text: and

text: white

text: is

text: sometimes

text: culturally

text: learned

text: to

text: be

text: associated

text: with

text: purity

text: Since

text: color

text: has

text: a

text: strong

text: association

text: with

text: high-level

text: semantic

text: concepts

text: producing

text: palettes

text: from

text: text

text: input

text: is

text: useful

text: in

text: aiding

text: artists

text: and

text: designers

text: and

text: allows

text: automatic

text: colorization

text: from

text: Dataset

text: and

text: codes

text: are

text: publicly

text: available

text: at

text: ##LTFigure##

text: ##LTLine##


text: 10 Fig. 5. Attention analysis: Attention scores measured by the TPN for two text input samples. Each box color (in green) denotes the attention score computed in producing the corresponding color shown on top. The dashed-line boxes indicate the word that each color output attended. Discriminator. As our discriminator D , we use a variant of the DCGAN ar- 1 chitecture [37]. The image and conditioning variable p are concatenated and fed into a series of conv-leaky relu layers to jointly learn features across the image and the palette. Then, it is fed into a fully-connected layer to classify whether the image is real or fake. 4.3 Implementation Details We ﬁrst train D and G of TPN for 500 epochs using the PAT dataset. We 0 0 then train D and G of the PCN for 100 epochs, using the extracted palette 1 1 from a ground truth image. Finally, we use the trained generators G and G 0 1 during test time to colorize a grayscale image with generated palette yˆ from a text input x. All networks are trained using the Adam optimizer [38] with a learning rate of 0.0002. Weights were initialized from a Gaussian distribution with zero mean and standard deviation of 0.05. We set other hyper parameters as δ = 1, λ = 100, and λ = 0.5. H KL 5 Experimental Results This section presents both quantitative and qualitative analyses of our pro- posed model. We evaluate the TPN (Section 1) based on our PAT dataset. For the training of the PCN (Section 1), we use two diﬀerent datasets, CUB-200-2011 (CUB) [39] and ImageNet ILSVRC Object Detection (ImageNet dataset) [40]. 5.1 Analysis on Attention Outputs The attention module (Section 1) plays a role of attending to particular words in text input to predict the most proper colors that match the text input. Fig. 5 illustrates how the predicted colors are inﬂuenced by attention scores. The green-colored boxes show attention scores computed for each word token when predicting each corresponding color in the palette. Higher scores are indicated by dashed-line boxes. We observe that three colors generated by attending to ghoul are all dark and gloomy, while the other two colors attending to fun are ##LTFigure##

text: Fig.

text: Attention

text: analysis:

text: Attention

text: scores

text: measured

text: by

text: the

text: TPN

text: for

text: two

text: text

text: input

text: samples.

text: Each

text: box

text: color

text: (in

text: green)

text: denotes

text: the

text: attention

text: score

text: computed

text: in

text: producing

text: the

text: corresponding

text: color

text: shown

text: on

text: top.

text: The

text: dashed-line

text: boxes

text: indicate

text: the

text: word

text: that

text: each

text: color

text: output

text: attended.

text: Discriminator.

text: As

text: our

text: discriminator

text: D

text: we

text: use

text: a

text: variant

text: of

text: the

text: DCGAN

text: ar-

text: chitecture

text: The

text: image

text: and

text: conditioning

text: variable

text: p

text: are

text: concatenated

text: and

text: fed

text: into

text: a

text: series

text: of

text: conv-leaky

text: relu

text: layers

text: to

text: jointly

text: learn

text: features

text: across

text: the

text: image

text: and

text: the

text: palette.

text: Then,

text: it

text: is

text: fed

text: into

text: a

text: fully-connected

text: layer

text: to

text: classify

text: whether

text: the

text: image

text: is

text: real

text: or

text: fake.

text: Implementation

text: Details

text: We

text: ﬁrst

text: train

text: D

text: and

text: G

text: of

text: TPN

text: for

text: epochs

text: using

text: the

text: PAT

text: dataset.

text: We

text: then

text: train

text: D

text: and

text: G

text: of

text: the

text: PCN

text: for

text: epochs,

text: using

text: the

text: extracted

text: palette

text: from

text: a

text: ground

text: truth

text: image.

text: Finally,

text: we

text: use

text: the

text: trained

text: generators

text: G

text: and

text: G

text: during

text: test

text: time

text: to

text: colorize

text: a

text: grayscale

text: image

text: with

text: generated

text: palette

text: yˆ

text: from

text: a

text: text

text: input

text: x.

text: All

text: networks

text: are

text: trained

text: using

text: the

text: Adam

text: optimizer

text: with

text: a

text: learning

text: rate

text: of

text: Weights

text: were

text: initialized

text: from

text: a

text: Gaussian

text: distribution

text: with

text: zero

text: mean

text: and

text: standard

text: deviation

text: of

text: We

text: set

text: other

text: hyper

text: parameters

text: as

text: δ

text: λ

text: and

text: λ

text: H

text: KL

text: Experimental

text: Results

text: This

text: section

text: presents

text: both

text: quantitative

text: and

text: qualitative

text: analyses

text: of

text: our

text: pro-

text: posed

text: model.

text: We

text: evaluate

text: the

text: TPN

text: (Section

text: based

text: on

text: our

text: PAT

text: dataset.

text: For

text: the

text: training

text: of

text: the

text: PCN

text: (Section

text: we

text: use

text: two

text: diﬀerent

text: datasets,

text: (CUB)

text: and

text: ImageNet

text: ILSVRC

text: Object

text: Detection

text: (ImageNet

text: dataset)

text: Analysis

text: on

text: Attention

text: Outputs

text: The

text: attention

text: module

text: (Section

text: plays

text: a

text: role

text: of

text: attending

text: to

text: particular

text: words

text: in

text: text

text: input

text: to

text: predict

text: the

text: most

text: proper

text: colors

text: that

text: match

text: the

text: text

text: input.

text: Fig.

text: illustrates

text: how

text: the

text: predicted

text: colors

text: are

text: inﬂuenced

text: by

text: attention

text: scores.

text: The

text: green-colored

text: boxes

text: show

text: attention

text: scores

text: computed

text: for

text: each

text: word

text: token

text: when

text: predicting

text: each

text: corresponding

text: color

text: in

text: the

text: palette.

text: Higher

text: scores

text: are

text: indicated

text: by

text: dashed-line

text: boxes.

text: We

text: observe

text: that

text: three

text: colors

text: generated

text: by

text: attending

text: to

text: ghoul

text: are

text: all

text: dark

text: and

text: gloomy,

text: while

text: the

text: other

text: two

text: colors

text: attending

text: to

text: fun

text: are

text: ##LTFigure##


text: 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 1.304 1.306 1.308 1.31 1.312 1.314 1.316 1.318 1.32 1.322 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 1.49 1.5 1.51 1.52 1.53 1.54 1.55 1.56 1.57 1.58 0.06 0.05 0.04 0.03 0.02 0.01 0 1.66 1.68 1.7 1.72 1.74 1.76 1.78 1.8 (cid:101) − FIG. 8. Tests of the scaling in Eq. (6.7): numerical plots for (g g )/T 1/z as a function of χ T 1/2 at c 0 J = 1, t = 1 and k = 1, k = 1.5 and k = 1.7. We see that at z = 0.75, z = 3.1 and z = 5.6 all lines almost exactly overlap, as expected in the critical region. 19 ##LTFigure## ##LTFigure## ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: FIG.

text: Tests

text: of

text: the

text: scaling

text: in

text: Eq.

text: numerical

text: plots

text: for

text: (g

text: g

text: )/T

text: as

text: a

text: function

text: of

text: χ

text: T

text: at

text: c

text: J

text: t

text: and

text: k

text: k

text: and

text: k

text: We

text: see

text: that

text: at

text: z

text: z

text: and

text: z

text: all

text: lines

text: almost

text: exactly

text: overlap,

text: as

text: expected

text: in

text: the

text: critical

text: region.

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: VII. CONCLUSIONS Our paper has presented an exactly solvable model of fractionalization in metallic states in the presence of disorder and interactions. We considered a t-J model in which the electron, c, Z fractionalizes into a fermion f and a boson φ both carrying gauge charges. As the fermions f 2 carries both the global U(1) charge and SU(2) spin of the electron, these fractionalized phases can be considered as realizations of the ‘orthogonal metal’ of Ref. [25]. The phase diagram of our model is schematically presented in Fig. 7. There are two extended phases, separated either by a ﬁrst order transition, or a critical line. In one phase, the boson φ is gapped, while the fermion f is gapless. This implies that the electron ∼ c fφ also has a gapped spectral function. On the other hand, thermodynamic properties are largely controlled by the gapless fermions. We propose this gapped boson phase as a toy model for the pseudogap regime of the cuprates. In the other phase, and also on the critical line, both the fermions and bosons are gapless, | |− and decay with time as τ 2∆f,b, where the values of the exponents are speciﬁed in Fig. 7. In a Z Higgs phase, in which the charges are conﬁned, the boson correlator would decay to a non-zero 2 constant. As the boson decay here is a power-law in time, we labeled this phase as a quasi-Higgs phase. One of the most interesting properties of the quasi-Higgs phase follows from the exponent identity in Eq. (3.5). The Green’s function of the electron operator, c, decays with time as 1/τ (Eq. (5.8)), which is the form of the local Green’s function in a Fermi liquid. This result is a consequence of the relevance of the hopping term, t, in the Hamiltonian which transfers single electrons between sites. Unlike previous SYK models, the present model balances the hopping (t) and interaction (J) terms against each other, rather than one of them dominating; this leads to the Fermi liquid form of the one-electron Green’s function. However, despite this form, most other properties are not Fermi liquid-like e.g. the spin susceptibility is dominated by the response of the f fermions which have an anomalous scaling dimension ∆ . These intruiging properties are f suggestive of the overdoped regime of the cuprates, where there are indications of an extended non-Fermi liquid regime, although photoemission indicates a well-formed large Fermi surface [35– 38]. Extending our toy model to a more realistic model of the cuprates requires introducing spatial structure and examining transport properties. A number of methods of doing so have been intro- ducing recently [7–12] for the SYK model, and it would be interesting to apply these, or others, to models similar to the one presented here. It would also be useful to examine holographic duals of the phases and phase transitions pre- sented here. Given the mapping of the SYK model to AdS gravity [19, 21, 22], and the conformal 2 21

text: VII.

label: section

text: CONCLUSIONS

text: Our

text: paper

text: has

text: presented

text: an

text: exactly

text: solvable

text: model

text: of

text: fractionalization

text: in

text: metallic

text: states

text: in

text: the

text: presence

text: of

text: disorder

text: and

text: interactions.

text: We

text: considered

text: a

text: t-J

text: model

text: in

text: which

text: the

text: electron,

text: c,

text: Z

text: fractionalizes

text: into

text: a

text: fermion

text: f

text: and

text: a

text: boson

text: φ

text: both

text: carrying

text: gauge

text: charges.

text: As

text: the

text: fermions

text: f

text: carries

text: both

text: the

text: global

text: charge

text: and

text: spin

text: of

text: the

text: electron,

text: these

text: fractionalized

text: phases

text: can

text: be

text: considered

text: as

text: realizations

text: of

text: the

text: ‘orthogonal

text: metal’

text: of

text: Ref.

text: The

text: phase

text: diagram

text: of

text: our

text: model

text: is

text: schematically

text: presented

text: in

text: Fig.

text: There

text: are

text: two

text: extended

text: phases,

text: separated

text: either

text: by

text: a

text: ﬁrst

text: order

text: transition,

text: or

text: a

text: critical

text: line.

text: In

text: one

text: phase,

text: the

text: boson

text: φ

text: is

text: gapped,

text: while

text: the

text: fermion

text: f

text: is

text: gapless.

text: This

text: implies

text: that

text: the

text: electron

text: c

text: fφ

text: also

text: has

text: a

text: gapped

text: spectral

text: function.

text: On

text: the

text: other

text: hand,

text: thermodynamic

text: properties

text: are

text: largely

text: controlled

text: by

text: the

text: gapless

text: fermions.

text: We

text: propose

text: this

text: gapped

text: boson

text: phase

text: as

text: a

text: toy

text: model

text: for

text: the

text: pseudogap

text: regime

text: of

text: the

text: cuprates.

text: In

text: the

text: other

text: phase,

text: and

text: also

text: on

text: the

text: critical

text: line,

text: both

text: the

text: fermions

text: and

text: bosons

text: are

text: gapless,

text: and

text: decay

text: with

text: time

text: as

text: τ

text: where

text: the

text: values

text: of

text: the

text: exponents

text: are

text: speciﬁed

text: in

text: Fig.

text: In

text: a

text: Z

text: Higgs

text: phase,

text: in

text: which

text: the

text: charges

text: are

text: conﬁned,

text: the

text: boson

text: correlator

text: would

text: decay

text: to

text: a

text: non-zero

text: constant.

text: As

text: the

text: boson

text: decay

text: here

text: is

text: a

text: power-law

text: in

text: time,

text: we

text: labeled

text: this

text: phase

text: as

text: a

text: quasi-Higgs

text: phase.

text: One

text: of

text: the

text: most

text: interesting

text: properties

text: of

text: the

text: quasi-Higgs

text: phase

text: follows

text: from

text: the

text: exponent

text: identity

text: in

text: Eq.

text: The

text: Green’s

text: function

text: of

text: the

text: electron

text: operator,

text: c,

text: decays

text: with

text: time

text: as

text: (Eq.

text: which

text: is

text: the

text: form

text: of

text: the

text: local

text: Green’s

text: function

text: in

text: a

text: Fermi

text: liquid.

text: This

text: result

text: is

text: a

text: consequence

text: of

text: the

text: relevance

text: of

text: the

text: hopping

text: term,

text: t,

text: in

text: the

text: Hamiltonian

text: which

text: transfers

text: single

text: electrons

text: between

text: sites.

text: Unlike

text: previous

text: SYK

text: models,

text: the

text: present

text: model

text: balances

text: the

text: hopping

text: (t)

text: and

text: interaction

text: (J)

text: terms

text: against

text: each

text: other,

text: rather

text: than

text: one

text: of

text: them

text: dominating;

text: this

text: leads

text: to

text: the

text: Fermi

text: liquid

text: form

text: of

text: the

text: one-electron

text: Green’s

text: function.

text: However,

text: despite

text: this

text: form,

text: most

text: other

text: properties

text: are

text: not

text: Fermi

text: liquid-like

text: e.g.

text: the

text: spin

text: susceptibility

text: is

text: dominated

text: by

text: the

text: response

text: of

text: the

text: f

text: fermions

text: which

text: have

text: an

text: anomalous

text: scaling

text: dimension

text: These

text: intruiging

text: properties

text: are

text: f

text: suggestive

text: of

text: the

text: overdoped

text: regime

text: of

text: the

text: cuprates,

text: where

text: there

text: are

text: indications

text: of

text: an

text: extended

text: non-Fermi

text: liquid

text: regime,

text: although

text: photoemission

text: indicates

text: a

text: well-formed

text: large

text: Fermi

text: surface

text: Extending

text: our

text: toy

text: model

text: to

text: a

text: more

text: realistic

text: model

text: of

text: the

text: cuprates

text: requires

text: introducing

text: spatial

text: structure

text: and

text: examining

text: transport

text: properties.

text: A

text: number

text: of

text: methods

text: of

text: doing

text: so

text: have

text: been

text: intro-

text: ducing

text: recently

text: for

text: the

text: SYK

text: model,

text: and

text: it

text: would

text: be

text: interesting

text: to

text: apply

text: these,

text: or

text: others,

text: to

text: models

text: similar

text: to

text: the

text: one

text: presented

text: here.

text: It

text: would

text: also

text: be

text: useful

text: to

text: examine

text: holographic

text: duals

text: of

text: the

text: phases

text: and

text: phase

text: transitions

text: pre-

text: sented

text: here.

text: Given

text: the

text: mapping

text: of

text: the

text: SYK

text: model

text: to

text: AdS

text: gravity

text: and

text: the

text: conformal


text: Kitaev models,” Journal of High Energy Physics 5, 125 (2017), arXiv:1609.07832 [hep-th]. [8] R. A. Davison, W. Fu, A. Georges, Y. Gu, K. Jensen, and S. Sachdev, “Thermoelectric transport in disordered metals without quasiparticles: The Sachdev-Ye-Kitaev models and holography,” Phys. Rev. B 95, 155131 (2017), arXiv:1612.00849 [cond-mat.str-el]. [9] Y. Gu, A. Lucas, and X.-L. Qi, “Energy diﬀusion and the butterﬂy eﬀect in inhomogeneous Sachdev- Ye-Kitaev chains,” SciPost Phys. 2, 018 (2017), arXiv:1702.08462 [hep-th]. [10] X.-Y. Song, C.-M. Jian, and L. Balents, “Strongly Correlated Metal Built from Sachdev-Ye-Kitaev Models,” Phys. Rev. Lett. 119, 216601 (2017), arXiv:1705.00117 [cond-mat.str-el]. [11] A. A. Patel, J. McGreevy, D. P. Arovas, and S. Sachdev, “Magnetotransport in a model of a disordered strange metal,” ArXiv e-prints (2017), arXiv:1712.05026 [cond-mat.str-el]. [12] D. Chowdhury, Y. Werman, E. Berg, and T. Senthil, “Translationally invariant non-Fermi liquid metals with critical Fermi-surfaces: Solvable models,” ArXiv e-prints (2018), arXiv:1801.06178 [cond- mat.str-el]. [13] J. Maldacena and D. Stanford, “Remarks on the Sachdev-Ye-Kitaev model,” Phys. Rev. D 94, 106002 (2016), arXiv:1604.07818 [hep-th]. [14] J. Maldacena, S. H. Shenker, and D. Stanford, “A bound on chaos,” Journal of High Energy Physics 2016, 106 (2016), arXiv:1503.01409 [hep-th]. [15] S. Sachdev, Quantum Phase Transitions, 1st ed. (Cambridge University Press, Cambridge, UK, 1999). [16] J. Sonner and M. Vielma, “Eigenstate thermalization in the Sachdev-Ye-Kitaev model,” JHEP 11, 149 (2017), arXiv:1707.08013 [hep-th]. [17] J. M. Deutsch, “Quantum statistical mechanics in a closed system,” Phys. Rev. A 43, 2046 (1991). [18] M. Srednicki, “Chaos and quantum thermalization,” Phys. Rev. E 50, 888 (1994), cond-mat/9403051. [19] S. Sachdev, “Holographic metals and the fractionalized Fermi liquid,” Phys. Rev. Lett. 105, 151602 (2010), arXiv:1006.3794 [hep-th]. [20] S. Sachdev, “Strange metals and the AdS/CFT correspondence,” J. Stat. Mech. 1011, P11022 (2010), arXiv:1010.0682 [cond-mat.str-el]. [21] J. Maldacena, D. Stanford, and Z. Yang, “Conformal symmetry and its breaking in two dimensional Nearly Anti-de-Sitter space,” PTEP 2016, 12C104 (2016), arXiv:1606.01857 [hep-th]. [22] A. Kitaev and S. J. Suh, “The soft mode in the Sachdev-Ye-Kitaev model and its gravity dual,” (2017), arXiv:1711.08467 [hep-th]. [23] J. Maldacena, D. Stanford, and Z. Yang, “Diving into traversable wormholes,” Fortsch. Phys. 65, 1700034 (2017), arXiv:1704.05333 [hep-th]. [24] S. Sachdev and D. Chowdhury, “The novel metallic states of the cuprates: Fermi liquids with topo- logical order, and strange metals,” Progress of Theoretical and Experimental Physics 2016, 12C102 32

text: Kitaev

text: models,”

text: Journal

text: of

text: High

text: Energy

text: Physics

text: [hep-th].

text: R.

text: A.

text: Davison,

text: W.

text: Fu,

text: A.

text: Georges,

text: Y.

text: Gu,

text: K.

text: Jensen,

text: and

text: S.

text: Sachdev,

text: “Thermoelectric

text: transport

text: in

text: disordered

text: metals

text: without

text: quasiparticles:

text: The

text: Sachdev-Ye-Kitaev

text: models

text: and

text: holography,”

text: Phys.

text: Rev.

text: B

text: [cond-mat.str-el].

text: Y.

text: Gu,

text: A.

text: Lucas,

text: and

text: X.-L.

text: Qi,

text: “Energy

text: diﬀusion

text: and

text: the

text: butterﬂy

text: eﬀect

text: in

text: inhomogeneous

text: Sachdev-

text: Ye-Kitaev

text: chains,”

text: SciPost

text: Phys.

text: [hep-th].

text: X.-Y.

text: Song,

text: C.-M.

text: Jian,

text: and

text: L.

text: Balents,

text: “Strongly

text: Correlated

text: Metal

text: Built

text: from

text: Sachdev-Ye-Kitaev

text: Models,”

text: Phys.

text: Rev.

text: Lett.

text: [cond-mat.str-el].

text: A.

text: A.

text: Patel,

text: J.

text: McGreevy,

text: D.

text: P.

text: Arovas,

text: and

text: S.

text: Sachdev,

text: “Magnetotransport

text: in

text: a

text: model

text: of

text: a

text: disordered

text: strange

text: metal,”

text: ArXiv

text: e-prints

text: [cond-mat.str-el].

text: D.

text: Chowdhury,

text: Y.

text: Werman,

text: E.

text: Berg,

text: and

text: T.

text: Senthil,

text: “Translationally

text: invariant

text: non-Fermi

text: liquid

text: metals

text: with

text: critical

text: Fermi-surfaces:

text: Solvable

text: models,”

text: ArXiv

text: e-prints

text: [cond-

text: mat.str-el].

text: J.

text: Maldacena

text: and

text: D.

text: Stanford,

text: “Remarks

text: on

text: the

text: Sachdev-Ye-Kitaev

text: model,”

text: Phys.

text: Rev.

text: D

text: [hep-th].

text: J.

text: Maldacena,

text: S.

text: H.

text: Shenker,

text: and

text: D.

text: Stanford,

text: “A

text: bound

text: on

text: chaos,”

text: Journal

text: of

text: High

text: Energy

text: Physics

text: [hep-th].

text: S.

text: Sachdev,

text: Quantum

text: Phase

text: Transitions,

text: ed.

text: (Cambridge

text: University

text: Press,

text: Cambridge,

text: UK,

text: J.

text: Sonner

text: and

text: M.

text: Vielma,

text: “Eigenstate

text: thermalization

text: in

text: the

text: Sachdev-Ye-Kitaev

text: model,”

text: JHEP

text: [hep-th].

text: J.

text: M.

text: Deutsch,

text: “Quantum

text: statistical

text: mechanics

text: in

text: a

text: closed

text: system,”

text: Phys.

text: Rev.

text: A

text: M.

text: Srednicki,

text: “Chaos

text: and

text: quantum

text: thermalization,”

text: Phys.

text: Rev.

text: E

text: S.

text: Sachdev,

text: “Holographic

text: metals

text: and

text: the

text: fractionalized

text: Fermi

text: liquid,”

text: Phys.

text: Rev.

text: Lett.

text: [hep-th].

text: S.

text: Sachdev,

text: “Strange

text: metals

text: and

text: the

text: AdS/CFT

text: correspondence,”

text: J.

text: Stat.

text: Mech.

text: [cond-mat.str-el].

text: J.

text: Maldacena,

text: D.

text: Stanford,

text: and

text: Z.

text: Yang,

text: “Conformal

text: symmetry

text: and

text: its

text: breaking

text: in

text: two

text: dimensional

text: Nearly

text: Anti-de-Sitter

text: space,”

text: PTEP

text: [hep-th].

text: A.

text: Kitaev

text: and

text: S.

text: J.

text: Suh,

text: “The

text: soft

text: mode

text: in

text: the

text: Sachdev-Ye-Kitaev

text: model

text: and

text: its

text: gravity

text: dual,”

text: [hep-th].

text: J.

text: Maldacena,

text: D.

text: Stanford,

text: and

text: Z.

text: Yang,

text: “Diving

text: into

text: traversable

text: wormholes,”

text: Fortsch.

text: Phys.

text: [hep-th].

text: S.

text: Sachdev

text: and

text: D.

text: Chowdhury,

text: “The

text: novel

text: metallic

text: states

text: of

text: the

text: cuprates:

text: Fermi

text: liquids

text: with

text: topo-

text: logical

text: order,

text: and

text: strange

text: metals,”

text: Progress

text: of

text: Theoretical

text: and

text: Experimental

text: Physics


text: III. GAPLESS SOLUTIONS First, we search for solutions of Eqs. (2.4), (2.5), and (2.8) in which both the fermions and the bosons are gapless. In our initial analysis, we will work on the imaginary frequency axis at T = 0 (see Appendix for deﬁnitions of spectral functions). The extension to T > 0 appears in Section . → ∞ For the gapless solutions, we make the ansatzes valid as τ at T = 0 − F G(τ) = sgn(τ) | | (J τ )2∆f C/J χ(τ) = | | , (3.1) (J τ )2∆b where F > 0 and C > 0, and they are both dimensionless. The Fourier transforms at T = 0 are − F/J2∆f − G(iω) = 2i sgn(ω)| | − cos(π∆ )Γ(1 2∆ ) ω 1 2∆f f f C/J2∆b+1 − χ(iω) = 2 |ω|1−2∆b sin(π∆b)Γ(1 2∆b) . (3.2) (cid:32) (cid:33) From Eq. (2.4) and (2.5), the self energies are (cid:101) − J2F 3 k t 2F C2/J2 Σ(τ) = sg(cid:101)n(τ) | | + | | (J τ )6∆f (J τ )2∆f+4∆b 2 t 2F 2C/J P (τ) = | | , (3.3) (J τ )4∆f+2∆b (cid:18) and their Fourier transforms are − − J2 6∆f F 3 − (cid:33) Σ(iω) = 2i sgn(ω)(cid:101) | | − cos(3π∆ )Γ(1 6∆ ) ω 1 6∆f f f − − k(t/J2)2J2 2∆f 4∆bF C2 − − (cid:101) + | | − − cos(π(∆ + 2∆ ))Γ(1 2∆ 4∆ ) ω 1 2∆f 4∆b f b f b − − (t/J2)2J3 4∆f 2∆bF 2C − − P (iω) = 4 | | − − sin(π(2∆ + ∆ ))Γ(1 4∆ 2∆ ) . (3.4) ω 1 4∆f 2∆b f b f b − − From Eqns (3.2) and (3.4), and using G(iω)Σ(iω) = 1 and χ(iω)P (iω) = 1 in the limit of low ω, we see that solutions are only possible when ∆ + ∆ = 1/2 . (3.5) f b Further examination of the saddle point equations shows that two classes of solutions are possible, depending upon whether ∆ > 1/4 or ∆ = 1/4. We will examine these solutions in the following f f subsections. 7 ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: III.

text: GAPLESS

text: SOLUTIONS

text: First,

text: we

text: search

text: for

text: solutions

text: of

text: Eqs.

text: and

text: in

text: which

text: both

text: the

text: fermions

text: and

text: the

text: bosons

text: are

text: gapless.

text: In

text: our

text: initial

text: analysis,

text: we

text: will

text: work

text: on

text: the

text: imaginary

text: frequency

text: axis

text: at

text: T

text: (see

text: Appendix

text: for

text: deﬁnitions

text: of

text: spectral

text: functions).

text: The

text: extension

text: to

text: T

text: appears

text: in

text: Section

text: For

text: the

text: gapless

text: solutions,

text: we

text: make

text: the

text: ansatzes

text: valid

text: as

text: τ

text: at

text: T

text: F

text: G(τ)

text: sgn(τ)

text: (J

text: τ

text: C/J

text: χ(τ)

text: (J

text: τ

text: where

text: F

text: and

text: C

text: and

text: they

text: are

text: both

text: dimensionless.

text: The

text: Fourier

text: transforms

text: at

text: T

text: are

text: G(iω)

text: sgn(ω)|

text: cos(π∆

text: 2∆

text: ω

text: f

text: f

text: χ(iω)

text: From

text: Eq.

text: and

text: the

text: self

text: energies

text: are

text: k

text: t

text: Σ(τ)

text: (J

text: τ

text: (J

text: τ

text: t

text: P

text: (τ)

text: (J

text: τ

text: and

text: their

text: Fourier

text: transforms

text: are

text: F

text: Σ(iω)

text: 6∆

text: ω

text: f

text: f

text: cos(π(∆

text: 2∆

text: 2∆

text: 4∆

text: ω

text: f

text: b

text: f

text: b

text: P

text: (iω)

text: 4∆

text: 2∆

text: ω

text: f

text: b

text: f

text: b

text: From

text: Eqns

text: and

text: and

text: using

text: G(iω)Σ(iω)

text: and

text: χ(iω)P

text: (iω)

text: in

text: the

text: limit

text: of

text: low

text: ω,

text: we

text: see

text: that

text: solutions

text: are

text: only

text: possible

text: when

text: f

text: b

text: Further

text: examination

text: of

text: the

text: saddle

text: point

text: equations

text: shows

text: that

text: two

text: classes

text: of

text: solutions

text: are

text: possible,

text: depending

text: upon

text: whether

text: or

text: We

text: will

text: examine

text: these

text: solutions

text: in

text: the

text: following

text: f

text: f

text: subsections.

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##