text: Evidence for Dynamic Excitation-Inhibition Ratio in Networks of Cortical Neurons Netta Haroush and Shimon Marom Technion — Israel Institute of Technology, Haifa 32000, Israel. In this report trial-to-trial variations in the synchronized responses of neural networks are oﬀered as evidence for excitation-inhibition ratio being a dynamic variable over time scales of minutes. Synchronized network responses to stimuli were studied in ex-vivo large scale cortical networks. We show that sub-second measures of the individual synchronous response, namely – its latency and decay duration, are related to minutes-scale network response dynamics. Network responsiveness is reﬂected as residency in, or shifting amongst, areas of the latency-decay plane. The diﬀerent sensitivities of latency and decay durations to synaptic blockers imply that these two measures reﬂect the eﬀective impacts of inhibitory and excitatory neuronal populations on response dynamics. Taken together, the data suggest that network response variations under repeated stimuli result from excitation-inhibition ratio being a dynamic variable rather than a parameter. Introduction Evoked transient synchronous activity is acknowledged as signiﬁcant in both normal and pathological neural conditions [1]. In mammalian brains, as well as in their ex-vivo reduced experimental preparations (slices, cultured networks), an evoked transient synchronous activity has a temporally-stretched spike like shape in the population ﬁring rate trace, a characteristic time scale (ca. 100 msec), the ﬂavor of a threshold-governed event and a refractory period that lasts several seconds [2–6]. With these properties in mind we adhere to a previously oﬀered terminology and use the name Network Spike (NS) to designate evoked transient synchronous activity [4, 7]. Trial-to-trial variations in the occurrence and the ﬁne structure of stimulus evoked NSs were reported in both anesthetized and behaving animals as well as in reduced ex-vivo preparations [6, 8–11]. The response variations reﬂect a multitude of factors that determine the network excitability status at the time of stimulus arrival. These include neuronal and synaptic noise, refractoriness of neuronal and synaptic activities and the context of ongoing activity within which a stimulus is applied [6, 10, 12, 13]. Here we took advantage of a relatively controlled experimental approach to large-scale cortical networks developing ex-vivo, in order to expose the nature of response variations under repeated input over an extended range of time scales (from milliseconds to many minutes). We provide indications for the existence of a hierarchy of timescales in the structure of trial-to-trial response variations, ranging from sub-seconds to many minutes. We then show that response dynamics over minutes are reducible to the interplay of two instantaneous (i.e. single NS, sub-second scale) observables: (1) the latency from stimulus to the peak of the NS ﬁring rate envelop, and (2) the decay duration from that peak to baseline activity. We show that these two instantaneous measures of the network excitability state—latency and decay duration—are diﬀerentially sensitive to speciﬁc pharmacological blockers of inhibitory and excitatory synaptic transmissions, suggesting that long-term network response variations reﬂect a dynamic excitation-inhibition ratio. Materials and Methods Cell preparation Cortical neurons were obtained from newborn rats (Sprague-Dawley) within 24 hours after birth using mechanical and enzymatic procedures described in earlier studies [14]. The neurons were isolated and plated directly onto substrate- integrated multi electrode arrays. They were allowed to develop into functionally and structurally mature networks over a period of 2 weeks and were used in experiments within the period of 2–6 weeks post plating. The number of plated neurons was in the order of 450,000, covering an area of about 380 mm2 with heat-inactivated horse serum (5%), glutamine (0.5 mM), glucose (20 mM), and gentamycin (10 µg/ml), and maintained in an atmosphere of 37oC, 5% CO and 95% air in an incubator as well as during the recording phases. An array of 60 Ti/Au extracellular 2 electrodes, 30 µm in diameter, spaced 500 µm from each other (MultiChannelSystems, Reutlingen, Germany) was used. The insulation layer (silicon nitride) is pretreated with polyethyleneimine (Sigma, 0.01% in 0.1 M Borate buﬀer solution). Electrophysiology A commercial ampliﬁer (MEA-1060-inv-BC, MCS, Reutlingen, Germany) with frequency limits of 150–3,000 Hz and a

text: Evidence

text: for

text: Dynamic

text: Excitation-Inhibition

text: Ratio

text: in

text: Networks

text: of

text: Cortical

text: Neurons

text: Netta

label: author

text: Haroush

text: and

text: Shimon

text: Marom

text: Technion

text: Israel

text: Institute

text: of

text: Technology,

text: Haifa

text: Israel.

text: In

text: this

text: report

text: trial-to-trial

text: variations

text: in

text: the

text: synchronized

text: responses

text: of

text: neural

text: networks

text: are

text: oﬀered

text: as

text: evidence

text: for

text: excitation-inhibition

text: ratio

text: being

text: a

text: dynamic

text: variable

text: over

text: time

text: scales

text: of

text: minutes.

text: Synchronized

text: network

text: responses

text: to

text: stimuli

text: were

text: studied

text: in

text: ex-vivo

text: large

text: scale

text: cortical

text: networks.

text: We

text: show

text: that

text: sub-second

text: measures

text: of

text: the

text: individual

text: synchronous

text: response,

text: namely

text: its

text: latency

text: and

text: decay

text: duration,

text: are

text: related

text: to

text: minutes-scale

text: network

text: response

text: dynamics.

text: Network

text: responsiveness

text: is

text: reﬂected

text: as

text: residency

text: in,

text: or

text: shifting

text: amongst,

text: areas

text: of

text: the

text: latency-decay

text: plane.

text: The

text: diﬀerent

text: sensitivities

text: of

text: latency

text: and

text: decay

text: durations

text: to

text: synaptic

text: blockers

text: imply

text: that

text: these

text: two

text: measures

text: reﬂect

text: the

text: eﬀective

text: impacts

text: of

text: inhibitory

text: and

text: excitatory

text: neuronal

text: populations

text: on

text: response

text: dynamics.

text: Taken

text: together,

text: the

text: data

text: suggest

text: that

text: network

text: response

text: variations

text: under

text: repeated

text: stimuli

text: result

text: from

text: excitation-inhibition

text: ratio

text: being

text: a

text: dynamic

text: variable

text: rather

text: than

text: a

text: parameter.

text: Introduction

text: Evoked

text: transient

text: synchronous

text: activity

text: is

text: acknowledged

text: as

text: signiﬁcant

text: in

text: both

text: normal

text: and

text: pathological

text: neural

text: conditions

text: In

text: mammalian

text: brains,

text: as

text: well

text: as

text: in

text: their

text: ex-vivo

text: reduced

text: experimental

text: preparations

text: (slices,

text: cultured

text: networks),

text: an

text: evoked

text: transient

text: synchronous

text: activity

text: has

text: a

text: temporally-stretched

text: spike

text: like

text: shape

text: in

text: the

text: population

text: ﬁring

text: rate

text: trace,

text: a

text: characteristic

text: time

text: scale

text: (ca.

text: msec),

text: the

text: ﬂavor

text: of

text: a

text: threshold-governed

text: event

text: and

text: a

text: refractory

text: period

text: that

text: lasts

text: several

text: seconds

text: With

text: these

text: properties

text: in

text: mind

text: we

text: adhere

text: to

text: a

text: previously

text: oﬀered

text: terminology

text: and

text: use

text: the

text: name

text: Network

text: Spike

text: (NS)

text: to

text: designate

text: evoked

text: transient

text: synchronous

text: activity

text: Trial-to-trial

text: variations

text: in

text: the

text: occurrence

text: and

text: the

text: ﬁne

text: structure

text: of

text: stimulus

text: evoked

text: NSs

text: were

text: reported

text: in

text: both

text: anesthetized

text: and

text: behaving

text: animals

text: as

text: well

text: as

text: in

text: reduced

text: ex-vivo

text: preparations

text: The

text: response

text: variations

text: reﬂect

text: a

text: multitude

text: of

text: factors

text: that

text: determine

text: the

text: network

text: excitability

text: status

text: at

text: the

text: time

text: of

text: stimulus

text: arrival.

text: These

text: include

text: neuronal

text: and

text: synaptic

text: noise,

text: refractoriness

text: of

text: neuronal

text: and

text: synaptic

text: activities

text: and

text: the

text: context

text: of

text: ongoing

text: activity

text: within

text: which

text: a

text: stimulus

text: is

text: applied

text: Here

text: we

text: took

text: advantage

text: of

text: a

text: relatively

text: controlled

text: experimental

text: approach

text: to

text: large-scale

text: cortical

text: networks

text: developing

text: ex-vivo,

text: in

text: order

text: to

text: expose

text: the

text: nature

text: of

text: response

text: variations

text: under

text: repeated

text: input

text: over

text: an

text: extended

text: range

text: of

text: time

text: scales

text: (from

text: milliseconds

text: to

text: many

text: minutes).

text: We

text: provide

text: indications

text: for

text: the

text: existence

text: of

text: a

text: hierarchy

text: of

text: timescales

text: in

text: the

text: structure

text: of

text: trial-to-trial

text: response

text: variations,

text: ranging

text: from

text: sub-seconds

text: to

text: many

text: minutes.

text: We

text: then

text: show

text: that

text: response

text: dynamics

text: over

text: minutes

text: are

text: reducible

text: to

text: the

text: interplay

text: of

text: two

text: instantaneous

text: (i.e.

text: single

text: NS,

text: sub-second

text: scale)

text: observables:

text: the

text: latency

text: from

text: stimulus

text: to

text: the

text: peak

text: of

text: the

text: NS

text: ﬁring

text: rate

text: envelop,

text: and

text: the

text: decay

text: duration

text: from

text: that

text: peak

text: to

text: baseline

text: activity.

text: We

text: show

text: that

text: these

text: two

text: instantaneous

text: measures

text: of

text: the

text: network

text: excitability

text: state—latency

text: and

text: decay

text: duration—are

text: diﬀerentially

text: sensitive

text: to

text: speciﬁc

text: pharmacological

text: blockers

text: of

text: inhibitory

text: and

text: excitatory

text: synaptic

text: transmissions,

text: suggesting

text: that

text: long-term

text: network

text: response

text: variations

text: reﬂect

text: a

text: dynamic

text: excitation-inhibition

text: ratio.

text: Materials

text: and

text: Methods

text: Cell

text: preparation

text: Cortical

text: neurons

text: were

text: obtained

text: from

text: newborn

text: rats

text: (Sprague-Dawley)

text: within

text: hours

text: after

text: birth

text: using

text: mechanical

text: and

text: enzymatic

text: procedures

text: described

text: in

text: earlier

text: studies

text: The

text: neurons

text: were

text: isolated

text: and

text: plated

text: directly

text: onto

text: substrate-

text: integrated

text: multi

text: electrode

text: arrays.

text: They

text: were

text: allowed

text: to

text: develop

text: into

text: functionally

text: and

text: structurally

text: mature

text: networks

text: over

text: a

text: period

text: of

text: weeks

text: and

text: were

text: used

text: in

text: experiments

text: within

text: the

text: period

text: of

text: weeks

text: post

text: plating.

text: The

text: number

text: of

text: plated

text: neurons

text: was

text: in

text: the

text: order

text: of

text: covering

text: an

text: area

text: of

text: about

text: with

text: heat-inactivated

text: horse

text: serum

text: glutamine

text: mM),

text: glucose

text: mM),

text: and

text: gentamycin

text: µg/ml),

text: and

text: maintained

text: in

text: an

text: atmosphere

text: of

text: CO

text: and

text: air

text: in

text: an

text: incubator

text: as

text: well

text: as

text: during

text: the

text: recording

text: phases.

text: An

text: array

text: of

text: Ti/Au

text: extracellular

text: electrodes,

text: µm

text: in

text: diameter,

text: spaced

text: µm

text: from

text: each

text: other

text: (MultiChannelSystems,

text: Reutlingen,

text: Germany)

text: was

text: used.

text: The

text: insulation

text: layer

text: (silicon

text: nitride)

text: is

text: pretreated

text: with

text: polyethyleneimine

text: (Sigma,

text: in

text: M

text: Borate

text: buﬀer

text: solution).

text: Electrophysiology

text: A

text: commercial

text: ampliﬁer

text: MCS,

text: Reutlingen,

text: Germany)

text: with

text: frequency

text: limits

text: of

text: Hz

text: and

text: a


text: 2 gain of x1024 was used for obtaining data. Data was digitized using an acquisition board (PD2-MF-64-3M/12H, UEI, Walpole, MA, USA). Each channel was sampled at a frequency of 16 kHz, and detects electrical activity that might be originated from several sources (typically 2–3 neurons) as the recording electrodes were surrounded by several cell bodies. We have used a Simulink-based software for on-line control of data collection (see Zrenner et.al. 2010 for more details). Voltage stimulation was applied in the form of a mono-phasic square pulse 200 µsec 800–950 mV through extracellular electrodes using a dedicated stimulus generator (MCS, Reutlingen, Germany). Action potentials timestamps were detected on-line by threshold crossing of negative voltage. Detection of NSs was performed oﬀ-line using a previously described algorithm [4] based on threshold crossing of the network ﬁring rate (binned to 3 msec). Pharmacology Inhibitory synaptic transmission was blocked with Bicuculline-Methiodide (Sigma-Aldrich) that was incrementally added to the bathing solution (ﬁnal concentrations used: 0.5, 1, 1.5, 2, 2.5, 3, 4, 5 and 7 µM). Excitatory synaptic blocker (APV; amino-5-phosphonovaleric, Sigma-Aldrich) was added to networks that are already under a Bicuculine blockade. Speciﬁcally, APV was added to networks that respond to stimuli in (or close to) a 1:1 manner, and where further application of Bicuculline did not change their response probability (5–12 µM Bicuculline). Final APV concentrations used were 50, 150 and 300 µM. In all the experiments with pharmacological manipulations the − stimulation rate was chosen to be 1/5 sec 1 or slower, thus maintaining high response probabilities; once tuned, the stimulation rate was kept ﬁxed for each network throughout the experiment. Data Analysis Firing rate histograms and responsiveness. Once a NS was detected within 1 second following a stimulus, action potentials recorded in all the electrodes within 1500 ms following the stimulus were extracted. Post-stimulus time histograms were constructed using a 1 ms time bin, and smoothed with a 5 ms moving average. The latency measure was deﬁned as the time between stimulus onset and the ﬁrst maximum of the smoothed ﬁring rate histogram. To compute the decay duration, the ﬁring rate histogram was further smoothed by a wider window of 30–100 ms, thus avoiding impacts of oscillations within that phase on the measure. Decay duration was deﬁned as the time between the histogram ﬁrst maximum and the ﬁrst drop below 0.15 spikes/ms along the falling phase of the NS. Fano Factor. Network activity was represented as a point process (NS trains, composed of absolute detection times, i.e. ot referenced to stimulation times). The variance of the network activity under repeating stimulation over extended time periods (24 hours) was estimated by calculating its Fano factor [16, 17]. Count sequences Z(T(n)) were calculated using logarithmically spaced bin sizes T(n). For each count sequence the Fano factor, FF(T), deﬁned as the variance of Z(T) divided by the mean of Z(T), was plotted as a function of the bin size T. The same analysis was applied to a surrogate data set generated by randomizing the inter-NS-intervals. The ﬁnal rising section of the FF curves was ﬁtted to a power law (αTβ). (cid:80) Autocorrelation. The time series of latency and decay duration were used to calculate the autocorrelation of these processes, as if they were equally spaced in time, resulting in a warped time axis. Therefore, the (cid:80)autocorrelation was − − − computed for “serial index lags” (see Figure 5A), instead of actual time lags, according to n l (xi µˆ)(xl+−i µˆ) Where i=1 ni=1(xi µˆ)2 the lag l is the diﬀerence between the serial indexes, in the stream of successful responses. Results The basic experimental procedure involved recording of activity while repetitively stimulating the network from a single site with constant amplitude at a constant rate. The data reported here originate from experiments (n = 27 networks) in which the stimulation amplitude was between 800 to 950 millivolts, delivered at a constant rate in the − range of 1/3 to 1/12 sec 1 (in diﬀerent preparations and/or experimental sessions). The data were reduced to a series of network spike time-stamps using a threshold-based criterion for identiﬁcation of synchronous activity (evoked as well as spontaneously occurring; see methods). The wide range of time scales present in neural population activity under stimulation is shown in Figure 1: Panel A demonstrates a typical evoked network spike; the latency from stimulus to the peak of the network spike, as well as the decay of activity from that peak, are processes that take place within the tens and hundreds of milliseconds, constituting the shortest timescales in our neuronal population data. The next timescale in the hierarchy is the network spike refractory period, which is the minimal time between two consecutive network spikes, ranging between 1–10 seconds [4, 18, 19]. The network refractory period is thus more than one order of magnitude longer compared to the above mentioned characteristic latency and decay times. To identify further, slower timescales in the dynamics of network spikes under ongoing stimulation at a constant rate, we resorted to a measure of statistical variance that was introduced by Teich and colleagues for the analysis of spike number distributions [20]. Speciﬁcally, time series of network spike occurrence were sliced to bins of T seconds durations, and the variance-to-mean ratio of event counts, a.k.a. Fano factor, was calculated as a function of T. ##LTLine##

label: paragraph

text: gain

text: of

text: was

text: used

text: for

text: obtaining

text: data.

text: Data

text: was

text: digitized

text: using

text: an

text: acquisition

text: board

text: UEI,

text: Walpole,

text: MA,

text: USA).

text: Each

text: channel

text: was

text: sampled

text: at

text: a

text: frequency

text: of

text: kHz,

text: and

text: detects

text: electrical

text: activity

text: that

text: might

text: be

text: originated

text: from

text: several

text: sources

text: (typically

text: neurons)

text: as

text: the

text: recording

text: electrodes

text: were

text: surrounded

text: by

text: several

text: cell

text: bodies.

text: We

text: have

text: used

text: a

text: Simulink-based

text: software

text: for

text: on-line

text: control

text: of

text: data

text: collection

text: (see

text: Zrenner

text: et.al.

text: for

text: more

text: details).

text: Voltage

text: stimulation

text: was

text: applied

text: in

text: the

text: form

text: of

text: a

text: mono-phasic

text: square

text: pulse

text: µsec

text: mV

text: through

text: extracellular

text: electrodes

text: using

text: a

text: dedicated

text: stimulus

text: generator

text: (MCS,

text: Reutlingen,

text: Germany).

text: Action

text: potentials

text: timestamps

text: were

text: detected

text: on-line

text: by

text: threshold

text: crossing

text: of

text: negative

text: voltage.

text: Detection

text: of

text: NSs

text: was

text: performed

text: oﬀ-line

text: using

text: a

text: previously

text: described

text: algorithm

text: based

text: on

text: threshold

text: crossing

text: of

text: the

text: network

text: ﬁring

text: rate

text: (binned

text: to

text: msec).

text: Pharmacology

text: Inhibitory

text: synaptic

text: transmission

text: was

text: blocked

text: with

text: Bicuculline-Methiodide

text: (Sigma-Aldrich)

text: that

text: was

text: incrementally

text: added

text: to

text: the

text: bathing

text: solution

text: (ﬁnal

text: concentrations

text: used:

text: and

text: µM).

text: Excitatory

text: synaptic

text: blocker

text: (APV;

text: Sigma-Aldrich)

text: was

text: added

text: to

text: networks

text: that

text: are

text: already

text: under

text: a

text: Bicuculine

text: blockade.

text: Speciﬁcally,

text: APV

text: was

text: added

text: to

text: networks

text: that

text: respond

text: to

text: stimuli

text: in

text: (or

text: close

text: to)

text: a

text: manner,

text: and

text: where

text: further

text: application

text: of

text: Bicuculline

text: did

text: not

text: change

text: their

text: response

text: probability

text: µM

text: Bicuculline).

text: Final

text: APV

text: concentrations

text: used

text: were

text: and

text: µM.

text: In

text: all

text: the

text: experiments

text: with

text: pharmacological

text: manipulations

text: the

text: stimulation

text: rate

text: was

text: chosen

text: to

text: be

text: sec

text: or

text: slower,

text: thus

text: maintaining

text: high

text: response

text: probabilities;

text: once

text: tuned,

text: the

text: stimulation

text: rate

text: was

text: kept

text: ﬁxed

text: for

text: each

text: network

text: throughout

text: the

text: experiment.

text: Data

text: Analysis

text: Firing

text: rate

text: histograms

text: and

text: responsiveness.

text: Once

text: a

text: NS

text: was

text: detected

text: within

text: second

text: following

text: a

text: stimulus,

text: action

text: potentials

text: recorded

text: in

text: all

text: the

text: electrodes

text: within

text: ms

text: following

text: the

text: stimulus

text: were

text: extracted.

text: Post-stimulus

text: time

text: histograms

text: were

text: constructed

text: using

text: a

text: ms

text: time

text: bin,

text: and

text: smoothed

text: with

text: a

text: ms

text: moving

text: average.

text: The

text: latency

text: measure

text: was

text: deﬁned

text: as

text: the

text: time

text: between

text: stimulus

text: onset

text: and

text: the

text: ﬁrst

text: maximum

text: of

text: the

text: smoothed

text: ﬁring

text: rate

text: histogram.

text: To

text: compute

text: the

text: decay

text: duration,

text: the

text: ﬁring

text: rate

text: histogram

text: was

text: further

text: smoothed

text: by

text: a

text: wider

text: window

text: of

text: ms,

text: thus

text: avoiding

text: impacts

text: of

text: oscillations

text: within

text: that

text: phase

text: on

text: the

text: measure.

text: Decay

text: duration

text: was

text: deﬁned

text: as

text: the

text: time

text: between

text: the

text: histogram

text: ﬁrst

text: maximum

text: and

text: the

text: ﬁrst

text: drop

text: below

text: spikes/ms

text: along

text: the

text: falling

text: phase

text: of

text: the

text: NS.

text: Fano

text: Factor.

text: Network

text: activity

text: was

text: represented

text: as

text: a

text: point

text: process

text: (NS

text: trains,

text: composed

text: of

text: absolute

text: detection

text: times,

text: i.e.

text: ot

text: referenced

text: to

text: stimulation

text: times).

text: The

text: variance

text: of

text: the

text: network

text: activity

text: under

text: repeating

text: stimulation

text: over

text: extended

text: time

text: periods

text: hours)

text: was

text: estimated

text: by

text: calculating

text: its

text: Fano

text: factor

text: Count

text: sequences

text: Z(T(n))

text: were

text: calculated

text: using

text: logarithmically

text: spaced

text: bin

text: sizes

text: T(n).

text: For

text: each

text: count

text: sequence

text: the

text: Fano

text: factor,

text: FF(T),

text: deﬁned

text: as

text: the

text: variance

text: of

text: Z(T)

text: divided

text: by

text: the

text: mean

text: of

text: Z(T),

text: was

text: plotted

text: as

text: a

text: function

text: of

text: the

text: bin

text: size

text: T.

text: The

text: same

text: analysis

text: was

text: applied

text: to

text: a

text: surrogate

text: data

text: set

text: generated

text: by

text: randomizing

text: the

text: inter-NS-intervals.

text: The

text: ﬁnal

text: rising

text: section

text: of

text: the

text: FF

text: curves

text: was

text: ﬁtted

text: to

text: a

text: power

text: law

text: (αTβ).

text: Autocorrelation.

text: The

text: time

text: series

text: of

text: latency

text: and

text: decay

text: duration

text: were

text: used

text: to

text: calculate

text: the

text: autocorrelation

text: of

text: these

text: processes,

text: as

text: if

text: they

text: were

text: equally

text: spaced

text: in

text: time,

text: resulting

text: in

text: a

text: warped

text: time

text: axis.

text: Therefore,

text: the

text: was

text: computed

text: for

text: “serial

text: index

text: lags”

text: (see

text: Figure

text: instead

text: of

text: actual

text: time

text: lags,

text: according

text: to

text: n

text: l

text: (xi

text: µˆ)(xl+−i

text: µˆ)

text: Where

text: µˆ)2

text: the

text: lag

text: l

text: is

text: the

text: diﬀerence

text: between

text: the

text: serial

text: indexes,

text: in

text: the

text: stream

text: of

text: successful

text: responses.

text: Results

text: The

text: basic

text: experimental

text: procedure

text: involved

text: recording

text: of

text: activity

text: while

text: repetitively

text: stimulating

text: the

text: network

text: from

text: a

text: single

text: site

text: with

text: constant

text: amplitude

text: at

text: a

text: constant

text: rate.

text: The

text: data

text: reported

text: here

text: originate

text: from

text: experiments

text: (n

text: networks)

text: in

text: which

text: the

text: stimulation

text: amplitude

text: was

text: between

text: to

text: millivolts,

text: delivered

text: at

text: a

text: constant

text: rate

text: in

text: the

text: range

text: of

text: to

text: sec

text: (in

text: diﬀerent

text: preparations

text: and/or

text: experimental

text: sessions).

text: The

text: data

text: were

text: reduced

text: to

text: a

text: series

text: of

text: network

text: spike

text: time-stamps

text: using

text: a

text: threshold-based

text: criterion

text: for

text: identiﬁcation

text: of

text: synchronous

text: activity

text: (evoked

text: as

text: well

text: as

text: spontaneously

text: occurring;

text: see

text: methods).

text: The

text: wide

text: range

text: of

text: time

text: scales

text: present

text: in

text: neural

text: population

text: activity

text: under

text: stimulation

text: is

text: shown

text: in

text: Figure

text: Panel

text: A

text: demonstrates

text: a

text: typical

text: evoked

text: network

text: spike;

text: the

text: latency

text: from

text: stimulus

text: to

text: the

text: peak

text: of

text: the

text: network

text: spike,

text: as

text: well

text: as

text: the

text: decay

text: of

text: activity

text: from

text: that

text: peak,

text: are

text: processes

text: that

text: take

text: place

text: within

text: the

text: tens

text: and

text: hundreds

text: of

text: milliseconds,

text: constituting

text: the

text: shortest

text: timescales

text: in

text: our

text: neuronal

text: population

text: data.

text: The

text: next

text: timescale

text: in

text: the

text: hierarchy

text: is

text: the

text: network

text: spike

text: refractory

text: period,

text: which

text: is

text: the

text: minimal

text: time

text: between

text: two

text: consecutive

text: network

text: spikes,

text: ranging

text: between

text: seconds

text: The

text: network

text: refractory

text: period

text: is

text: thus

text: more

text: than

text: one

text: order

text: of

text: magnitude

text: longer

text: compared

text: to

text: the

text: above

text: mentioned

text: characteristic

text: latency

text: and

text: decay

text: times.

text: To

text: identify

text: further,

text: slower

text: timescales

text: in

text: the

text: dynamics

text: of

text: network

text: spikes

text: under

text: ongoing

text: stimulation

text: at

text: a

text: constant

text: rate,

text: we

text: resorted

text: to

text: a

text: measure

text: of

text: statistical

text: variance

text: that

text: was

text: introduced

text: by

text: Teich

text: and

text: colleagues

text: for

text: the

text: analysis

text: of

text: spike

text: number

text: distributions

text: Speciﬁcally,

text: time

text: series

text: of

text: network

text: spike

text: occurrence

text: were

text: sliced

text: to

text: bins

text: of

text: T

text: seconds

text: durations,

text: and

text: the

text: variance-to-mean

text: ratio

text: of

text: event

text: counts,

text: a.k.a.

text: Fano

text: factor,

text: was

text: calculated

text: as

text: a

text: function

text: of

text: T.

text: ##LTLine##


text: noitalerrocotuA noitalerrocotuA ]sm[ yaceD noitalerrocotuA noitalerrocotuA ]sm[ yaceD 7 A 1 C 1 Latency Decay Latency Decay 0 0 −1 Serial Index Lag −1 Serial Index Lag 0 50 100 150 200 250 0 50 100 150 200 250 0 500 1000 1500 2000 0 500 1000 1500 2000 2500 3000 Time Lag Estimation [sec] Time Lag Estimation [sec] 1 Latency Deca y 1 Latency Deca y 0 0 −1 Serial Index Lag −1 Serial Index Lag 0 50 100 150 200 250 0 50 100 150 200 250 0 500 1000 1500 2000 2500 0 500 1000 1500 2000 2500 Time Lag Estimation [sec] Time Lag Estimation [sec] B D 3 3 3 3 10 10 10 10 2 2 2 2 10 10 10 10 2 2 2 2 10 10 10 10 Latency [ms] Latency [ms] Latency [ms] Latency [ms] FIG. 5: Latency-decay relations: Latency and decay duration were estimated from a data set of 17 networks during 50 minutes − of repeating stimulation at 1/3–1/5 sec 1. Examples for the diﬀerent manners by which latency and decay duration behave and interact are presented. Panel A depicts the autocorrelations of the time series of latency (black/red traces) and decay duration (gray traces) from two diﬀerent networks, recorded over 50 minutes; both manifesting oscillations persisting over minutes. The autocorrelations are displayed with two lag-axes: the upper axis (black) displays gaps between serial numbers of consecutive successful responses (see Methods), while the lower axis (light blue) displays an estimation of the actual time lags, evaluated from the average inter-response-intervals. Pairs of latency and decay duration of individual responses, obtained from the two networks of panel A (accordingly colored), are plotted using a log-log scale in panel B, demonstrating organized activity of repeating trajectories on the latency-decay plane. The gray dots represent individual responses from a single trajectory, the colored curves depict the smoothed version of this trajectory (calculated using a moving average procedure), and the gray curves depict 50 minutes of the smoothed traces of latency-decay pairs from each network. Panel C shows the autocorrelation of latency and decay duration time series from two other networks demonstrating the more common cases, where networks vagrantly explore the latency-decay plane with no apparent repeating trajectories. The latency series autocorrelations are depicted by the colored lines, and the decay series autocorrelations are depicted in gray. The top inset displays a case of long lasting correlations, while the bottom inset demonstrates a case of no correlation at all in both measures. The latency-decay traces of these networks are displayed in panel D, the colored lines demonstrate an apparently disorganized wandering across the latency-decay plane within a trace of 50 consecutive responses. discussion on the option of excitation-inhibition ratio being a dynamical variable that changes over minutes rather than a parameter that determines the nature of network activity in general, and its responsiveness in particular. Furthermore, we oﬀer a method to instantaneously estimate the network state in terms of its E/I ratio directly at the population level. While not free of limitations, this estimation of E/I ratio is less sensitive to synaptic ﬁltering introduced to estimations that are based on intracellular measurements. The latter are informative about the eﬀective input at the single neuron level but remote from the neural population state. We imagine at least three challenges that are entailed by the notion of excitation-inhibition ratio as a dynamical system variable, rather than a parameter. The ﬁrst challenge concerns modeling. The level of organization of the observed phenomenon (variability of population response) precludes models that focus on microscopic, this-or-that channel or synaptic receptor mechanisms underlying the rich network dynamics. In this respect it makes sense to use the terms ‘exciting forces’ and ‘restoring forces’ rather than ‘excitation’ and ‘inhibition’, thus shifting the load from local synaptic processes towards a richer repertoire of potential mechanisms that contribute to changes in network dynamics. Models that are formulated in terms of global adaptation of both exciting and restoring forces and their interaction with time scales of stimulation and spontaneous activity are wanted. Such models might shed light on possible connections between the dynamics of stimulus evoked responses reported here and the well-documented complex statistics of spontaneous series of network spikes [32–35]. The second challenge we envision is related to functional aspects of network activity in general, and the impacts of slow dynamics of exciting and restoring forces on the eﬃcacy of diﬀerent representation schemes, in particular. Whether such schemes involve population or spike- ##LTFigure## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: noitalerrocotuA

font: default

text: noitalerrocotuA

text: ]sm[

text: yaceD

text: noitalerrocotuA

text: noitalerrocotuA

text: ]sm[

text: yaceD

text: A

text: C

text: Latency

text: Decay

text: Latency

text: Decay

text: −1

text: Serial

text: Index

text: Lag

text: −1

text: Serial

text: Index

text: Lag

text: Time

text: Lag

text: Estimation

text: [sec]

text: Time

text: Lag

text: Estimation

text: [sec]

text: Latency

text: Deca

text: y

text: Latency

text: Deca

text: y

text: −1

text: Serial

text: Index

text: Lag

text: −1

text: Serial

text: Index

text: Lag

text: Time

text: Lag

text: Estimation

text: [sec]

text: Time

text: Lag

text: Estimation

text: [sec]

text: B

text: D

text: Latency

text: [ms]

text: Latency

text: [ms]

text: Latency

text: [ms]

text: Latency

text: [ms]

text: FIG.

text: Latency-decay

text: relations:

text: Latency

text: and

text: decay

text: duration

text: were

text: estimated

text: from

text: a

text: data

text: set

text: of

text: networks

text: during

text: minutes

text: of

text: repeating

text: stimulation

text: at

text: sec

text: Examples

text: for

text: the

text: diﬀerent

text: manners

text: by

text: which

text: latency

text: and

text: decay

text: duration

text: behave

text: and

text: interact

text: are

text: presented.

text: Panel

text: A

text: depicts

text: the

text: autocorrelations

text: of

text: the

text: time

text: series

text: of

text: latency

text: (black/red

text: traces)

text: and

text: decay

text: duration

text: (gray

text: traces)

text: from

text: two

text: diﬀerent

text: networks,

text: recorded

text: over

text: minutes;

text: both

text: manifesting

text: oscillations

text: persisting

text: over

text: minutes.

text: The

text: autocorrelations

text: are

text: displayed

text: with

text: two

text: lag-axes:

text: the

text: upper

text: axis

text: (black)

text: displays

text: gaps

text: between

text: serial

text: numbers

text: of

text: consecutive

text: successful

text: responses

text: (see

text: Methods),

text: while

text: the

text: lower

text: axis

text: (light

text: blue)

text: displays

text: an

text: estimation

text: of

text: the

text: actual

text: time

text: lags,

text: evaluated

text: from

text: the

text: average

text: inter-response-intervals.

text: Pairs

text: of

text: latency

text: and

text: decay

text: duration

text: of

text: individual

text: responses,

text: obtained

text: from

text: the

text: two

text: networks

text: of

text: panel

text: A

text: (accordingly

text: colored),

text: are

text: plotted

text: using

text: a

text: log-log

text: scale

text: in

text: panel

text: B,

text: demonstrating

text: organized

text: activity

text: of

text: repeating

text: trajectories

text: on

text: the

text: latency-decay

text: plane.

text: The

text: gray

text: dots

text: represent

text: individual

text: responses

text: from

text: a

text: single

text: trajectory,

text: the

text: colored

text: curves

text: depict

text: the

text: smoothed

text: version

text: of

text: this

text: trajectory

text: (calculated

text: using

text: a

text: moving

text: average

text: procedure),

text: and

text: the

text: gray

text: curves

text: depict

text: minutes

text: of

text: the

text: smoothed

text: traces

text: of

text: latency-decay

text: pairs

text: from

text: each

text: network.

text: Panel

text: C

text: shows

text: the

text: autocorrelation

text: of

text: latency

text: and

text: decay

text: duration

text: time

text: series

text: from

text: two

text: other

text: networks

text: demonstrating

text: the

text: more

text: common

text: cases,

text: where

text: networks

text: vagrantly

text: explore

text: the

text: latency-decay

text: plane

text: with

text: no

text: apparent

text: repeating

text: trajectories.

text: The

text: latency

text: series

text: autocorrelations

text: are

text: depicted

text: by

text: the

text: colored

text: lines,

text: and

text: the

text: decay

text: series

text: autocorrelations

text: are

text: depicted

text: in

text: gray.

text: The

text: top

text: inset

text: displays

text: a

text: case

text: of

text: long

text: lasting

text: correlations,

text: while

text: the

text: bottom

text: inset

text: demonstrates

text: a

text: case

text: of

text: no

text: correlation

text: at

text: all

text: in

text: both

text: measures.

text: The

text: latency-decay

text: traces

text: of

text: these

text: networks

text: are

text: displayed

text: in

text: panel

text: D,

text: the

text: colored

text: lines

text: demonstrate

text: an

text: apparently

text: disorganized

text: wandering

text: across

text: the

text: latency-decay

text: plane

text: within

text: a

text: trace

text: of

text: consecutive

text: responses.

text: discussion

text: on

text: the

text: option

text: of

text: excitation-inhibition

text: ratio

text: being

text: a

text: dynamical

text: variable

text: that

text: changes

text: over

text: minutes

text: rather

text: than

text: a

text: parameter

text: that

text: determines

text: the

text: nature

text: of

text: network

text: activity

text: in

text: general,

text: and

text: its

text: responsiveness

text: in

text: particular.

text: Furthermore,

text: we

text: oﬀer

text: a

text: method

text: to

text: instantaneously

text: estimate

text: the

text: network

text: state

text: in

text: terms

text: of

text: its

text: E/I

text: ratio

text: directly

text: at

text: the

text: population

text: level.

text: While

text: not

text: free

text: of

text: limitations,

text: this

text: estimation

text: of

text: E/I

text: ratio

text: is

text: less

text: sensitive

text: to

text: synaptic

text: ﬁltering

text: introduced

text: to

text: estimations

text: that

text: are

text: based

text: on

text: intracellular

text: measurements.

text: The

text: latter

text: are

text: informative

text: about

text: the

text: eﬀective

text: input

text: at

text: the

text: single

text: neuron

text: level

text: but

text: remote

text: from

text: the

text: neural

text: population

text: state.

text: We

text: imagine

text: at

text: least

text: three

text: challenges

text: that

text: are

text: entailed

text: by

text: the

text: notion

text: of

text: excitation-inhibition

text: ratio

text: as

text: a

text: dynamical

text: system

text: variable,

text: rather

text: than

text: a

text: parameter.

text: The

text: ﬁrst

text: challenge

text: concerns

text: modeling.

text: The

text: level

text: of

text: organization

text: of

text: the

text: observed

text: phenomenon

text: (variability

text: of

text: population

text: response)

text: precludes

text: models

text: that

text: focus

text: on

text: microscopic,

text: this-or-that

text: channel

text: or

text: synaptic

text: receptor

text: mechanisms

text: underlying

text: the

text: rich

text: network

text: dynamics.

text: In

text: this

text: respect

text: it

text: makes

text: sense

text: to

text: use

text: the

text: terms

text: ‘exciting

text: forces’

text: and

text: ‘restoring

text: forces’

text: rather

text: than

text: ‘excitation’

text: and

text: ‘inhibition’,

text: thus

text: shifting

text: the

text: load

text: from

text: local

text: synaptic

text: processes

text: towards

text: a

text: richer

text: repertoire

text: of

text: potential

text: mechanisms

text: that

text: contribute

text: to

text: changes

text: in

text: network

text: dynamics.

text: Models

text: that

text: are

text: formulated

text: in

text: terms

text: of

text: global

text: adaptation

text: of

text: both

text: exciting

text: and

text: restoring

text: forces

text: and

text: their

text: interaction

text: with

text: time

text: scales

text: of

text: stimulation

text: and

text: spontaneous

text: activity

text: are

text: wanted.

text: Such

text: models

text: might

text: shed

text: light

text: on

text: possible

text: connections

text: between

text: the

text: dynamics

text: of

text: stimulus

text: evoked

text: responses

text: reported

text: here

text: and

text: the

text: well-documented

text: complex

text: statistics

text: of

text: spontaneous

text: series

text: of

text: network

text: spikes

text: The

text: second

text: challenge

text: we

text: envision

text: is

text: related

text: to

text: functional

text: aspects

text: of

text: network

text: activity

text: in

text: general,

text: and

text: the

text: impacts

text: of

text: slow

text: dynamics

text: of

text: exciting

text: and

text: restoring

text: forces

text: on

text: the

text: eﬃcacy

text: of

text: diﬀerent

text: representation

text: schemes,

text: in

text: particular.

text: Whether

text: such

text: schemes

text: involve

text: population

text: or

text: spike-

text: ##LTFigure##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##


text: 3 or YUV), and SR is applied only on the luminance chan- Patch extraction and representation: this operation extracts nel. There are also works attempting to super-resolve all (overlapping) patches from the low-resolution image Y and channels simultaneously. For example, Kim and Kwon [25] represents each patch as a high-dimensional vector. These and Dai et al. [7] apply their model to each RGB channel vectors comprise a set of feature maps, of which the number and combined them to produce the ﬁnal results. However, equals to the dimensionality of the vectors. none of them has analyzed the SR performance of different Non-linear mapping: this operation nonlinearly maps each channels, and the necessity of recovering all three channels. high-dimensional vector onto another high-dimensional 2.2 Convolutional Neural Networks vector. Each mapped vector is conceptually the represen- tation of a high-resolution patch. These vectors comprise Convolutional neural networks (CNN) date back another set of feature maps. decades [27] and deep CNNs have recently shown an Reconstruction: this operation aggregates the above high- explosive popularity partially due to its success in image resolution patch-wise representations to generate the ﬁnal classiﬁcation [18], [26]. They have also been successfully high-resolution image. This image is expected to be similar applied to other computer vision ﬁelds, such as object de- to the ground truth X. tection [34], [40], [52], face recognition [39], and pedestrian We will show that all these operations form a convo- detection [35]. Several factors are of central importance lutional neural network. An overview of the network is in this progress: (i) the efﬁcient training implementation depicted in Figure 2. Next we detail our deﬁnition of each on modern powerful GPUs [26], (ii) the proposal of the operation. Rectiﬁed Linear Unit (ReLU) [33] which makes convergence much faster while still presents good quality [26], and (iii) 3.1.1 Patch extraction and representation the easy access to an abundance of data (like ImageNet [9]) A popular strategy in image restoration (e.g., [1]) is to for training larger models. Our method also beneﬁts from densely extract patches and then represent them by a set these progresses. of pre-trained bases such as PCA, DCT, Haar, etc. This 2.3 Deep Learning for Image Restoration is equivalent to convolving the image by a set of ﬁlters, There have been a few studies of using deep learning each of which is a basis. In our formulation, we involve techniques for image restoration. The multi-layer perceptron the optimization of these bases into the optimization of (MLP), whose all layers are fully-connected (in contrast to the network. Formally, our ﬁrst layer is expressed as an convolutional), is applied for natural image denoising [3] operation F : 1 and post-deblurring denoising [36]. More closely related to ∗ our work, the convolutional neural network is applied for F (Y) = max (0, W Y + B ) , (1) 1 1 1 natural image denoising [22] and removing noisy patterns (dirt/rain) [12]. These restoration problems are more or less denoising-driven. Cui et al. [5] propose to embed auto- where W∗1 and B1 represent the ﬁlters and biases respec- encoder networks in their super-resolution pipeline under tively, and ’ ’ denotes the convolution×oper×ation. Here, W1 the notion internal example-based approach [16]. The deep corresponds to n1 ﬁlters of support c f1 f1, where c is model is not speciﬁcally designed to be an end-to-end so- the number of channels in the input image, f1 is the spatial lution, since each layer of the cascade requires independent size of a ﬁlter. Intuitively, W1 applies n1 convoluti×ons o×n the optimization of the self-similarity search process and the image, and each convolution has a kernel size c f1 f1. auto-encoder. On the contrary, the proposed SRCNN opti- The output is composed of n1 feature maps. B1 is an n1- dimensional vector, whose each element is associated with a mizes an end-to-end mapping. Further, the SRCNN is faster at speed. It is not only a quantitatively superior method, but ﬁlter. We apply the Rectiﬁed Linear Unit (ReLU, max(0, x)) [33] on the ﬁlter responses4. also a practically useful one. 3.1.2 Non-linear mapping 3 CONVOLUTIONAL NEURAL NETWORKS FOR SUPER-RESOLUTION The ﬁrst layer extracts an n1-dimensional feature for each patch. In the second operation, we map each of these 3.1 Formulation n -dimensional vectors into an n -dimensional one. This is 1 2 equivalent to applying n ﬁlters which have a trivial spatial × 2 × Consider a single low-resolution image, we ﬁrst upscale support 1 1. This interpretation is only valid for 1 1 ﬁlters. × × it to the desired size using bicubic interpolation, which is But it is easy to generalize to larger ﬁlters like 3 3 or 5 5. the only pre-processing we perform3. Let us denote the In that case, the non-linear mapping is not on a patch of the × × interpolated image as Y. Our goal is to recover from Y input image; instead, it is on a 3 3 or 5 5 “patch” of the an image F(Y) that is as similar as possible to the ground feature map. The operation of the second layer is: truth high-resolution image X. For the ease of presentation, we still call Y a “low-resolution” image, although it has F (Y) = max (0, W ∗ F (Y) + B ) . (2) 2 2 1 2 the same size as X. We wish to learn a mapping F, which conceptually consists of three operations: × × Here W contains n ﬁlters of size n f f , and B is 2 2 1 2 2 2 3. Bicubic interpolation is also a convolutional operation, so it can n2-dimensional. Each of the output n2-dimensional vectors be formulated as a convolutional layer. However, the output size of this layer is larger than the input size, so there is a fractional stride. To 4. The ReLU can be equivalently considered as a part of the second take advantage of the popular well-optimized implementations such as operation (Non-linear mapping), and the ﬁrst operation (Patch extrac- cuda-convnet [26], we exclude this “layer” from learning. tion and representation) becomes purely linear convolution.

font: OYHPQW+NimbusSanL-Regu

text: or

font: KGPLSV+URWPalladioL-Roma

text: YUV),

text: and

text: SR

text: is

text: applied

text: only

text: on

text: the

text: luminance

text: chan-

text: Patch

font: AZYNAJ+URWPalladioL-Bold

text: extraction

text: and

text: representation:

text: this

text: operation

text: extracts

text: nel.

text: There

text: are

text: also

text: works

text: attempting

text: to

text: super-resolve

text: all

text: (overlapping)

text: patches

text: from

text: the

text: low-resolution

text: image

text: Y

text: and

text: channels

text: simultaneously.

text: For

text: example,

text: Kim

text: and

text: Kwon

text: represents

text: each

text: patch

text: as

text: a

text: high-dimensional

text: vector.

text: These

text: and

text: Dai

text: et

text: al.

text: apply

text: their

text: model

text: to

text: each

text: RGB

text: channel

text: vectors

text: comprise

text: a

text: set

text: of

text: feature

text: maps,

text: of

text: which

text: the

text: number

text: and

text: combined

text: them

text: to

text: produce

text: the

text: ﬁnal

text: results.

text: However,

text: equals

text: to

text: the

text: dimensionality

text: of

text: the

text: vectors.

text: of

text: them

text: has

text: analyzed

text: the

text: SR

text: performance

text: of

text: different

text: Non-linear

text: mapping:

text: this

text: operation

text: nonlinearly

text: maps

text: each

text: channels,

text: and

text: the

text: necessity

text: of

text: recovering

text: all

text: three

text: channels.

text: high-dimensional

text: vector

text: onto

text: another

text: high-dimensional

text: Convolutional

text: Neural

text: Networks

text: vector.

text: Each

text: mapped

text: vector

text: is

text: conceptually

text: the

text: represen-

text: tation

text: of

text: a

text: high-resolution

text: patch.

text: These

text: vectors

text: comprise

text: Convolutional

text: neural

text: networks

text: (CNN)

text: date

text: back

text: another

text: set

text: of

text: feature

text: maps.

text: decades

text: and

text: deep

text: CNNs

text: have

text: recently

text: shown

text: an

text: Reconstruction:

text: this

text: operation

text: aggregates

text: the

text: above

text: high-

text: explosive

text: popularity

text: partially

text: due

text: to

text: its

text: success

text: in

text: image

text: resolution

text: patch-wise

text: representations

text: to

text: generate

text: the

text: ﬁnal

text: classiﬁcation

text: They

text: have

text: also

text: been

text: successfully

text: high-resolution

text: image.

text: This

text: image

text: is

text: expected

text: to

text: be

text: similar

text: applied

text: to

text: other

text: computer

text: vision

text: ﬁelds,

text: such

text: as

text: object

text: de-

text: to

text: the

text: ground

text: truth

text: X.

text: tection

text: face

text: recognition

text: and

text: pedestrian

text: We

text: will

text: show

text: that

text: all

text: these

text: operations

text: form

text: a

text: convo-

text: detection

text: Several

text: factors

text: are

text: of

text: central

text: importance

text: lutional

text: neural

text: network.

text: An

text: overview

text: of

text: the

text: network

text: is

text: in

text: this

text: progress:

text: (i)

text: the

text: efﬁcient

text: training

text: implementation

text: depicted

text: in

text: Figure

text: Next

text: we

text: detail

text: our

text: deﬁnition

text: of

text: each

text: on

text: modern

text: powerful

text: GPUs

text: (ii)

text: the

text: proposal

text: of

text: the

text: operation.

text: Rectiﬁed

text: Linear

text: Unit

text: (ReLU)

text: which

text: makes

text: convergence

text: much

text: faster

text: while

text: still

text: presents

text: good

text: quality

text: and

text: (iii)

text: Patch

text: extraction

text: and

text: representation

text: the

text: easy

text: access

text: to

text: an

text: abundance

text: of

text: data

text: (like

text: ImageNet

text: A

text: popular

text: strategy

text: in

text: image

text: restoration

text: (e.g.,

text: is

text: to

text: for

text: training

text: larger

text: models.

text: Our

text: method

text: also

text: beneﬁts

text: from

text: densely

text: extract

text: patches

text: and

text: then

text: represent

text: them

text: by

text: a

text: set

text: these

text: progresses.

text: of

text: pre-trained

text: bases

text: such

text: as

text: PCA,

text: DCT,

text: Haar,

text: etc.

text: This

text: Deep

text: Learning

text: for

text: Image

text: Restoration

text: is

text: equivalent

text: to

text: convolving

text: the

text: image

text: by

text: a

text: set

text: of

text: ﬁlters,

text: There

text: have

text: been

text: a

text: few

text: studies

text: of

text: using

text: deep

text: learning

text: each

text: of

text: which

text: is

text: a

text: basis.

text: In

text: our

text: formulation,

text: we

text: involve

text: techniques

text: for

text: image

text: restoration.

text: The

text: multi-layer

text: perceptron

text: the

text: optimization

text: of

text: these

text: bases

text: into

text: the

text: optimization

text: of

text: (MLP),

text: whose

text: all

text: layers

text: are

text: fully-connected

text: (in

text: contrast

text: to

text: the

text: network.

text: Formally,

text: our

text: ﬁrst

text: layer

text: is

text: expressed

text: as

text: an

text: convolutional),

text: is

text: applied

text: for

text: natural

text: image

text: denoising

text: operation

text: F

text: and

text: post-deblurring

text: denoising

text: More

text: closely

text: related

text: to

text: our

text: work,

text: the

text: convolutional

text: neural

text: network

text: is

text: applied

text: for

text: F

text: (Y)

text: max

text: W

text: Y

text: B

text: natural

text: image

text: denoising

text: and

text: removing

text: noisy

text: patterns

text: (dirt/rain)

text: These

text: restoration

text: problems

text: are

text: more

text: or

text: less

text: denoising-driven.

text: Cui

text: et

text: al.

text: propose

text: to

text: embed

text: auto-

text: where

text: and

text: represent

text: the

text: ﬁlters

text: and

text: biases

text: respec-

text: encoder

text: networks

text: in

text: their

text: super-resolution

text: pipeline

text: under

text: tively,

text: and

text: denotes

text: the

text: convolution×oper×ation.

text: Here,

text: the

text: notion

text: internal

text: example-based

text: approach

text: The

text: deep

text: corresponds

text: to

text: ﬁlters

text: of

text: support

text: c

text: where

text: c

text: is

text: model

text: is

text: not

text: speciﬁcally

text: designed

text: to

text: be

text: an

text: end-to-end

text: so-

text: the

text: number

text: of

text: channels

text: in

text: the

text: input

text: image,

text: is

text: the

text: spatial

text: lution,

text: since

text: each

text: layer

text: of

text: the

text: cascade

text: requires

text: independent

text: size

text: of

text: a

text: ﬁlter.

text: Intuitively,

text: applies

text: convoluti×ons

text: o×n

text: the

text: optimization

text: of

text: the

text: self-similarity

text: search

text: process

text: and

text: the

text: image,

text: and

text: each

text: convolution

text: has

text: a

text: kernel

text: size

text: c

text: auto-encoder.

text: On

text: the

text: contrary,

text: the

text: proposed

text: SRCNN

text: opti-

text: The

text: output

text: is

text: composed

text: of

text: feature

text: maps.

text: is

text: an

text: dimensional

text: vector,

text: whose

text: each

text: element

text: is

text: associated

text: with

text: a

text: mizes

text: an

text: end-to-end

text: mapping.

text: Further,

text: the

text: SRCNN

text: is

text: faster

text: at

text: speed.

text: It

text: is

text: not

text: only

text: a

text: quantitatively

text: superior

text: method,

text: but

text: ﬁlter.

text: We

text: apply

text: the

text: Rectiﬁed

text: Linear

text: Unit

text: (ReLU,

text: x))

text: on

text: the

text: ﬁlter

text: also

text: a

text: practically

text: useful

text: one.

text: Non-linear

text: mapping

text: CONVOLUTIONAL

text: NEURAL

text: NETWORKS

text: FOR

text: SUPER-RESOLUTION

text: The

text: ﬁrst

text: layer

text: extracts

text: an

text: feature

text: for

text: each

text: patch.

text: In

text: the

text: second

text: operation,

text: we

text: map

text: each

text: of

text: these

text: Formulation

text: n

text: -dimensional

text: vectors

text: into

text: an

text: n

text: -dimensional

text: one.

text: This

text: is

text: equivalent

text: to

text: applying

text: n

text: ﬁlters

text: which

text: have

text: a

text: trivial

text: spatial

text: Consider

text: a

text: single

text: low-resolution

text: image,

text: we

text: ﬁrst

text: upscale

text: support

text: This

text: interpretation

text: is

text: only

text: valid

text: for

text: ﬁlters.

text: it

text: to

text: the

text: desired

text: size

text: using

text: bicubic

text: interpolation,

text: which

text: is

text: But

text: it

text: is

text: easy

text: to

text: generalize

text: to

text: larger

text: ﬁlters

text: like

text: or

text: the

text: only

text: pre-processing

text: we

text: Let

text: us

text: denote

text: the

text: In

text: that

text: case,

text: the

text: non-linear

text: mapping

text: is

text: not

text: on

text: a

text: patch

text: of

text: the

text: interpolated

text: image

text: as

text: Y.

text: Our

text: goal

text: is

text: to

text: recover

text: from

text: Y

text: input

text: image;

text: instead,

text: it

text: is

text: on

text: a

text: or

text: “patch”

text: of

text: the

text: an

text: image

text: F(Y)

text: that

text: is

text: as

text: similar

text: as

text: possible

text: to

text: the

text: ground

text: feature

text: map.

text: The

text: operation

text: of

text: the

text: second

text: layer

text: is:

text: truth

text: high-resolution

text: image

text: X.

text: For

text: the

text: ease

text: of

text: presentation,

text: we

text: still

text: call

text: Y

text: a

text: “low-resolution”

text: image,

text: although

text: it

text: has

text: F

text: (Y)

text: max

text: W

text: F

text: (Y)

text: B

text: the

text: same

text: size

text: as

text: X.

text: We

text: wish

text: to

text: learn

text: a

text: mapping

text: F,

text: which

text: conceptually

text: consists

text: of

text: three

text: operations:

text: Here

text: W

text: contains

text: n

text: ﬁlters

text: of

text: size

text: n

text: f

text: f

text: and

text: B

text: is

text: Bicubic

text: interpolation

text: is

text: also

text: a

text: convolutional

text: operation,

text: so

text: it

text: can

text: Each

text: of

text: the

text: output

text: vectors

text: be

text: formulated

text: as

text: a

text: convolutional

text: layer.

text: However,

text: the

text: output

text: size

text: of

text: this

text: layer

text: is

text: larger

text: than

text: the

text: input

text: size,

text: so

text: there

text: is

text: a

text: fractional

text: stride.

text: To

text: The

text: ReLU

text: can

text: be

text: equivalently

text: considered

text: as

text: a

text: part

text: of

text: the

text: second

text: take

text: advantage

text: of

text: the

text: popular

text: well-optimized

text: implementations

text: such

text: as

text: operation

text: (Non-linear

text: mapping),

text: and

text: the

text: ﬁrst

text: operation

text: (Patch

text: extrac-

text: cuda-convnet

text: we

text: exclude

text: this

text: “layer”

text: from

text: learning.

text: tion

text: and

text: representation)

text: becomes

text: purely

text: linear

text: convolution.


text: 4 feature maps feature maps of low-resolution image of high-resolution image Low-resolution High-resolution image (input) image (output) Patch extraction Non-linear mapping Reconstruction and representation Fig. 2. Given a low-resolution image Y, the ﬁrst convolutional layer of the SRCNN extracts a set of feature maps. The second layer maps these feature maps nonlinearly to high-resolution patch representations. The last layer combines the predictions within a spatial neighbourhood to produce the ﬁnal high-resolution image F(Y). is conceptually a representation of a high-resolution patch 3.2 Relationship to Sparse-Coding-Based Methods that will be used for reconstruction. We show that the sparse-coding-based SR methods [49], It is possible to add more convolutional layers to increase [50] can be viewed as a convolutional neural network. the non-line×arity.×But t×his can increase the complexity of the Figure 3 shows an illustration. model (n2 f2 f2 n2 parameters for one layer), and In the sparse-coding-based methods, let us consider that × thus demands more training time. We will explore deeper an f f low-resolution patch is extracted from the input 1 1 structures by introducing additional non-linear mapping image. Then the sparse coding solver, like Feature-Sign [29], layers in Section . will ﬁrst project the patch onto a (low-resolution) dictionary. 3.1.3 Reconstruction If the dictionary size is n , this is equivalent to applying × 1 n linear ﬁlters (f f ) on the input image (the mean In the traditional methods, the predicted overlapping 1 1 1 subtraction is also a linear operation so can be absorbed). high-resolution patches are often averaged to produce the This is illustrated as the left part of Figure 3. ﬁnal full image. The averaging can be considered as a The sparse coding solver will then iteratively process the pre-deﬁned ﬁlter on a set of feature maps (where each n coefﬁcients. The outputs of this solver are n coefﬁcients, position is the “ﬂattened” vector form of a high-resolution 1 2 and usually n = n in the case of sparse coding. These patch). Motivated by this, we deﬁne a convolutional layer 2 1 n coefﬁcients are the representation of the high-resolution to produce the ﬁnal high-resolution image: 2 patch. In this sense, the sparse coding solver behaves as a special case of a non-linear mapping operator, whose spatial × ∗ support is 1 1. See the middle part of Figure 3. However, F(Y) = W F (Y) + B . (3) 3 2 3 the sparse coding solver is not feed-forward, i.e.,it is an iterative algorithm. On the contrary, our non-linear operator × × is fully feed-forward and can be computed efﬁciently. If we Here W3 corresponds to c ﬁlters of a size n2 f3 f3, set f2 = 1, then our non-linear operator can be considered and B3 is a c-dimensional vector. as a pixel-wise fully-connected layer. It is worth noting that If the representations of the high-resolution patches “the sparse coding solver” in SRCNN refers to the ﬁrst are in the image domain (i.e.,we can simply reshape each two layers, but not just the second layer or the activation representation to form the patch), we expect that the ﬁl- function (ReLU). Thus the nonlinear operation in SRCNN is ters act like an averaging ﬁlter; if the representations of also well optimized through the learning process. the high-resolution patches are in some other domains The above n coefﬁcients (after sparse coding) are 2 (e.g.,coefﬁcients in terms of some bases), we expect that W then projected onto another (high-resolution) dictionary to 3 behaves like ﬁrst projecting the coefﬁcients onto the image produce a high-resolution patch. The overlapping high- domain and then averaging. In either way, W is a set of resolution patches are then averaged. As discussed above, 3 linear ﬁlters. this is equivalent to linear convolutions on the n feature 2 maps. If the high-resolution patches used for reconstruction × Interestingly, although the above three operations are are of size f f , then the linear ﬁlters have an equivalent 3 3 × motivated by different intuitions, they all lead to the same spatial support of size f f . See the right part of Figure 3. 3 3 form as a convolutional layer. We put all three operations to- The above discussion shows that the sparse-coding- gether and form a convolutional neural network (Figure 2). based SR method can be viewed as a kind of convolutional In this model, all the ﬁltering weights and biases are to be neural network (with a different non-linear mapping). But optimized. Despite the succinctness of the overall structure, not all operations have been considered in the optimization our SRCNN model is carefully developed by drawing ex- in the sparse-coding-based SR methods. On the contrary, tensive experience resulted from signiﬁcant progresses in in our convolutional neural network, the low-resolution super-resolution [49], [50]. We detail the relationship in the dictionary, high-resolution dictionary, non-linear mapping, next section. together with mean subtraction and averaging, are all in- ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure## ##LTFigure##

text: feature

text: maps

text: feature

text: maps

text: of

text: low-resolution

text: image

text: of

text: high-resolution

text: image

text: Low-resolution

text: High-resolution

text: image

text: (input)

text: image

text: (output)

text: Patch

text: extraction

text: Non-linear

text: mapping

text: Reconstruction

text: and

text: representation

text: Fig.

text: Given

label: caption

text: a

text: low-resolution

text: image

text: Y,

text: the

text: ﬁrst

text: convolutional

text: layer

text: of

text: the

text: SRCNN

text: extracts

text: a

text: set

text: of

text: feature

text: maps.

text: The

text: second

text: layer

text: maps

text: these

text: feature

text: maps

text: nonlinearly

text: to

text: high-resolution

text: patch

text: representations.

text: The

text: last

text: layer

text: combines

text: the

text: predictions

text: within

text: a

text: spatial

text: neighbourhood

text: to

text: produce

text: the

text: ﬁnal

text: high-resolution

text: image

text: F(Y).

text: is

text: conceptually

text: a

text: representation

text: of

text: a

text: high-resolution

text: patch

text: Relationship

text: to

text: Sparse-Coding-Based

text: Methods

text: that

text: will

text: be

text: used

text: for

text: reconstruction.

text: We

text: show

text: that

text: the

text: sparse-coding-based

text: SR

text: methods

text: It

text: is

text: possible

text: to

text: add

text: more

text: convolutional

text: layers

text: to

text: increase

text: can

text: be

text: viewed

text: as

text: a

text: convolutional

text: neural

text: network.

text: the

text: non-line×arity.×But

text: t×his

text: can

text: increase

text: the

text: complexity

text: of

text: the

text: Figure

text: shows

text: an

text: illustration.

text: model

text: parameters

text: for

text: one

text: layer),

text: and

text: In

text: the

text: sparse-coding-based

text: methods,

text: let

text: us

text: consider

text: that

text: thus

text: demands

text: more

text: training

text: time.

text: We

text: will

text: explore

text: deeper

text: an

text: f

text: f

text: low-resolution

text: patch

text: is

text: extracted

text: from

text: the

text: input

text: structures

text: by

text: introducing

text: additional

text: non-linear

text: mapping

text: image.

text: Then

text: the

text: sparse

text: coding

text: solver,

text: like

text: Feature-Sign

text: layers

text: in

text: Section

text: will

text: ﬁrst

text: project

text: the

text: patch

text: onto

text: a

text: (low-resolution)

text: dictionary.

text: Reconstruction

text: If

text: the

text: dictionary

text: size

text: is

text: n

text: this

text: is

text: equivalent

text: to

text: applying

text: n

text: linear

text: ﬁlters

text: (f

text: f

text: on

text: the

text: input

text: image

text: (the

text: mean

text: In

text: the

text: traditional

text: methods,

text: the

text: predicted

text: overlapping

text: subtraction

text: is

text: also

text: a

text: linear

text: operation

text: so

text: can

text: be

text: absorbed).

text: high-resolution

text: patches

text: are

text: often

text: averaged

text: to

text: produce

text: the

text: This

text: is

text: illustrated

text: as

text: the

text: left

text: part

text: of

text: Figure

text: ﬁnal

text: full

text: image.

text: The

text: averaging

text: can

text: be

text: considered

text: as

text: a

text: The

text: sparse

text: coding

text: solver

text: will

text: then

text: iteratively

text: process

text: the

text: pre-deﬁned

text: ﬁlter

text: on

text: a

text: set

text: of

text: feature

text: maps

text: (where

text: each

text: n

text: coefﬁcients.

text: The

text: outputs

text: of

text: this

text: solver

text: are

text: n

text: coefﬁcients,

text: position

text: is

text: the

text: “ﬂattened”

text: vector

text: form

text: of

text: a

text: high-resolution

text: and

text: usually

text: n

text: n

text: in

text: the

text: case

text: of

text: sparse

text: coding.

text: These

text: patch).

text: Motivated

text: by

text: this,

text: we

text: deﬁne

text: a

text: convolutional

text: layer

text: n

text: coefﬁcients

text: are

text: the

text: representation

text: of

text: the

text: high-resolution

text: to

text: produce

text: the

text: ﬁnal

text: high-resolution

text: image:

text: patch.

text: In

text: this

text: sense,

text: the

text: sparse

text: coding

text: solver

text: behaves

text: as

text: a

text: special

text: case

text: of

text: a

text: non-linear

text: mapping

text: operator,

text: whose

text: spatial

text: support

text: is

text: See

text: the

text: middle

text: part

text: of

text: Figure

text: However,

text: F(Y)

text: W

text: F

text: (Y)

text: B

text: the

text: sparse

text: coding

text: solver

text: is

text: not

text: feed-forward,

text: i.e.,it

text: is

text: an

text: iterative

text: algorithm.

text: On

text: the

text: contrary,

text: our

text: non-linear

text: operator

text: is

text: fully

text: feed-forward

text: and

text: can

text: be

text: computed

text: efﬁciently.

text: If

text: we

text: Here

text: corresponds

text: to

text: c

text: ﬁlters

text: of

text: a

text: size

text: set

text: then

text: our

text: non-linear

text: operator

text: can

text: be

text: considered

text: and

text: is

text: a

text: c-dimensional

text: vector.

text: as

text: a

text: pixel-wise

text: fully-connected

text: layer.

text: It

text: is

text: worth

text: noting

text: that

text: If

text: the

text: representations

text: of

text: the

text: high-resolution

text: patches

text: “the

text: sparse

text: coding

text: solver”

text: in

text: SRCNN

text: refers

text: to

text: the

text: ﬁrst

text: are

text: in

text: the

text: image

text: domain

text: (i.e.,we

text: can

text: simply

text: reshape

text: each

text: two

text: layers,

text: but

text: not

text: just

text: the

text: second

text: layer

text: or

text: the

text: activation

text: representation

text: to

text: form

text: the

text: patch),

text: we

text: expect

text: that

text: the

text: ﬁl-

text: function

text: (ReLU).

text: Thus

text: the

text: nonlinear

text: operation

text: in

text: SRCNN

text: is

text: ters

text: act

text: like

text: an

text: averaging

text: ﬁlter;

text: if

text: the

text: representations

text: of

text: also

text: well

text: optimized

text: through

text: the

text: learning

text: process.

text: the

text: high-resolution

text: patches

text: are

text: in

text: some

text: other

text: domains

text: The

text: above

text: n

text: coefﬁcients

text: (after

text: sparse

text: coding)

text: are

text: (e.g.,coefﬁcients

text: in

text: terms

text: of

text: some

text: bases),

text: we

text: expect

text: that

text: W

text: then

text: projected

text: onto

text: another

text: (high-resolution)

text: dictionary

text: to

text: behaves

text: like

text: ﬁrst

text: projecting

text: the

text: coefﬁcients

text: onto

text: the

text: image

text: produce

text: a

text: high-resolution

text: patch.

text: The

text: overlapping

text: high-

text: domain

text: and

text: then

text: averaging.

text: In

text: either

text: way,

text: W

text: is

text: a

text: set

text: of

text: resolution

text: patches

text: are

text: then

text: averaged.

text: As

text: discussed

text: above,

text: linear

text: ﬁlters.

text: this

text: is

text: equivalent

text: to

text: linear

text: convolutions

text: on

text: the

text: n

text: feature

text: maps.

text: If

text: the

text: high-resolution

text: patches

text: used

text: for

text: reconstruction

text: Interestingly,

text: although

text: the

text: above

text: three

text: operations

text: are

text: are

text: of

text: size

text: f

text: f

text: then

text: the

text: linear

text: ﬁlters

text: have

text: an

text: equivalent

text: motivated

text: by

text: different

text: intuitions,

text: they

text: all

text: lead

text: to

text: the

text: same

text: spatial

text: support

text: of

text: size

text: f

text: f

text: See

text: the

text: right

text: part

text: of

text: Figure

text: form

text: as

text: a

text: convolutional

text: layer.

text: We

text: put

text: all

text: three

text: operations

text: to-

text: The

text: above

text: discussion

text: shows

text: that

text: the

text: sparse-coding-

text: gether

text: and

text: form

text: a

text: convolutional

text: neural

text: network

text: (Figure

text: based

text: SR

text: method

text: can

text: be

text: viewed

text: as

text: a

text: kind

text: of

text: convolutional

text: In

text: this

text: model,

text: all

text: the

text: ﬁltering

text: weights

text: and

text: biases

text: are

text: to

text: be

text: neural

text: network

text: (with

text: a

text: different

text: non-linear

text: mapping).

text: But

text: optimized.

text: Despite

text: the

text: succinctness

text: of

text: the

text: overall

text: structure,

text: not

text: all

text: operations

text: have

text: been

text: considered

text: in

text: the

text: optimization

text: our

text: SRCNN

text: model

text: is

text: carefully

text: developed

text: by

text: drawing

text: ex-

text: in

text: the

text: sparse-coding-based

text: SR

text: methods.

text: On

text: the

text: contrary,

text: tensive

text: experience

text: resulted

text: from

text: signiﬁcant

text: progresses

text: in

text: in

text: our

text: convolutional

text: neural

text: network,

text: the

text: low-resolution

text: super-resolution

text: We

text: detail

text: the

text: relationship

text: in

text: the

text: dictionary,

text: high-resolution

text: dictionary,

text: non-linear

text: mapping,

text: next

text: section.

text: together

text: with

text: mean

text: subtraction

text: and

text: averaging,

text: are

text: all

text: in-

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##

text: ##LTFigure##


text: 14 LEONID SLAVIN → ∞ (cid:3) Taking the limit as p , we obtain the statement of the lemma. (cid:0) (cid:1) − To prove the corollary, write γ = α β. Then | | (cid:104) (cid:105) (cid:104) − (cid:105) | | (cid:104) (cid:105) (cid:104) − (cid:105) (cid:104) − (cid:105) (cid:104) − (cid:105)− c(α,β)(w) = J w α w 1 β R (w) = J w w 1 α w 1 γ R (w) = c(α)(w) w 1 γ. J J J J J J J J J J { } (α) Since c (w) is a Carleson sequence with norm not exceeding K ([w] ), the result follows J α Ad2 (cid:3) from Lemma 3.1. 4. Necessary conditions on Bellman candidates; induction on scales When seeking the Bellman function for an inequality, one typically determines key condi- tions that this function must satisfy directly from its deﬁnition and then derives (or otherwise presents) a candidate function that possess these properties. The candidate is then shown to equal the true Bellman function – or at least bound it from above or below, as appropriate. The actual proof of the inequality consists of revealing the exact relationship between the Bell- man candidate and the Bellman function and, technically, does not require one to know where the candidate comes from. However, the process of constructing a candidate both illustrates the main inductive component of the proof and is intrinsically tied with the construction of optimizers in the original inequality. Accordingly, we present it in some detail. The following lemma lists three key properties of B and b that follow from deﬁni- Q,Φ Q,Φ tions (2.3) and (2.4). Lemma 4.1. The functions B and b satisfy Q,Φ Q,Φ − ∈ − ∈ (1) Main inequality. For all points x , x+ Ω such that (x + x+)/2 Ω , Q Q (cid:20) (cid:21) (cid:16) (cid:17) (4.1) − − − − − (cid:16)x + x+ (cid:17) (cid:62) 1 − 1 (cid:20) (x x+)2 (x x+)(cid:21)2 B B (x ) + B (x+) + Φ(x x ) 1 1 + 2 2 , Q,Φ 2 2 Q,Φ 2 Q,Φ 1 2 x2 x2 1 2 − − − − − x + x+ (cid:54) 1 − 1 (x x+)2 (x x+)2 (4.2) b b (x ) + b (x+) + Φ(x x ) 1 1 + 2 2 . Q,Φ 2 2 Q,Φ 2 Q,Φ 1 2 x2 x2 1 2 ∈ ∞ (2) Boundary condition. For all x (0, ), 1 (4.3) B (x , 1/x ) = 0, b (x , 1/x ) = 0. Q,Φ 1 1 Q,Φ 1 1 ∈ (3) Homogeneity. For all x√ ΩQ, √ √ √ (4.4) B (x , x ) = B ( x x , x x ), b (x , x ) = b ( x x , x x ). Q,Φ 1 2 Q,Φ 1 2 1 2 Q,Φ 1 2 Q,Φ 1 2 1 2 Proof. It suﬃces to prove these statements for B as the proof for b is almost identical. Q,Φ ∈ Q,Φ To prove (4.1), ﬁrst observe that fo(cid:16)r any(cid:113)point x =(cid:17) (x1, x2) ΩQ, there exists a weight w ∈ Ad,Q(I) such that (cid:104)w(cid:105) = x and (cid:104)w−1(cid:105) = x . For instance, one can take 2 I 1 I 2 (cid:16) (cid:113) (cid:17)  − − ∈ − x 1 1 1 , t I , 1 x1x2 (4.5) w(t) = − ∈ x 1 + 1 1 , t I+. 1 x1x2 ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine## ##LTLine##

text: LEONID

text: SLAVIN

text: Taking

text: the

text: limit

text: as

text: p

text: we

text: obtain

text: the

text: statement

text: of

text: the

text: lemma.

text: To

text: prove

text: the

text: corollary,

text: write

text: γ

text: α

text: β.

text: Then

text: c(α,β)(w)

text: J

text: w

text: α

text: w

text: β

text: R

text: (w)

text: J

text: w

text: w

text: α

text: w

text: γ

text: R

text: (w)

text: c(α)(w)

text: w

text: γ.

text: J

text: J

text: J

text: J

text: J

text: J

text: J

text: J

text: J

text: J

text: (α)

text: Since

text: c

text: (w)

text: is

text: a

text: Carleson

text: sequence

text: with

text: norm

text: not

text: exceeding

text: K

text: ([w]

text: the

text: result

text: follows

text: J

text: α

text: from

text: Lemma

text: Necessary

text: conditions

text: on

text: Bellman

text: candidates;

text: induction

text: on

text: scales

text: When

text: seeking

text: the

text: Bellman

text: function

text: for

text: an

text: inequality,

text: one

text: typically

text: determines

text: key

text: condi-

text: tions

text: that

text: this

text: function

text: must

text: satisfy

text: directly

text: from

text: its

text: deﬁnition

text: and

text: then

text: derives

text: (or

text: otherwise

text: presents)

text: a

text: candidate

text: function

text: that

text: possess

text: these

text: properties.

text: The

text: candidate

text: is

text: then

text: shown

text: to

text: equal

text: the

text: true

text: Bellman

text: function

text: or

text: at

text: least

text: bound

text: it

text: from

text: above

text: or

text: below,

text: as

text: appropriate.

text: The

text: actual

text: proof

text: of

text: the

text: inequality

text: consists

text: of

text: revealing

text: the

text: exact

text: relationship

text: between

text: the

text: Bell-

text: man

text: candidate

text: and

text: the

text: Bellman

text: function

text: and,

text: technically,

text: does

text: not

text: require

text: one

text: to

text: know

text: where

text: the

text: candidate

text: comes

text: from.

text: However,

text: the

text: process

text: of

text: constructing

text: a

text: candidate

text: both

text: illustrates

text: the

text: main

text: inductive

text: component

text: of

text: the

text: proof

text: and

text: is

text: intrinsically

text: tied

text: with

text: the

text: construction

text: of

text: optimizers

text: in

text: the

text: original

text: inequality.

text: Accordingly,

text: we

text: present

text: it

text: in

text: some

text: detail.

text: The

text: following

text: lemma

text: lists

text: three

text: key

text: properties

text: of

text: B

text: and

text: b

text: that

text: follow

text: from

text: deﬁni-

text: Q,Φ

text: Q,Φ

text: tions

text: and

text: Lemma

text: The

text: functions

text: B

text: and

text: b

text: satisfy

text: Q,Φ

text: Q,Φ

text: Main

text: inequality.

text: For

text: all

text: points

text: x

text: x+

text: Ω

text: such

text: that

text: (x

text: Ω

text: Q

text: Q

text: x+

text: (x

text: (x

text: B

text: B

text: (x

text: B

text: (x+)

text: Φ(x

text: x

text: Q,Φ

text: Q,Φ

text: Q,Φ

text: x

text: x+

text: (x

text: (x

text: b

text: b

text: (x

text: b

text: (x+)

text: Φ(x

text: x

text: Q,Φ

text: Q,Φ

text: Q,Φ

text: Boundary

text: condition.

text: For

text: all

text: x

text: B

text: (x

text: b

text: (x

text: Q,Φ

text: Q,Φ

text: Homogeneity.

text: For

text: all

text: x√

text: ΩQ,

text: B

text: (x

text: x

text: B

text: x

text: x

text: x

text: x

text: b

text: (x

text: x

text: b

text: x

text: x

text: x

text: x

text: Q,Φ

text: Q,Φ

text: Q,Φ

text: Q,Φ

text: Proof.

text: It

text: suﬃces

text: to

text: prove

text: these

text: statements

text: for

text: B

text: as

text: the

text: proof

text: for

text: b

text: is

text: almost

text: identical.

text: Q,Φ

text: Q,Φ

text: To

text: prove

text: ﬁrst

text: observe

text: that

text: x

text: ΩQ,

text: there

text: exists

text: a

text: weight

text: w

text: Ad,Q(I)

text: such

text: that

text: x

text: and

text: x

text: For

text: instance,

text: one

text: can

text: take

text: I

text: I

text: x

text: t

text: I

text: w(t)

text: x

text: t

text: I+.

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##

text: ##LTLine##