EURASIP Journal on Applied Signal Processing 2003:7, 620–628 c ○ 2003 Hindawi Publishing Corporation Analog VLSI Circuits for Short-Term Dynamic Synapses

Short-term dynamical synapses increase the computational power of neuronal networks. These synapses act as additional ﬁlters to the inputs of a neuron before the subsequent integration of these signals at its cell body. In this work, we describe a model of depressing and facilitating synapses derived from a hardware circuit implementation. This model is equivalent to theoretical models of short-term synaptic dynamics in network simulations. These circuits have been added to a network of leaky integrate-and-ﬁre neurons. A cortical model of direction-selectivity that uses short-term dynamic synapses has been implemented with this network.


INTRODUCTION
Cortical neurons show a wide variety of neuronal and synaptic responses to their input signals.Networks with simplified models of spiking neurons and synapses and consisting of one or two time constants already exhibit a large number of possible operating regimes [1,2].Simulations of these spiking networks can take a long time on a serial computer.
In most network simulations, synapses are assumed to be static.Recent physiological data, however, show that synapses frequently show activity-dependent plasticity which vary on a time scale of milliseconds to seconds.In particular, shortterm dynamical synapses [3,4,5,6,7] with time constants of hundreds of milliseconds are seen in many parts of the visual cortex.When these synapses are stimulated with a train of input spikes, the amplitude of the membrane potential of the neuron or the excitatory postsynaptic potential (EPSP) decreases (depressing synapse) or increases (facilitating synapse) with each subsequent spike.The recovery time of the maximum synaptic amplitude is in the order of hundreds of milliseconds.These synapses encode the history of their inputs and can be treated as time-invariant filters with fading memory [8].
These activity-dependent synapses, when added to the network, allow for different forms of dynamical networks that can process time-varying patterns [9,10].Examples of how these synapses could contribute to visual cortical responses include direction selectivity [11] and automatic gain control [12].The simulation time of spiking networks with different types of activity-dependent synapses consisting of different time constants will increase significantly.This simulation time can be shortened by using a hardware implemen-tation of a network with spiking neurons and these activitydependent synapses.
Here, we describe a circuit model of short-term synaptic dynamics based on the silicon implementation of synaptic depression and facilitation in [13].The dynamics of this circuit model is qualitatively comparable to the dynamics of two theoretical models [14]: the phenomenological model from [6,9,15] and the model from [5,12].Measurements from these circuits on a fabricated chip show how these synapses filter the inputs to a leaky integrate-and-fire neuron under transient and steady-state conditions.
The dynamics of short-term plastic synapses are dependent on the frequency of the presynaptic input.In the case of a neuron which is stimulated through a depressing synapse by a regular input spike train, the firing rate of the neuron decreases over time due to the decrease in synaptic input with each presynaptic spike.Interestingly, a class of neurons in the cortex also adapt their firing rate over time in response to a regular spike input through a normal synapse.This output adaptation mechanism is noninput specific whereas the first mechanism involves the filtering of specific inputs.
The inclusion of these short-term synapses into networks of neurons allow processing of time-varying inputs.However, the simulation time of such networks on a computer increases substantially as more different types of time constants are added to the circuits.The previous constructions of neuron circuits ranging from Hodgkin-Huxley models of neurons [16,17] to integrate-and-fire neurons [18,19,20,21,22], together with long-time constant learning synapses [23,24] and short-term dynamic synapses [13] can be used to develop realistic, real-time, low-power, and spike-based networks.

SYNAPSES
Synaptic circuits have been implemented using very few transistors [13,25].However, their dynamics are usually different from the exponential dynamics of synaptic models used in simulations.To implement the exponential dynamics, we would have to use a linear resistor to obtain the exponential dynamics.A transistor can act as a linear resistor as long as the terminal voltages satisfy certain criteria.Additional circuitry would be needed to satisfy these criteria, thus increasing the final size of the circuit.One alternative is to replace the linear-resistor dynamics with diode dynamics which is easily obtained with one diode-connected transistor.We will discuss the difference between the diode-connected transistor dynamics and the exponential dynamics for the different types of synapses.

Normal synapses
In simulations, the synaptic current i(t) is either treated as a point current source at the time of the spike t sp : where I f is a fixed current, or as a current source with a finite decay time: where τ g is the time constant of the decay and t is measured right after a spike.The point current source can be implemented by two transistors (e.g., M 2 and M 3 in Figure 1a).If we need a synaptic current with a finite decay time, we include the currentmirror circuit M 1 , M 4 , and C. Unlike the dynamics in (4), the synaptic current I d has a 1/t decay dynamics [25] rather than exponential dynamics.The decay of I d is described by where Q T = CU T , A = e (κVdd−Vgain)/UT , U T is the thermal voltage, and I d0 is the value of I d at the time of the spike t = t sp .

Short-term synaptic dynamics
Dynamical synapses can be depressing, facilitating, or a combination of both.In a depressing synapse, the synaptic strength decreases after each spike and recovers towards its maximal value with a time constant τ d .In facilitating synapses, the strength increases after each spike and recovers towards its minimum value with a time constant τ f .Two prevalent models that are used in network simulations and also for fitting physiological data are the phenomenological model in [6,9,15] and the model from [5,12].We only consider the dynamics of the model from Abbott et al. [12] in this work.

Simulation model of short-term dynamic synapses
The dynamics of the depressing synapse is similar to the adaptation dynamics of the photoreceptor.Both elements code primarily changes in the input rather than the absolute level of the input.The photoreceptor amplifies the contrast of the visual signal and has a low gain to background illumination.The output of the depressing synapse codes primarily changes in the presynaptic frequency.The synaptic strength adapts to a steady-state value that is approximately inversely dependent on the input frequency.Thus, the depressing synapse acts like a band-pass filter to spike rates, much like the photoreceptor has a band-pass response to illumination.The facilitating synapse, on the other hand, acts like a low-pass filter to changes in spike rates.A step increase in presynaptic firing rate leads to an increase in the synaptic strength.Both types of synapses can be treated as time-invariant fading memory filters [8].
In the theoretical model from Abbott et al. [12], the depression in the synaptic strength is defined by a variable D varying between 0 and 1.The synaptic strength is given by gD(t) where g is the maximum synaptic strength.The recovery dynamics of D is described by where τ d is the recovery time constant of the depression, and the update dynamics is where d (d < 1) is the amount by which D is decreased right after the spike.In the case of a regular spike train, the average steady-state value of D is In the facilitating case, the facilitation is defined by a variable F ≥ 1.The synaptic strength is g f F(t), where g f is the maximum synaptic strength.The recovery dynamics of F is where τ f is the time constant in which F recovers exponentially back to 1.The update dynamics is now additive instead of subtractive: where f ( f < 1) is the amount by which F is increased right after the spike.The variable F is updated additively because multiplicative facilitation can lead to increases of synaptic strength without bounds, especially at high frequencies, for the recovery dynamics in (7).

Circuit model of short-term dynamic synapses
As before, we replace the exponential dynamics in (4) with the diode-connected transistor dynamics.This replacement gives rise to the synaptic depressing circuit in Figure 1b which was proposed in [13].The new circuit gives rise to the following recovery dynamics for the depressing variable D: where M is the equivalent of 1/τ d and κ is a transistor parameter which is less than 1 in subthreshold operation.The update dynamics are similar to (5):

Depressing circuit
The detailed analysis leading to ( 9) and ( 10) for D is described in [14].The voltage V a determines the maximum synaptic strength g while the synaptic strength gD or I syn is exponential in the voltage V x .The subcircuit consisting of transistors M 1 , M 2 , and M 3 controls the dynamics of I syn .
The presynaptic input goes to the gate terminal of M 3 which acts like a switch.During a presynaptic spike, a quantity of charge (determined by V d ) is removed from the node V x .In between spikes, V x recovers towards V a through the diodeconnected transistor M 1 .Also during the presynaptic spike, transistor M 4 turns on and the synaptic current I syn flows into the membrane potential of the neuron.We can convert the I syn current source into an equivalent current I d with some gain and a "time constant" through the current-mirror circuit consisting of M 6 , M 7 , and the capacitor C 2 , and by adjusting the voltage V gain .
The synaptic strength is given by where g = I on e κVa/UT , and where I r f := I op e κ(Vdd−Vx)/UT .The recovery time constant Because it is difficult to compute a closed-form solution for (9) for any value of κ, we look at a simple case where κ = 0.5 and solve for D(t) after a spike has occurred at t = t 0 . 1he actual value of κ changes for different operating conditions and also depends on fabrication parameters.The recovery equation in (13) includes the current dynamics of the diode-connected transistor (M 1 in Figure 1b) in the region when D is close to the maximum value.The equation for If D is not close to its maximum value of 1, we can approximate the dynamics to dD/dt = M (regardless of κ) and solve for D(t): In this regime, D(t) follows a linear trajectory.Note that the same is true for (4) when t τ d .

Model of facilitating synapse
The schematic for the facilitating synapse is shown in Figure 2. The difference in this circuit from the depressing synaptic circuit is that the node V x goes to the gate of a pFET instead of an nFET.The synaptic strength is now I syn (t) = I op e κ(Vdd−Vx)/UT and is directly proportional to the current variable I r f = I op e κ(Vdd−Vx)/UT , so where g f = I op e κ(Vdd−Va)/UT and F(t Figure 2: Synaptic facilitation circuit.The circuit on the left is the same as part of the circuit in Figure 1b.The voltage V x determines the synaptic strength and the current I d goes to the neuron.This circuit would have to be inverted so that it can be combined with the neuron circuit in Figure 3.
The update dynamics is multiplicative instead of additive as in Abbott's model: where The recovery dynamics is given by where M = (I op κ/Q T )e −(1−κ)(Vdd−Va)/UT .In steady state, F 1.
Using κ = 0.5, the equation for However, if F is far from its resting value of 1, we obtain the simpler dynamics dF/dt = −MF(t) 2 and solve for F(t): The circuit model for facilitation is quite dissimilar to (7) and (8).Even though the update is multiplicative, the variable F will not increase without bounds because the recovery dynamics of the diode-connected transistor which is a negative-feedback element.In Section 5, we will see that the steady-state value of F is approximately linear in the presynaptic rate r.

NEURON CIRCUIT
The dynamics of the neuron circuit are similar to that of a leaky integrate-and-fire neuron with a constant leak (Figure 3).The circuit is described in detail in [26,27].It is a modified version of previous designs [18,22] and also includes the circuitry which models firing-rate adaptation [21,25] frequently seen in pyramidal cells.The equation for the depolarization of the soma is as follows: where i(t) is the synaptic current to the soma, I leak is the leakage current, and I ahp is the after-hyperpolarization potassium (K) current which causes the adaptation in the firing rate of the cells.When V m (t) increases above V thresh at t = t s (t s is the time of spike), it increases by a step increment determined by the capacitive coupling C 1 and C m .The output V o becomes active at this time and turns on the discharging current path through transistors M 5 and M 6 .The time during which V o remains high, T P , depends on the time taken for V m to discharge below V thresh .In this design, the pulse width T P is determined by the rate at which V m is discharged which in turn depends on the difference between the input current I d , the leak current I leak , and the current I pw .In other designs, V m is reset immediately below V thresh when V o becomes active because either the input current is blocked from charging the membrane or the current I pw is much larger than the input current.The refractory period T R is determined by V refr which keeps V o high so that I d cannot charge up the membrane.The spike output is taken from the node V o .
The time taken for the neuron to charge up to threshold is and, in the case of a constant input current I d , the spike rate is

Spike adaptation
Transistors M 1 to M 4 and the capacitor C a in Figure 3 implement the spike adaptation mechanism.The data in Figure 8b show the adaptation of the output spike rate when the neuron was driven by a 100 Hz regular input spike train through a nonplastic synapse.The amount of charge dumped on C a is determined by V ca .The dynamics of the current mirror circuit (M 3 , M 4 , and C a ) are used to set the dynamics of the I ahp current.The adapted spike rate is reduced from the initial rate by a factor γ = (1+A c q a /Q th ) [21], where q a is the charge that is dumped onto the capacitor C a during each postsynaptic spike (i.e., when V o is high), Q th is the amount of charge needed for V m to reach threshold, and A c = e κVt .

TRANSIENT RESPONSE
The data in the figures in the remainder of the paper are obtained from a multineuron circuit with depressing and facilitating synapses fabricated in a 0.8 µm CMOS process.To show the effect of synaptic depression, we measured V x over time as the input was driven by a regular spike train as shown in Figure 4. Remember that the synaptic strength gD is exponential in V x .When there are no spikes, V x is approximately equal to V a .During a spike, V x is decreased by an amount dependent on V d .This node recovers in-between spikes at a rate that depends on the difference in voltage between V x and V a .The recovery rate is faster when V x is far from V a .The dependence of the recovery rate on this difference is due to the current-mirror circuit dynamics.The parameter V a controls both the synaptic strength and the recovery time constant.For a fixed V a , the dynamics and the steady-state value of D can be set by changing V d (or d) as shown in Figure 4b.
The subsequent effect on the neuron is seen by measuring the EPSP response when a presynaptic spike occurs.The EPSPs recorded when the neuron was stimulated by a regular spiking input through these synapses are shown in Figure 5.The parameters of the synapse and the neuron have been tuned so that the EPSPs do not add up with each incoming spike.In Figure 5a, the EPSP amplitude decreases with each incoming spike, while in Figure 5b the amplitude increases instead.The EPSPs in response to the first few spikes in Figure 5b are not observable because the leak current is larger than the synaptic current.The amplitude reaches a steady-state value after a finite number of spikes.The number of spikes needed to reach a steady state can be tuned by the parameters V a and V d .Different V d and V f values lead to different amounts of depression and facilitation as shown in Figure 6.The fits between the circuit model and the simulation model are described in [14].

Depression and facilitation
We can obtain a combination of facilitation and depression dynamics in I d from the depressing synaptic circuit in Figure 1b by choosing certain circuit parameters.The output of the current-mirror synaptic circuit in Figure 1a can produce paired-pulse facilitation [25].The equation for I d is the same as (15) for I syn in the facilitating synapse circuit and I r f d := I op e κ(Vdd−Vdx)/UT .This equation also applies to I d in Figure 1b.The difference between both circuits is that the factor f d that determines the change in F d right after a spike is constant in one circuit and varies for the other circuit ( f d = e κIr f δt/QT ).In the depressing synaptic circuit, I r f is not constant and depends on the input spike activity, whereas in the current-mirror synaptic circuit, I r f (= I r ) is constant.So, for certain parameter settings in the depressing circuit, the EPSPs show initial facilitation before depressing in response to a step input of a regular 100 Hz spike train as shown in Figure 7.

Depression or adaptation
Both the synaptic depression and spike adaptation mechanisms lead to adaptation in the neuron's firing rate to a step increase in the input rate as shown in Figure 8.In fact, the transient response to a step increase in the rate of a regular spiking input is almost indistinguishable using either mechanism.Although both mechanisms lead to gain control in the neuron, the individual mechanisms are sensitive to different signals.Synaptic depression gives rise to sensitivity in input rate changes whereas spike adaptation makes the neuron sensitive to changes in the neuron's output rate.For example, if one of the inputs to the neuron is highly active, the spike adaptation mechanism of the neuron reduces its sensitivity to the continuous large input current regardless of the origin of the large input.On the other hand, the synaptic depressing mechanism only turns down the sensitivity of that particular active input so the neuron is still selective to all other inputs.The role of depression and facilitation in implementing gain control has been described in [12,28,29].

STEADY-STATE RESPONSE
The dependence of the steady-state values of D and F on the presynaptic frequency can be determined easily in the case of a regular spiking input.In the case of depression, we use (10) and ( 13) to compute the steady-state value of D: For the simpler dynamics of dD/dt = M, we use (14) instead of ( 13) and obtain a simpler expression for D ss : Thus the steady-state EPSP amplitude is inversely dependent on the presynaptic rate r as shown in Figure 9.The form of the curve is similar to the results obtained in the work of Abbott et al. [12] where the data can be fitted with (6).In the case of facilitation, we use ( 16) and ( 18) to compute the steady-state value, F ss : In the simpler case, where dF/dt = −MF(t) 2 ,  which shows that the steady-state value of F is linear in the presynaptic rate and it does not increase without bounds as in the case of the exponential dynamics model for F.

DIRECTION SELECTIVITY USING SHORT-TERM SYNAPTIC DEPRESSION
Depressing synapses have been implicated in the appearance of certain visual cortical cell responses, for example, direction-selectivity.Because these synapses act like a highpass filter in the frequency domain, the response of the neuron shows a phase advance over its response if stimulated through a nonplastic synapse.This feature was exploited in a model that described the direction-selective responses of visual cortical neurons [11].In this model, the neuron was driven by the outputs of a set of cells in the lateral geniculate nucleus (LGN) through depressing synapses and the outputs  of a spatially shifted set of LGN cells through nondepressing synapses.We have attempted the same experiment by driving a "cortical neuron" on our chip with spikes recorded from an LGN cell in the cat visual cortex during stimulation with a drifting sinusoidal grating (courtesy of K. Martin) and a temporally shifted version of these spikes.An example of the direction-selective response is shown in Figure 10 [30].The direction-selective results were qualitatively similar to the data in [11].This chip has been used for exploring other spike-based cortical models, for example, orientation selectivity [27].

CONCLUSION
The addition of short-term dynamical synapses to neuronal networks increases the computational power of such networks, especially in processing time-varying inputs.Because of the similarity of the dynamics of the silicon models to the theoretical models, a silicon network of leaky integrateand-fire neurons which incorporate these synapses can provide an alternative to network simulations on the computer.This type of spike-based network runs in real-time and the computational time does not scale with the size of the network.This chip is a basic module in a reconfigurable, rewireable, and spike-based system that provides ease for prototyping computational models.The system can also be useful for possible applications, for example, in interfacing with neural wetware.

Figure 1 :
Figure 1: Current-mode circuits for a normal synapse (a) and a depressing synapse (b).

Figure 3 :
Figure3: Schematic of the leaky integrate-and-fire neuron.The parameters, V refr sets the refractory period, V thresh sets the threshold voltage, V pw sets the pulse width of the spike, and V leak sets the leak current I leak .The circuit within the dashed-dotted inset implements the spike adaptation mechanism.The parameters V Ca and V t set the adaptation dynamics.

Figure 4 :
Figure 4: Response of V x to a regular spiking input of 20 Hz with different values of V d .(a) Change of V x over time.It is decreased when an input spike arrives and it recovers back to the quiescent value at different rates dependent on its distance from the resting value of about 0.33 V. (b) The steady-state value and dynamics of V x can be tuned by changing V d .

Figure 5 :
Figure5: Transient response of a neuron (by measuring its membrane potential, V m ) when stimulated by a regular spiking input through a depressing synapse (a) and a facilitating synapse (b).The leak current of the neuron has been adjusted so that the neuron does not reach threshold.(a) The EPSP decreases with each incoming input spike for a depressing synapse.(b) The EPSP increases with each incoming input spike for a facilitating synapse.The initial EPSPs are not seen because the leak current of the neuron is larger than the synaptic current.

Figure 6 :
Figure 6: Transient response of a neuron to a regular spiking input for various values of V d and V f .(a) The amount by which each EPSP depresses for each subsequent pulse is set by V d .(b) The amount by which each EPSP facilitates is set by V f .

Figure 7 :
Figure 7: Change in the membrane potential for two different settings of V d and V a .(a) There is initial facilitation of the EPSPs before depression.V d = 0.75 V and V a = 0.2 V. (b) Only depression is seen in the EPSPs.V d = 0.817 V and V a = 0.3 V.

Figure 8 :
Figure 8: Mechanisms for spike frequency adaptation.(a) Adaptation due to synaptic depression.Different adapted rates are obtained by using V d = 0.2 V (top curve) and V d = 0.1 V (bottom curve).(b) Adaptation due to different after-hyperpolarization currents (V ca = 4.3 V (top curve) and V ca = 4.5 V (bottom curve)).The sharp excursions of the membrane potential represent the output spikes of the neuron.

Figure 9 :
Figure 9: Average steady-state EPSP amplitude versus input frequency in the case of a depressing synapse.The curve shows an inverse dependence of the amplitude on the frequency.

Figure 10 :
Figure 10: Response to a drifting 1-Hz sinusoidal grating.The spikes from an LGN cell in the cat visual cortex in response to the drifting grating are depicted at the bottom of the curve.The spikes from a putative spatially shifted LGN cell were generated from these spikes by shifting them in time by 60 ms.The top curve shows the response of the silicon "cortical neuron" when the stimulus drifted in the preferred direction.The sharp excursions at the top of the potential are the output spikes of the neuron.The middle curve shows the response to the stimulus in the null direction.The membrane potential did not build up to threshold.Figure adapted from in [14, Figure 8] with permission.