AN ANALOG IMPLEMENTATION OF THE BOLTZMANN MACHINE WITH PROGRAMMABLE LEARNING ALGORITHMS
INTRODUCTION
Most current neural nets experiments use the MultiLayer Perceptrons and the Backpropagation leaming algorithms. Whereas the experimentation speed is insufficient, few analog integrated circuits have been realized for these algorithms, because neither their implementation nor their parallelization is obvious. On the other hand, Boltzmann Machines show a number of very attractive features, including high recognition rates, but their simulations are desperately slow. Therefore mixed analog/digital implementations have been described, whose leaming algorithm is hardwired. However, several learning algorithms were recently proposed by Azencott (1989) and (1990). Whereas they have a more general scope than the older algorithms, their effectiveness for the application of analog circuits has to be assessed. For this purpose, we present in this paper a new analog/digital implementation of the Boltzmann Machine whose learning algorithm is programmable. We present the Boltzmann Machine and its different learning algorithms in the first section. We describe a faithful analog implementation of the Boltzmann machine relaxation in the second section. We then introduce a new architecture for the programmable synaptic cell in the third section and a prototype implementation in the fourth section.
BOLTZMANN MACHINES
The Boltzmann Machine model was introduced by Hinton and Sejnowski in 1984. It is an asynchronous model, where a single neuron updates its state at each iteration. However, in a parallel hardware implementation, it is natural for all the neurons to update their state simultaneously at each iteration. Therefore, a new model was introduced by Azencott in 1989 to deal with this case, which is called the Synchronous Boltzmann Machine. We describe now the operation of these models.
Several mixed analog/digital implementations were described. They included a hardwired learning algorithm with the thresholded weight update rules of asynchronous networks.
It is not proved that these rules are the best suited to an analog implementation. Whereas software simulations were performed to assess this point, they do not capture enough characteristics of the analog circuits to lead to a definite conclusion. Therefore it is interesting to provide the possibility of experimenting with different leaming algorithms in an analog circuit and we propose a new architecture for this purpose. We describe in the two following sections firstly its analog part and then its programmable part.
ANALOG CELLS
The functional analog cells implement the computations and store the variables they require. Firstly, the synaptic cell is in charge of the storage of the weight, its learning or refreshing, and the computation of one Wij Xjn product. The synaptic cell is divided into two subcells. The first one is in charge of the relaxation of the network: it is an analog cell and it is described in this section. The second one is a digital processor used for weight learning and it will be described in the next section.
The network contribution is computed according to the equation by a single rail current summation. The analog part of the synaptic cell includes a linear voltage-to-current converter driven by the weight capacitor. The neuron cell is in charge of the update of the neuron state. It takes as input the sum 1i of the currents 1ij which represents the network contribution this current is then converted into a voltage. So that the converter gain implements the temperature T.
A sigmoidal function is then applied to the output of this converter. Finally, the neuron cell has two inputs, the random voltage and the current li, and one output, the state xi. It consists of a current to voltage converter, a sigmoidal function, an integrator, and a comparator. The minimum number of transistors for this cell is fifteen.
The random tossing required by the Boltzmann machine results in some troubles: the use of resistor thermal noise leads practically to correlated generators, and confirmed experimentally. The uniform law is built out of the integration of a sequence of unbiased independent binary values. This integration is performed by a switched-capacitor filter. As the states sequences of distant ce11s are uncorrelated, we use site spacing in order to generate in parallel the binary inputs of the neuron cells. This is a major advantage over the shift register imp1ementation. Moreover, the modular approach of this realization and its connection scheme is well suited to an integrated e1ectronic realization.
ARCHITECTURE OF TUE PROGRAMMABLE SYNAPTIC CELL
I propose a new architecture for the synaptic cell. It is built out of a bit-serial digital processor, called hereafter the processor, a digital-to-analog converter, and the analog cells included in the voltage to current converter described in the previous section.
Implementation
The implementation of the processor is depicted in Figure 5. It is similar to a usual bit-serial processor such as the GAPP from NCR. The processor includes an l-bit wide digital SRAM, 3 I-bit registers, and an ALU. This ALU is I-bit wide also, and it performs all the computation serially. It is very efficient for medium precision additions, subtractions, and scaling by powers of 2. The weight Wij is output from the digital SRAM in order to be used by the analog voltage to current converter described in the previous section. For this purpose, a serial digital to analog converter has been used, and this is homogeneous with the architecture of the bit-serial processor.
CONCLUSION
In this blog, I have described the architecture of a new synaptic cell for the mixed digital-analog implementation of the Boltzmann Machines, and we introduced several original points. This synaptic cell is programmable, and this allows to tailor of the learning algorithm for the network and the application under study. Simple algorithms may be used for small networks learning easy tasks, whereas complex algorithms may be preferred for large networks whose learning is difficult. As the algorithms are performed in a bit-serial way, their duration is proportional to the complexity of the algorithm. Finally, the width of the variables which are used for the learning, such as the cooccurrences or the weight, may easily be adapted to the task under learning.
References
Alspector J. et al., “A neuromorphic V.L.S.I.learning system”, Stanford Conference on VLSI, MJ.T. Press, 1987
Azencott R., “Boltzmann Machines: high-order interactions and Synchronous learning”, Procs Stochastic models, statistical methods and algorithms in image analysis, Ed. by P.Barone and AFrigessi, Lecture Notes in Statistics, Springer-Verlag, 1990
Garda P.& al., “An analog circuit with digital 110for Synehronous Boltzmann Machines”, VLSI for Artificial Intelligence and Neural Networks, Oxford, September 2–4 1990, Ed. by J.Delgado-Frias et W. Moore, Plenum Publishing Corp., 1991, pp. 245–253
Hinton G. et al., “Boltzmann Machines”, C.M.U. Technical Report CMU-CS-84–119, Carnegie Mellon University, 1984
