Wouldn’t it be interesting to do machine learning with analog circuits?
Lately, large language models have become popular, but they require a lot of GPUs, money, and electricity. I thought it would be fascinating to try doing this with analog circuits.
The calculations performed by neural networks are essentially multiplication and addition.
So, this time I’ll give a rough overview starting from the basic theory.
What is analog computation?
In digital computers, numbers are represented as combinations of 0s and 1s. Even if the voltage changes slightly during a calculation, the original information can be recovered as long as the change stays within the range where 0 and 1 can still be distinguished.
In contrast, an analog computer maps continuous physical quantities such as voltage and current to numerical values.
For example, if we define 0 V as 0 and 1 V as 1, then 0.5 V can be treated as 0.5. In practice, the relationship between numbers and voltages is set according to the circuit’s supply voltage and the range of values you wish to handle.
An analog computer doesn’t execute each computational step sequentially; instead, it uses the physical phenomena themselves to perform the calculation.
When a voltage is applied to a circuit, current flows, and the entire circuit settles into a state governed by physical laws. By measuring the voltage and current at that moment, you obtain the result of the computation.
Of course, the voltages and currents in analog circuits are affected by noise, temperature variations, component tolerances, and other factors. Because dealing with these difficulties is cumbersome, digital computers have become the mainstream today.
Nevertheless, for tasks that can tolerate some error—such as neural networks—analog computation may be a viable option.
Neural Network Computation
For now, let’s consider only the most basic element of a neural network: the neuron.
A neuron receives multiple inputs, multiplies each by a weight, and then sums them.
If we denote the inputs as x_1, x_2, ..., x_n and the weights as w_1, w_2, ..., w_n, the first step is to compute the following:
$$ z=\sum_{i=1}^{n}w_i x_i+b $$
b is the bias.
After that, the result z is passed through an activation function.
$$ y=f(z) $$
Thus, a neuron’s processing can be divided into two parts:
- Multiply each input by its weight and sum all the products.
- Pass the summed value through an activation function.
The first operation is called a multiply‑accumulate (MAC) operation. Neural networks perform this MAC operation many times.
So, let’s consider how a MAC operation could be expressed using voltage and current.
Performing Multiplication‑like Operations with Ohm’s Law
When a voltage V is applied to a resistor, Ohm’s law causes a current I to flow.
$$ I=\frac{V}{R} $$
R is the resistance value.
Here, we’ll call the reciprocal of the resistance the conductance G.
$$ G=\frac{1}{R} $$
Using this, Ohm’s law can be written as follows.
$$ I=GV $$
This form is essentially a multiplication.
If we map the neural‑network input x to a voltage V and the weight w to a conductance G, the relationship becomes:
| Neural Network | Analog Circuit |
|---|---|
Input x | Voltage V |
Weight w | Conductance G |
Product wx | Current GV |
Simply applying a voltage to a resistor yields a current that corresponds to the product of the input and the weight.
In other words, instead of executing a multiplication instruction on a digital computer, we can represent the product using the properties of resistors and voltage.
However, in an actual circuit the numeric value x is not directly turned into a voltage V. For example, you need to set an appropriate scaling—such as limiting the input range to 0 – 1 and mapping it to 0 – 1 V.
Performing Addition‑like Operations with Kirchhoff’s Law
A single neuron has multiple inputs.
When each input voltage V_i is applied to a conductance G_i that corresponds to its weight, the following current flows through each branch:
$$ I_i=G_iV_i $$
These currents are then collected at a common node.
According to Kirchhoff’s current law, the total current flowing into the node equals the total current flowing out of the node.
If the circuit is designed so that the current entering from the input side is routed to the output side, the output current becomes:
$$ I_{\mathrm{out}}=I_1+I_2+\cdots+I_n $$
Substituting I_i=G_iV_i for each current gives:
$$ I_{\mathrm{out}}=G_1V_1+G_2V_2+\cdots+G_nV_n $$
In summary,
$$ I_{\mathrm{out}}=\sum_{i=1}^{n}G_iV_i $$
That is the case.
This takes the same form as the neuron’s multiply‑accumulate operation
$$ z=\sum_{i=1}^{n}w_ix_i $$
Thus, by using Ohm’s law for multiplication and Kirchhoff’s current law for addition, we can create a circuit that implements the neuron’s multiply‑accumulate operation.
Trying Some Numbers
Let’s plug actual numbers into a neuron that has two inputs.
We set the inputs and weights as follows.
$$ x_1=2,\qquad x_2=3 $$
$$ w_1=0.5,\qquad w_2=0.2 $$
If we compute normally, the result is:
$$ z=0.5\times2+0.2\times3=1.6 $$
Expressed in terms of the circuit, this becomes:
$$ V_1=2,\qquad V_2=3 $$
$$ G_1=0.5,\qquad G_2=0.2 $$
The currents flowing through each resistor are
$$ I_1=G_1V_1=0.5\times2=1 $$
$$ I_2=G_2V_2=0.2\times3=0.6 $$
respectively.
When the two currents are combined, we obtain
$$ I_{\mathrm{out}}=I_1+I_2=1.6 $$
which matches the value produced by digital computation.
This does not mean that the actual circuit must use voltages of 2 V or 3 V. Instead, a scaling factor is set for the correspondence between numbers and voltages, and between weights and conductances. Here, for clarity, we map the numbers directly to voltages.
Bias and Negative Weights
In addition to the weighted sum of inputs, a neuron’s computation includes a bias b.
$$ z=\sum_{i=1}^{n}w_ix_i+b $$
A bias can be treated like a regular weight by adding an input that is constantly 1.
$$ z=\sum_{i=1}^{n}w_ix_i+w_b\cdot1 $$
In a circuit, a bias can be represented by supplying a fixed reference voltage as an extra input, or by using a constant current source.
There is another issue here. The conductance of a resistor is fundamentally a positive value.
$$ G\geq0 $$
However, neural‑network weights can be negative.
Therefore, we split each weight into a positive component and a negative component.
$$ w=w^+-w^- $$
By generating separate currents for the positive and negative parts and then taking their difference,
$$ I_{\mathrm{out}}=I^+-I^- $$
we can represent a negative weight.
This allows us to handle negative values that cannot be expressed with resistors alone. However, it requires additional circuitry, so the next circuit design will need to take these considerations into account.
How to Implement an Activation Function
A multiply‑accumulate operation alone is insufficient for a neural network.
Since a multiply‑accumulate is a linear operation, stacking multiple linear layers can be reduced to a single linear operation. To handle complex patterns, a nonlinear activation function must be inserted between the layers.
A common activation function is the ReLU.
$$ \operatorname{ReLU}(z)=\max(0,z) $$
It outputs the input unchanged when the input is positive, and outputs zero when the input is negative.
In analog circuits, using the nonlinear characteristics of diodes, transistors, or operational amplifiers should make it possible to implement a ReLU‑like function.
However, real components do not behave exactly according to the ideal equations; they may not switch abruptly at zero, and leakage currents can appear.
At this point, it’s no longer as simple as just wiring together resistors. Still, it would be interesting if we could at least reproduce the multiply‑accumulate operation with an analog circuit.
Does an Analog Circuit Have Infinite Precision?
Because the voltages and currents in analog circuits are continuous quantities, one might think they can compute with infinite precision.
In reality, however, that doesn’t happen. Various sources of error are present, such as:
- Manufacturing variations in resistance values and other component tolerances
- Thermal noise
- Fluctuations in the supply voltage
- Temperature‑induced changes in component characteristics
- Wiring resistance
- Measurement errors of the instruments
For example, even if you set the conductance to represent a weight of 0.5, the actual value may differ slightly.
In addition, you will likely need a DAC to convert digital data into an input voltage and an ADC to convert the circuit’s output back into a digital value. Even if the internal calculations are fast, the conversion at the input and output can consume time and power.
Analog circuits are not always faster or more power‑efficient than digital computers. It’s necessary to consider which parts of the computation should be handled by an analog circuit.
That said, we’ve realized that some portions of neural‑network calculations could be performed as changes in voltage and current.
Next Steps
Next, I plan to design a circuit.
For now, I’ll imagine a small neuron with two inputs and examine what current flows when a voltage is applied to a resistor. If needed, I plan to incorporate op‑amps or other components so that the result of the multiply‑accumulate operation can be measured.
It would be fascinating to see whether the experiment yields the results predicted by theory, or if simply connecting resistors isn’t enough.