A novel method for optimising “Deep Differential Logic Gate Networks”.
Abstract
Logic Gate Networks are neural networks in which the neurons are formed by binary logic gates. As they are built from logic gates (such as NAND and XOR) only, they allow orders of magnitude more efficient inference compared to conventional neural networks.
Advantages
- Significantly increased efficiency over conventional neural networks with the same hardware (increased by 1 to 4 orders of magnitude, depending on hardware and comparison method. 1-2 orders of magnitude compared to the most efficient methods to date, 2.5-4 orders of magnitude compared to regular methods).
- Higher accuracy
- Lower memory requirements
Fields of application
IoT Devices, Machine Learning on the edge, FPGA Applications, Control Devices, Time Critical Applications, ASIC Applications, Embedded Systems, Formal Verification of AI, More Efficient Cloud Inference, AI on Mobile Devices, Video / Image / Audio Compression / Encoding / Decoding.
Background
In “edge AI” and “embedded machine learning” approaches, models are executed directly on the end devices. This reduces data protection complexity, effort, and cost compared to conventional approaches where raw data has to be processed in the cloud. Additionally, it enables AI to use cases where conventional methods require too much computing capacity.
Logic Gate Networks also enable applications where decisions need to be made in less than a microsecond (or a few nanoseconds).
Problem
Due to hardware limitations, complex tasks and sophisticated models, AI often cannot be implemented on edge devices and embedded devices, e.g., because of memory constraints or when the processing time exceeds the desired limits.
Solution
A novel algorithm developed at the University of Konstanz increases AI applications' computational power and efficiency while maintaining the same hardware. For this purpose, Logic Gate Networks are learned via the gradient method.
The so-called "Deep Differentiable Logic Gate Networks" correspond to neural networks in which each neuron is formed by a binary logic gate and thus has only two inputs. Here, classical logical operators are replaced by real-valued logical operators and discrete decisions by weighted combination. To learn the network, the relaxation of the operators and the network enables the gradient descent method, which cannot be used with a classical logic gate network. The learned Deep Differential Logic Gate Networks can then be converted back into classical logic gate networks.
This results in increased performance over previous models, which have been validated on MONK's Problems, Adult and Breast Cancer, MNIST, and CIFAR-10 benchmarks.
![[Translate to english:] Figure 1: Overview of Logic Gate Networks: Boolean inputs are processed by a layer of neurons such that each neuron receives two inputs. The connectivity of the neurons remains fixed after an initial pseudo-random initialisation. Each neuron is continuously parameterised by a distribution over a logical operator. During training, this distribution is learned for each neuron, and the most likely operator for each neuron is used during inference. Multiple outputs per class are aggregated by bit counting, resulting in the class values. [Department of Computer and Information Science, Univerity of Konstanz]](/fileadmin/user_upload/img/Technologieangebote/21_062_Abb..png "[Translate to english:] Figure 1: Overview of Logic Gate Networks: Boolean inputs are processed by a layer of neurons such that each neuron receives two inputs. The connectivity of the neurons remains fixed after an initial pseudo-random initialisation. Each neuron is continuously parameterised by a distribution over a logical operator. During training, this distribution is learned for each neuron, and the most likely operator for each neuron is used during inference. Multiple outputs per class are aggregated by bit counting, resulting in the class values. [Department of Computer and Information Science, Univerity of Konstanz]")
[Translate to english:] Figure 1: Overview of Logic Gate Networks: Boolean inputs are processed by a layer of neurons such that each neuron receives two inputs. The connectivity of the neurons remains fixed after an initial pseudo-random initialisation. Each neuron is continuously parameterised by a distribution over a logical operator. During training, this distribution is learned for each neuron, and the most likely operator for each neuron is used during inference. Multiple outputs per class are aggregated by bit counting, resulting in the class values. [Department of Computer and Information Science, Univerity of Konstanz]