The computational systems known as cellular automata reproduce patterns of pixels by iteratively applying simple rules based loosely on the behavior of biological cells. New work extends their utility from reproducing images to generating new ones.
What’s new: Rasmus Berg Palm and colleagues at IT University of Copenhagen developed an image generator called Variational Neural Cellular Automata (VNCA). It combines a variational autoencoder with a neural cellular automaton, which updates pixels based on the output of a neural network and the states of neighboring pixels.
Key insight: A variational autoencoder (VAE) learns to generate data by using an encoder to map input examples to a distribution and a decoder to map samples of that distribution to input examples. Any architecture can serve as the decoder, as long as it can reconstruct data similar to the inputs. Given a distribution, a neural cellular automaton can use samples from it to generate new, rather than predetermined, data.
How it works: VNCA generates pixels by updating a grid of vectors, where each vector is considered a cell and each cell corresponds to a pixel. The encoder is a convolutional neural network, and the decoder is a neural cellular automaton (in practical terms, a convolutional neural network that updates vectors depending on the states of neighboring vectors). The authors trained the system to reconstruct images in the MNIST dataset of handwritten digits.
- The encoder learned to map an input image to the parameters of a Gaussian distribution. The system used this distribution to produce a two-by-two matrix of cells.
- The decoder updated each cell’s state based on that of its neighbors. After the update, the system duplicated each cell into a new two-by-two matrix while pushing other cells out of the way (see diagram above). This process was repeated to fill the output pixel resolution with cells.
- VNCA applied a sigmoid to the first value of each cell’s vector to determine the probability that the associated pixel should be white or black.
- The loss function encouraged the output image to be as similar as possible to the input while also encouraging the encoder to map images to the parameters of the standard Gaussian distribution, which has a mean of 0 and variance of 1.
- At inference, the decoder received a two-by-two matrix sampled from the standard Gaussian distribution and generated a new image accordingly.
Results: The authors showed that a cellular automaton can generate images, though not very well at this point. They evaluated VNCA using log likelihoods in natural units of information (nats), which gauge similarity between the system’s output and the training data (higher is better). VNCA achieved -84.23 nats, worse than the -77 nats achieved on MNIST by state-of-the-art models such as NVAE and BIVA.
Why it matters: This work demonstrates that a neural cellular automaton can generate new images. While it shows no clear advantage of using a neural cellular automaton in a VAE, the combination might lend itself to useful applications. For instance, neural cellular automata have an inherent regenerative ability: Deface an image, and they can regrow the damaged pixels. Thus a VNCA-type approach might be useful for image inpainting. Given an image, the encoder could map it to a Gaussian distribution. Then you could damage the image where you wanted to change it, sample from the distribution, and use the decoder to generate novel pixels in that area.
Yes, but: This approach may be challenging to scale. VNCA’s decoder used only 1.2 million parameters rather than the hundreds of millions used in other high-performing decoders. Adding parameters would increase its computational cost significantly, since it updates cells repeatedly based on the states of neighboring cells.
We’re thinking: Deep learning offers a widening array of neural image generators: GANs, VAEs, diffusion models, normalizing flows, and more. While each has its advantages and disadvantages, together they amount to an enticing playground for synthesizing data and producing visual art.