Are non-ordered (Slabbed) Neural Networks in wide modern use and what does Fiesler (1994) mean by Clamping Function and Ontogenic function?

Question

In Fiesler (1994) Neural Network Classification and Formalization, he talks a lot about a more general version of neural networks, one that is not ordered into layers, but rather the network is called slabbed and the "layers" are instead called clusters. Unlike for feed-forward networks, slabbed networks may be cyclical and terminate when a stable state has been reached, rather than when propagation has reached output neurons.

Much of this paper is outdated (terminology in particular). Is this an outdated concept or are these types of networks being used and/or researched?

Also from this paper, he mentions "clamping functions", but I am not sure what he means. Here is an excerpt where he talks about clamping functions:

"clamping functions, which determine if and when certain neurons will be insusceptible to incoming information, that is, they retain their present activation value independent of external stimuli"

I have seen a few more recent uses of "clamping" in literature, but they tend to be referring to something different. What exactly is he talking about here and is there a modern name for this?

Similarly, he refers to ontogenic functions and ontogenic networks, which I believe we call dynamic networks, but what is the ontogenic function? I understand it dictates how the network changes, but mathematically what are the inputs and outputs? What triggers the change to start and/or stop?

I am very interested in this formal definition of neural network as a mathematical object (this tuple with topology, interconnection structure, transition functions and intial state), does anyone know of a source that gives a similarly rigorous definition of a neural network, but with more modern terminology and ideology (I imagine some developments in theory may have allowed for networks to be more general)?

Answer 1

I have not heard of recent work on this kind of thing, but there's a huge amount of literature and I know only a tiny slice of it.

Today much of the work on neural networks is concerned with feedforward neural networks, which don't have a cyclical structure. The most common form is

$$z(x) = L_n(\cdots (a(L_1(x))))$$

where $f:\mathbb{R}^n \to \mathbb{N}^k$, each $L_i$ is a linear function, and each $a$ represents an activation function: i.e., $a(x_1,\dots,x_d) = (a(x_1),\dots,a(x_d))$ and $a$ is a simple nonlinear function called the activation function. The ReLU $a(x)=\max(0,x)$ is a common choice of activation function. There is some cleverness in the structure of the linear layers. Finally, the output of the neural network is $f(x)=\text{softmax}(z(x))$ where

$$\text{softmax}(z_1,\dots,x_k) = ({e^{z_1} \over e^{z_1} + \dots + e^{z_k}}, \dots, {e^{k_1} \over e^{z_1} + \dots + e^{z_k}}).$$

This is not the only architecture, but it is probably the most commonly seen, and as you can see, it has a relatively clean formalization.