2
$\begingroup$

Dynamical systems are those whose evolution can be described by a rule, evolves with time and is deterministic. In this context can I say that Neural networks have a rule of evolution which is the activation function $f(\text{sum of product of weights and features})$ ?

Are neural networks

  1. dynamical systems,
  2. linear or nonlinear dynamical systems?

Can somebody please shed some light on this?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ What matters is not what words we use to describe it, so much as what the underlying technical reality is. If you wanted to understand how a car works, trying to figure out whether or not it's OK to call it a "wheeled means of locomotion" isn't the right place to start; instead, you should start by understanding the engineering that goes into it. So, how will you use the answer? Why do you ask? Why do you care? How will it change your life to know whether or not it's OK to use those phrases to describe a neural network? $\endgroup$
    – D.W. ♦
    Commented Jul 29, 2014 at 16:56
  • $\begingroup$ AS per definition of dynamical system goes where there is a component of time explicitly, it is deterministic and can be expressed in terms of phase space; do NN possess these properties so that I can call them dynamical systems. $\endgroup$
    – Ria George
    Commented Jul 29, 2014 at 17:49

3 Answers 3

Reset to default
1
$\begingroup$

A particular neural network does not evolve with time. Its weights are fixed, so it defines a fixed, deterministic function from the input space to the output space.

The weights are typically derived through a training process (e.g., backpropagation). One could imagine building a system that periodically re-applies the training process to generate new weights every so often. Such a system would indeed evolve over time. However, it would be more accurate to call this "a system that includes a neural network as one component of it".

Anyway, at this point we are probably descending into quibbling over terminology, which might not be very productive. This site format is a better fit for objectively answerable questions with some substantive technical content.

Improve this answer
$\endgroup$
2
  • $\begingroup$ A neural network can be seen as a dynamical system when it's still learning I guess though. IIRC there are some papers that try to analyse learning process stability in these terms. $\endgroup$
    – BartoszKP
    Commented Jul 29, 2014 at 19:09
  • $\begingroup$ @D.W:I just checked the Neural network due to the activation function can give rise to chaotic nature.Chaos is a property of nonlinear dynamical system, so how come NN is not?Can you please sehd some light $\endgroup$
    – Ria George
    Commented Jul 30, 2014 at 11:19
2
$\begingroup$

The type of neural networks you have in mind (where each neuron is a threshold function taking on $0$ or $1$ values), with $n$ neurons is a dynamical system with state space $\{0,1\}^n$. After every tick of the clock, the neural network is in some new state, defined by the current values of its neurons.

Because the state space is discrete, I don't think your second question makes sense here. Linear vs. non-linear are terms applied to continuous-time dynamical systems.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your answer. I had the same intuition that they are dynamical system from the perspective of state space. But, I don't agree completely with your argument that non-linear terminology applies only to continuous time dynamical system. There are dynamical system like Tent map, logistic map which are dynamical deterministic system and can be chaotic. Similarly, neural networks can be chaotic also (Ref: K. Aihara, T. Takabe, M. Toyoda, Chaotic neural networks, Physics Letters A, Volume 144, Issues 6–7, 12 March 1990, Pages 333-340; CHAOS IN A NEURAL NETWORK CIRCUIT) $\endgroup$
    – Ria George
    Commented Feb 12, 2015 at 7:28
  • 1
    $\begingroup$ I think you are confusing discrete-time with discrete-space. Yes, the Tent map and Logistic maps are discrete in time, but their state spaces are subsets of the real line. The state of the Tent map at some time $t$ can be any of an uncountable number of different states. The state of a neural network at time $t$ can only be one of $2^n$ different states, where $n$ is the number of neurons in the network. My guess is that the paper you are referencing is not using a neural network of this form. $\endgroup$
    – MarkG
    Commented Feb 12, 2015 at 13:18
  • $\begingroup$ Thank you for clearing the terminologies. I will look at them carefully. $\endgroup$
    – Ria George
    Commented Feb 13, 2015 at 4:09
1
$\begingroup$

There are many different forms that a neural network can take. Some neural networks are trained before they are actually used in 'real world' scenarios. Other neural networks adjust themselves as they run, for instance a neural network with some sort of feedback loop or basic memory retention.

As for your second question, again there are different types of neural networks that can be either! If you consult the wikipedia page for Neural Networks there are lots of types of ANNs that are described:

http://en.wikipedia.org/wiki/Types_of_artificial_neural_networks#Dynamic_neural_networks

If you would like to do some further reading on types of neural networks, then I would suggest the book "Machine Learning" by "Tim Mitchel". It has a large section on Neural Networks and is reader friendly for beginners in this area.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.