Return to Answer

LSTM is designed to process a stream of data chunks (each chunk being the set of inputs for the network at this point in time) that arrive over time and observe features occurring in the data and yield output accordingly. The time lag (delay) between the occurrence of features to recognize may vary and may be prolonged.

One would then train the network by streaming training examples in randomized ordering which should also have some timeshift noise added in the form of idle passes (have the network activate when inputs are at default idle values eg: when no audio in the case of a speech processor) [exception: if any training data should obey periodic timeshift patterns such as music then the timeshift noise should keep the timeshifting synchronized eg: in music making sure a start-of-measure training example isn't shifted to mid-measure and so forth]

It is possible also to have a semi supervised setup where the network is always in a training configuration and it's trained with examples that expect output of an idle value when no feauture is present or the appropriate expected value when a feature is presented to train).

If feedback format training is desired it can be emulated by:

1. saving the internal state (time t)
2. activating the network on current inputs (now at t+1)
3. supervisory process evaluates the output obtained at t
   • 3a if correction is needed first rewind to the saved state (rewinds network back to t)
   • 3b generate a training example with the correction
   • 3c run a train (backprop) pass for this slice rather than an activation

thus one implements a feedback style system since training examples are basically only created while the network is "getting it wrong." The feedback format is useful if one wants the network to attempt improvisation (like Schmidhuber's music example).

• it should be pointed out that part of the correction feedback (and thus training examples) necessarily includes those that enforce idle valued output when features are not present at the current time

A note about resetting of the network

Any reset or recalling of a saved network state in the LSTM sense has a meaning of "go back in time" thus undoing any learning the LSTM performed prior to the reset.

Thus you were correct in stating LSTMs are not reset prior to each training sample nor tranining epoch. LSTMs want their data streamed, or provided in an 'online' manner, so-to-speak.

LSTM is designed to process a stream of data chunks (each chunk being the set of inputs for the network at this point in time) that arrive over time and observe features occurring in the data and yield output accordingly. The time lag (delay) between the occurrence of features to recognize may vary and may be prolonged.

One would then train the network by streaming training examples in randomized ordering which should also have some timeshift noise added in the form of idle passes (have the network activate when inputs are at default idle values eg: when no audio in the case of a speech processor) [exception: if any training data should obey periodic timeshift patterns such as music then the timeshift noise should keep the timeshifting synchronized eg: in music making sure a start-of-measure training example isn't shifted to mid-measure and so forth]

It is possible also to have a semi supervised setup where the network is always in a training configuration and it's trained with examples that expect output of an idle value when no feauture is present or the appropriate expected value when a feature is presented to train).

If feedback format training is desired it can be emulated by:

1. saving the internal state (time t)
2. activating the network on current inputs (now at t+1)
3. supervisory process evaluates the output obtained at t
   • 3a if correction is needed first rewind to the saved state (rewinds network back to t)
   • 3b generate a training example with the correction
   • 3c run a train (backprop) pass for this slice rather than an activation

thus one implements a feedback style system since training examples are basically only created while the network is "getting it wrong." The feedback format is useful if one wants the network to attempt improvisation (like Schmidhuber's music example).

• it should be pointed out that part of the correction feedback (and thus training examples) necessarily includes those that enforce idle valued output when features are not present at the current time

It was mentioned by the OP that [there is no separation of inputs] except that actually there is. If one thinks of a voice recognition scenario one has periods of utterances (features the LSTM should detect) interspersed with long periods of silence. So to address the concern it would be fair to say those periods of silence are in fact separating the sequenced groups of inputs (those silences too are actually a feature the network needs to detect and learn to respond with idle valued outputs ie: learn to do nothing when silence).

A note about resetting of the network

Any reset or recalling of a saved network state in the LSTM sense has a meaning of "go back in time" thus undoing any learning the LSTM performed prior to the reset.

Thus you were correct in stating LSTMs are not reset prior to each training sample nor tranining epoch. LSTMs want their data streamed, or provided in an 'online' manner, so-to-speak.

LSTM is designed to process a stream of data chunks (each chunk being the set of inputs for the network at this point in time) that arrive over time and observe features occurring in the data and yield output accordingly. The time lag (delay) between the occurrence of features to recognize may vary and may be prolonged.

One would then train the network by streaming training examples in randomized ordering which should also have some timeshift noise added in the form of idle passes (have the network activate when inputs are at default idle values eg: when no audio in the case of a speech processor) [exception: if any training data should obey periodic timeshift patterns such as music then the timeshift noise should keep the timeshifting synchronized eg: in music making sure a start-of-measure training example isn't shifted to mid-measure and so forth]

It is possible also to have a semi supervised setup where the network is always in a training configuration and it's trained with examples that expect output of an idle value when no feauture is present or the appropriate expected value when a feature is presented to train).

If feedback format training is desired it can be emulated by:

1. saving the internal state (time t)
2. activating the network on current inputs (now at t+1)
3. supervisory process evaluates the output obtained at t
   • 3a if correction is needed first rewind to the saved state (rewinds network back to t)
   • 3b generate a training example with the correction
   • 3c run a train (backprop) pass for this slice rather than an activation

thus one implements a feedback style system since training examples are basically only created while the network is "getting it wrong." The feedback format is useful if one wants the network to attempt improvisation (like Schmidhuber's music example).

• it should be pointed out that part of the correction feedback (and thus training examples) necessarily includes those that enforce idle valued output when features are not present at the current time

LSTM is designed to process a stream of data chunks (each chunk being the set of inputs for the network at this point in time) that arrive over time and observe features occurring in the data and yield output accordingly. The time lag (delay) between the occurrence of features to recognize may vary and may be prolonged.

One would then train the network by streaming training examples in randomized ordering which should also have some timeshift noise added in the form of idle passes (have the network activate when inputs are at default idle values eg: when no audio in the case of a speech processor) [exception: if any training data should obey periodic timeshift patterns such as music then the timeshift noise should keep the timeshifting synchronized eg: in music making sure a start-of-measure training example isn't shifted to mid-measure and so forth]

It is possible also to have a semi supervised setup where the network is always in a training configuration and it's trained with examples that expect output of an idle value when no feauture is present or the appropriate expected value when a feature is presented to train).

If feedback format training is desired it can be emulated by:

1. saving the internal state (time t)
2. activating the network on current inputs (now at t+1)
3. supervisory process evaluates the output obtained at t
   • 3a if correction is needed first rewind to the saved state (rewinds network back to t)
   • 3b generate a training example with the correction
   • 3c run a train (backprop) pass for this slice rather than an activation

thus one implements a feedback style system since training examples are basically only created while the network is "getting it wrong." The feedback format is useful if one wants the network to attempt improvisation (like Schmidhuber's music example).

• it should be pointed out that part of the correction feedback (and thus training examples) necessarily includes those that enforce idle valued output when features are not present at the current time

A note about resetting of the network

Any reset or recalling of a saved network state in the LSTM sense has a meaning of "go back in time" thus undoing any learning the LSTM performed prior to the reset.

Thus you were correct in stating LSTMs are not reset prior to each training sample nor tranining epoch. LSTMs want their data streamed, or provided in an 'online' manner, so-to-speak.

LSTM is designed to process a stream of data chunks (each chunk being the set of inputs for the network at this point in time) that arrive over time and observe features occurring in the data and yield output accordingly. The time lag (delay) between the occurrence of features to recognize may vary and may be prolonged.

One would then train the network by streaming training examples in randomized ordering which should also have some timeshift noise added in the form of idle passes (have the network activate when inputs are at default idle values eg: when no audio in the case of a speech processor) [exception: if any training data should obey periodic timeshift patterns such as music then the timeshift noise should keep the timeshifting synchronized eg: in music making sure a start-of-measure training example isn't shifted to mid-measure and so forth]

It is possible also to have a semi supervised setup where the network is always in a training configuration and it's trained with examples that expect output of an idle value when no feauture is present or the appropriate expected value when a feature is presented to train).

If feedback format training is desired it can be emulated by:

1. saving the internal state (time t)
2. activating the network on current inputs (now at t+1)
3. supervisory process evaluates the output obtained at t
   • 3a if correction is needed first rewind to the saved state (rewinds network back to t)
   • 3b generate a training example with the correction
   • 3c run a train (backprop) pass for this slice rather than an activation

thus one implements a feedback style system since training examples are basically only created while the network is "getting it wrong"

• it should be pointed out that part of the correction feedback (and thus training examples) necessarily includes those that enforce idle valued output when features are not present at the current time

LSTM is designed to process a stream of data chunks (each chunk being the set of inputs for the network at this point in time) that arrive over time and observe features occurring in the data and yield output accordingly. The time lag (delay) between the occurrence of features to recognize may vary and may be prolonged.

One would then train the network by streaming training examples in randomized ordering which should also have some timeshift noise added in the form of idle passes (have the network activate when inputs are at default idle values eg: when no audio in the case of a speech processor) [exception: if any training data should obey periodic timeshift patterns such as music then the timeshift noise should keep the timeshifting synchronized eg: in music making sure a start-of-measure training example isn't shifted to mid-measure and so forth]

It is possible also to have a semi supervised setup where the network is always in a training configuration and it's trained with examples that expect output of an idle value when no feauture is present or the appropriate expected value when a feature is presented to train).

If feedback format training is desired it can be emulated by:

1. saving the internal state (time t)
2. activating the network on current inputs (now at t+1)
3. supervisory process evaluates the output obtained at t
   • 3a if correction is needed first rewind to the saved state (rewinds network back to t)
   • 3b generate a training example with the correction
   • 3c run a train (backprop) pass for this slice rather than an activation

thus one implements a feedback style system since training examples are basically only created while the network is "getting it wrong." The feedback format is useful if one wants the network to attempt improvisation (like Schmidhuber's music example).

• it should be pointed out that part of the correction feedback (and thus training examples) necessarily includes those that enforce idle valued output when features are not present at the current time