Check out these links where there is some success in processing video with neural networks using a Convolutional Neural Network (CNN) followed by a Recursive Neural Network (RNN) using LSTM’s (Long Short-term Memory) modules.

- Large-scale Video Classification with Convolutional Neural Networks
- Deep Learning for Video Classification and Captioning
- Continuous online video classification with TensorFlow, Inception and a Raspberry Pi
- Five video classification methods implemented in Keras and TensorFlow
- Continuous video classification with TensorFlow, Inception and Recurrent Nets