Check out these links where there is some success in processing video with neural networks using a Convolutional Neural Network (CNN) followed by a Recursive Neural Network (RNN) using LSTM’s (Long Short-term Memory) modules.

• Large-scale Video Classification with Convolutional Neural Networks
• Deep Learning for Video Classification and Captioning
• Continuous online video classification with TensorFlow, Inception and a Raspberry Pi
• Five video classification methods implemented in Keras and TensorFlow
• Continuous video classification with TensorFlow, Inception and Recurrent Nets