Alternative Operations
Moving image installation
Shown here installed at Victoria & Albert Museum Digital Design Weekend.
Alternative Operations uses real-time action classification to display training videos from Moments in Time. At the time of its release it was the largest action classification dataset, but contained just 339 categories. As visitors move through the space, their actions are classified by a 2D ResNet model pre-trained on this dataset, triggering corresponding training videos to play back to them.
Moments in Time is a collection of one million annotated 3-second videos - designed to match the average duration of human short term memory - scraped primarily from YouTube and categorised by Amazon Mechanical Turk workers under MIT supervision. Their taxonomy is intentionally vague: 'opening' could refer to a door, a gate, or an eye. This vagueness forces overworked and underpaid Mechanical Turkers to make decisions across linguistic and cultural barriers, resulting in a dataset that often fails. In extreme cases, as Everest Pipkin has pointed out, categories like 'kissing' and 'hitting' document non-consensual acts and excessive violence.
The model itself is persistently inaccurate, defaulting to categories with higher visual consistency and compulsively seeking to recognize action even when none is performed. Through this process of interaction, visitors are encouraged to push the model's limits—performing movements that resist classification and reflecting on what their bodies are capable of doing beyond the dataset's narrow taxonomy of human action.
