Machine Understanding for interactive StorytElling (EU ICT FP7 FET)


The MUSE project introduces a new way of exploring and understanding textual information by “bringing text to life” through 3D interactive storytelling. Taking as input natural language text like children’s stories or medical patient education materials, MUSE will process the natural language, translate it into formal knowledge that represents the actions, actors, plots and surrounding world by means of advanced machine learning techniques, and then render these as virtual 3D worlds in which the user can explore through interaction, re-enactment and guided game play.

MUSE’s long-term goal is to enable the innovative text-to-virtual-world translation system to be used in many applications and a variety of domains. Current information communicated to citizens is often very complex making written text difficult to understand. Comparable to the invention of symbolic writing systems several millennia ago, MUSE contributes to a novel symbolic system of communicating natural language utterances.

LIIR is involved in the coordination of the project and in the advanced natural language processing research.


LIIR coordinates the MUSE project. KU Leuven co-promotors are Prof. Steven Bethard (partially affiliated with the University of Boulder, USA) and Prof. Johan De Tavernier. The other partners are: Jožef Stefan Institute (Prof. Nada Lavrac), Teesside University (Prof. Marc Cavazza), Leiden University (Prof. Paul van den Broek) and Haute Autorité de Santé (Dr. Gersende Georg).

Results (Demo)

In the frame of the EU projects MUSE and TERENCE we have built an integrated system for semantic role labeling, noun phrase coreference resolution, event classification and temporal relation extraction (see Story Annotation Service). We have developed a novel, scalable framework for jointly learning a model for knowledge extraction from natural language text the validity of which has been proven for extraction of relations, their composing entities and attributes. We have built a novel model for mapping text and its semantic annotations to the knowledge representation used to construct the virtual world. In the framework of the MUSE project we organize the SemEval 2015 task of SpaceEval and we are involved in the organization of the Workshop on Replicability and Reusability in Natural Language Processing: From Data to Software Sharing at the 2015 International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina on July 27, 2015.

Period From 2012-09-01 to 2015-08-31.
Financed by EU FP7-296703 (FET-open call).
Supervised by Marie-Francine Moens
Staff Parisa Kordjamshidi
Steven Bethard
Quynh Do Thi Ngoc
Juan Carlos Gomez
Oleksandr Kolomiyets
Oswaldo Ludwig
Denis Lukovnikov
Contact Marie-Francine Moens

More information can be found on the project website


  1. KOLOMIYETS, Oleksandr & MOENS, Marie-Francine MotionML: Motion Markup Language: Shallow Approach for Annotating Motions in Text. In Proceedings of Corpus Linguistics. 2013
  2. KOLOMIYETS, Oleksandr, KORDJAMSHIDI, Parisa, BETHARD, Steven & MOENS, Maie-Francine SemEval-2013 Task 3: Spatial Role Labeling. In Proceedings of the Second Joint Conference on Lexical and Computational Semantics, Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (pp. 255-262). ACL. 2013
  3. KOLOMIYETS, Oleksandr & MOENS, Moens Marie-Francine KUL: A Data-Driven Approach to Temporal Parsing of Documents. In Proceedings of the Second Joint Conference on Lexical and Computational Semantics, Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (pp. 83-87). ACL. 2013
  4. KOLOMIYETS, O. & MOENS, M.-F. Towards Animated Visualization of Actors and Actions in a Learning Environment. In Proceedings of the 3rd International Workshop on Evidence Based and User centred Technology Enhanced Learning (EbuTEL 2013) (Advances in Intelligent and Soft-Computing) (in press). Springer. 2014
  5. DE MULDER, Wim, DO THI, Ngoc Quynh, VAN DEN BROEK, Paul & MOENS, Marie-Francine Machine Understanding for Interactive Storytelling. In Proceedings of the 8th International Conference on Knowledge, Information and Creativity Support Systems (KICSS'2013) (pp. 73-80). 2013
  6. KORDJAMSHIDI, Parisa & MOENS, Marie-Francine Designing Constructive Machine Learning Models based on Generalized Linear Learning Techniques. In Proceedings of the NIPS Workshop on Constructive Machine Learning. 2013
  7. KORDJAMSHIDI, Parisa & MOENS, Marie-Francine Structured Learning for Mapping Natural Language to Spatial Calculi Models. Abstract published in Proceedings of the NIPS Workshop: Knowledge Extraction from Text (KET). 2013
  8. KORDJAMSHIDI, Parisa & MOENS, Marie-Francine Global Machine Learning for Ontology Population. Journal of Web Semantics, special issue on Semantic Search, 1-29. 2015
  9. NGOC THI DO, Q., BETHARD, S. & MOENS, M.-F. Text Mining for Open Domain Semi-Supervised Semantic Role Labeling. In Proceedings of the First International Workshop on Interactions between Data Mining and Natural Language Processing (pp. 33-48). 2014
  10. DE MULDER, Wim, BETHARD, Steven & MOENS, Marie-Francine A Survey on the Application of Recurrent Neural Networks to Statistical Language Modeling. Computer Speech & Language, 30 (1), 61-98. 2015
  11. CERNY, Martin & MOENS, Marie-Francine aMUSE: Translating Text to Point and Click Games. In Proceedings of the 2015 AI Games Conference. 2015

Back to all projects