Towards Emergent Play in Mixed Reality
Abstract
This paper presents a system to experience emergent play within a mixed reality environment. Real and virtual objects share a uni- fied representation to allow joint interactions. These objects may optionally contain an internal mental model to act autonomously based on their beliefs about the world. The experience utilizes intu- itive interaction patterns using voice, hand gestures and real object manipulation. We author experience by specifying dependency graphs and behavior models, which are extended to support player interactions. Feedback is provided to ensure the system and player share a common play experience, including awareness of obstacles and potential solutions. An author can mix features from game, story and agent-based experiences. We demonstrate our system through an example adventure game using the Microsoft HoloLens.
Downloads
Download data is not yet available.
References
[1] Ruth Aylett. 1999. Narrative in virtual environments-towards emergent narrative. In Proceedings of the AAAI fall symposium on narrative intelligence. 83–86.
[2] Hrvoje Benko, Ricardo Jota, and Andrew Wilson. 2012. MirageTable: freehand interaction on a projected augmented reality tabletop. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 199–208.
[3] Mark Billinghurst, Hirokazu Kato, and Ivan Poupyrev. 2008. Tangible augmented reality. ACM SIGGRAPH ASIA Courses 7 (2008).
[4] M. Bratman. 1987. Intention, plans, and practical reason. Harvard University Press.
[5] Josh Bycer. 2015. Examining Emergent Gameplay. http://www.gamasutra.com/ blogs/JoshBycer/20150916/253682/Examining_Emergent_Gameplay.php. (2015).
[6] R. G. Dromey. 2003. From requirements to design: formalizing the key steps. In First International Conference on Software Engineering and Formal Methods. 2–11.
https://doi.org/10.1109/SEFM.2003.1236202
[7] Max Froumentin. 2005. The W3C Speech Interface Framework Media Types: appli- cation/voicexml+ xml, application/ssml+ xml, application/srgs, application/srgs+ xml, application/ccxml+ xml, and application/pls+ xml. (2005).
[8] Ron Gilbert. 2014. Puzzle Dependency Charts. http://grumpygamer.com/puzzle_dependency_charts. (2014).
[9] Corinne Hutt. 1966. Exploration and play in children. In Symposia of the Zoological Society of London, Vol. 18. London, 61–81.
[10] PTC Inc. 2014. Vuforia Augmented Reality SDK. (May 2014). https://www. vuforia.com
[11] Warner Bros. Entertainment Inc. 2013. Scribblenauts. (2013). https://www. scribblenauts.com
[12] D Isla. 2005. GDC 2005 Proceeding: Handling Complexity in the Halo 2 AI. (2005). Retrieved August 31, 2017 from https://www.gamasutra.com/view/feature/130663/gdc_2005_proceeding_handling_.php
[13] Marcelo Kallmann and Daniel Thalmann. 1999. Direct 3D Interaction with Smart Objects. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST ’99). ACM, New York, NY, USA, 124–130. https://doi.org/10.1145/323663.323683
[14] Mubbasir Kapadia, Jessica Falk, Fabio Zünd, Marcel Marti, Robert W Sumner, and Markus Gross. 2015. Computer-assisted authoring of interactive narratives. In Proc. of the 19th Symposium on Interactive 3D Graphics and Games. ACM, 85–92.
[15] Mubbasir Kapadia, Seth Frey, Alexander Shoulson, Robert W. Sumner, and Markus Gross. 2016. CANVAS: Computer-assisted Narrative Animation Synthesis. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Anima- tion (SCA ’16). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 199–209. http://dl.acm.org/citation.cfm?id=2982818.2982846
[16] Mubbasir Kapadia, Alexander Shoulson, Cyril Steimer, Samuel Oberholzer, Robert W. Sumner, and Markus Gross. 2016. An Event-centric Approach to Authoring Stories in Crowds. In Proceedings of the 9th International Confer- ence on Motion in Games (MIG ’16). ACM, New York, NY, USA, 15–24. https://doi.org/10.1145/2994258.2994265
[17] Mubbasir Kapadia, Fabio Zünd, Jessica Falk, Marcel Marti, Robert W. Sumner, and Markus Gross. 2015. Evaluating the Authoring Complexity of Interactive Narratives with Interactive Behaviour Trees. In Foundations of Digital Games (FDG’15). 9.
[18] H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto, and K. Tachibana. 2000. Virtual object manipulation on a table-top AR environment. In Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000). 111–119. https://doi.org/10.1109/ISAR.2000.880934
[19] Hui Liang, Junsong Yuan, Daniel Thalmann, and Nadia Magnenat Thalmann. 2015. AR in Hand: Egocentric Palm Pose Tracking and Gesture Recognition for Augmented Reality Applications. In Proceedings of the 23rd ACM International Conference on Multimedia (MM ’15). ACM, New York, NY, USA, 743–744. https://doi.org/10.1145/2733373.2807972
[20] S. Malik, C. McDonald, and G. Roth. 2002. Hand tracking for interactive pattern- based augmented reality. In Proceedings. International Symposium on Mixed and Augmented Reality. 117–126. https://doi.org/10.1109/ISMAR.2002.1115080
[21] Michael Mateas and Andrew Stern. 2003. Façade: An experiment in building a fully-realized interactive drama. In Game developers conference, Vol. 2.
[22] Maxis. 2000. The Sims. [CD-ROM]. (2000).
[23] Mojang. 2011. Minecraft. [CD-ROM]. (2011).
[24] Julie Porteous, Marc Cavazza, and Fred Charles. 2010. Applying Planning to Interactive Storytelling: Narrative Control Using State Constraints. ACM Trans. Intell. Syst. Technol. 1, 2, Article 10 (Dec. 2010), 21 pages. https://doi.org/10.1145/1869397.1869399
[25] Mark O. Riedl and Vadim Bulitko. 2013. Interactive Narrative: An Intelligent Systems Approach. AI Magazine 34, 1 (2013), 67–77.
[26] Alexander Shoulson, Francisco M. Garcia, Matthew Jones, Robert Mead, and Norman I. Badler. 2011. Parameterizing Behavior Trees. In Proceedings of the 4th International Conference on Motion in Games (MIG’11). Springer-Verlag, Berlin, Heidelberg, 144–155. https://doi.org/10.1007/978- 3- 642- 25090- 3_13
[27] John Sweller, Jeroen J. G. van Merrienboer, and Fred G. W. C. Paas. 1998. Cognitive Architecture and Instructional Design. Educational Psychology Review 10, 3 (01 Sep 1998), 251–296. https://doi.org/10.1023/A:1022193728205
[28] Katie Salen Tekinbaş and Eric Zimmerman. 2003. Rules of play: game design fundamentals. MIT Press, Cambridge, Mass.
[29] Gerhard Weiss (Ed.). 2013. Multiagent systems (second edition ed.). The MIT Press, Cambridge, Massachusetts.
[30] Sean White, Levi Lister, and Steven Feiner. 2007. Visual hints for tangible gestures in augmented reality. In Mixed and Augmented Reality, 2007. ISMAR 2007. 6th IEEE and ACM International Symposium on. IEEE, 47–50.
[31] R Michael Young, Mark O Riedl, Mark Branly, Arnav Jhala, RJ Martin, and CJ Saretto. 2004. An architecture for integrating plan-based behavior generation with interactive game environments. Journal of Game Development 1, 1 (2004), 51–70.
[32] Feng Zhou, Henry Been-Lirn Duh, and Mark Billinghurst. 2008. Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE Computer Society, 193–202.
[2] Hrvoje Benko, Ricardo Jota, and Andrew Wilson. 2012. MirageTable: freehand interaction on a projected augmented reality tabletop. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 199–208.
[3] Mark Billinghurst, Hirokazu Kato, and Ivan Poupyrev. 2008. Tangible augmented reality. ACM SIGGRAPH ASIA Courses 7 (2008).
[4] M. Bratman. 1987. Intention, plans, and practical reason. Harvard University Press.
[5] Josh Bycer. 2015. Examining Emergent Gameplay. http://www.gamasutra.com/ blogs/JoshBycer/20150916/253682/Examining_Emergent_Gameplay.php. (2015).
[6] R. G. Dromey. 2003. From requirements to design: formalizing the key steps. In First International Conference on Software Engineering and Formal Methods. 2–11.
https://doi.org/10.1109/SEFM.2003.1236202
[7] Max Froumentin. 2005. The W3C Speech Interface Framework Media Types: appli- cation/voicexml+ xml, application/ssml+ xml, application/srgs, application/srgs+ xml, application/ccxml+ xml, and application/pls+ xml. (2005).
[8] Ron Gilbert. 2014. Puzzle Dependency Charts. http://grumpygamer.com/puzzle_dependency_charts. (2014).
[9] Corinne Hutt. 1966. Exploration and play in children. In Symposia of the Zoological Society of London, Vol. 18. London, 61–81.
[10] PTC Inc. 2014. Vuforia Augmented Reality SDK. (May 2014). https://www. vuforia.com
[11] Warner Bros. Entertainment Inc. 2013. Scribblenauts. (2013). https://www. scribblenauts.com
[12] D Isla. 2005. GDC 2005 Proceeding: Handling Complexity in the Halo 2 AI. (2005). Retrieved August 31, 2017 from https://www.gamasutra.com/view/feature/130663/gdc_2005_proceeding_handling_.php
[13] Marcelo Kallmann and Daniel Thalmann. 1999. Direct 3D Interaction with Smart Objects. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST ’99). ACM, New York, NY, USA, 124–130. https://doi.org/10.1145/323663.323683
[14] Mubbasir Kapadia, Jessica Falk, Fabio Zünd, Marcel Marti, Robert W Sumner, and Markus Gross. 2015. Computer-assisted authoring of interactive narratives. In Proc. of the 19th Symposium on Interactive 3D Graphics and Games. ACM, 85–92.
[15] Mubbasir Kapadia, Seth Frey, Alexander Shoulson, Robert W. Sumner, and Markus Gross. 2016. CANVAS: Computer-assisted Narrative Animation Synthesis. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Anima- tion (SCA ’16). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 199–209. http://dl.acm.org/citation.cfm?id=2982818.2982846
[16] Mubbasir Kapadia, Alexander Shoulson, Cyril Steimer, Samuel Oberholzer, Robert W. Sumner, and Markus Gross. 2016. An Event-centric Approach to Authoring Stories in Crowds. In Proceedings of the 9th International Confer- ence on Motion in Games (MIG ’16). ACM, New York, NY, USA, 15–24. https://doi.org/10.1145/2994258.2994265
[17] Mubbasir Kapadia, Fabio Zünd, Jessica Falk, Marcel Marti, Robert W. Sumner, and Markus Gross. 2015. Evaluating the Authoring Complexity of Interactive Narratives with Interactive Behaviour Trees. In Foundations of Digital Games (FDG’15). 9.
[18] H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto, and K. Tachibana. 2000. Virtual object manipulation on a table-top AR environment. In Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000). 111–119. https://doi.org/10.1109/ISAR.2000.880934
[19] Hui Liang, Junsong Yuan, Daniel Thalmann, and Nadia Magnenat Thalmann. 2015. AR in Hand: Egocentric Palm Pose Tracking and Gesture Recognition for Augmented Reality Applications. In Proceedings of the 23rd ACM International Conference on Multimedia (MM ’15). ACM, New York, NY, USA, 743–744. https://doi.org/10.1145/2733373.2807972
[20] S. Malik, C. McDonald, and G. Roth. 2002. Hand tracking for interactive pattern- based augmented reality. In Proceedings. International Symposium on Mixed and Augmented Reality. 117–126. https://doi.org/10.1109/ISMAR.2002.1115080
[21] Michael Mateas and Andrew Stern. 2003. Façade: An experiment in building a fully-realized interactive drama. In Game developers conference, Vol. 2.
[22] Maxis. 2000. The Sims. [CD-ROM]. (2000).
[23] Mojang. 2011. Minecraft. [CD-ROM]. (2011).
[24] Julie Porteous, Marc Cavazza, and Fred Charles. 2010. Applying Planning to Interactive Storytelling: Narrative Control Using State Constraints. ACM Trans. Intell. Syst. Technol. 1, 2, Article 10 (Dec. 2010), 21 pages. https://doi.org/10.1145/1869397.1869399
[25] Mark O. Riedl and Vadim Bulitko. 2013. Interactive Narrative: An Intelligent Systems Approach. AI Magazine 34, 1 (2013), 67–77.
[26] Alexander Shoulson, Francisco M. Garcia, Matthew Jones, Robert Mead, and Norman I. Badler. 2011. Parameterizing Behavior Trees. In Proceedings of the 4th International Conference on Motion in Games (MIG’11). Springer-Verlag, Berlin, Heidelberg, 144–155. https://doi.org/10.1007/978- 3- 642- 25090- 3_13
[27] John Sweller, Jeroen J. G. van Merrienboer, and Fred G. W. C. Paas. 1998. Cognitive Architecture and Instructional Design. Educational Psychology Review 10, 3 (01 Sep 1998), 251–296. https://doi.org/10.1023/A:1022193728205
[28] Katie Salen Tekinbaş and Eric Zimmerman. 2003. Rules of play: game design fundamentals. MIT Press, Cambridge, Mass.
[29] Gerhard Weiss (Ed.). 2013. Multiagent systems (second edition ed.). The MIT Press, Cambridge, Massachusetts.
[30] Sean White, Levi Lister, and Steven Feiner. 2007. Visual hints for tangible gestures in augmented reality. In Mixed and Augmented Reality, 2007. ISMAR 2007. 6th IEEE and ACM International Symposium on. IEEE, 47–50.
[31] R Michael Young, Mark O Riedl, Mark Branly, Arnav Jhala, RJ Martin, and CJ Saretto. 2004. An architecture for integrating plan-based behavior generation with interactive game environments. Journal of Game Development 1, 1 (2004), 51–70.
[32] Feng Zhou, Henry Been-Lirn Duh, and Mark Billinghurst. 2008. Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE Computer Society, 193–202.
Published
2018-01-11
How to Cite
MISTELI, Patrick et al.
Towards Emergent Play in Mixed Reality.
International SERIES on Information Systems and Management in Creative eMedia (CreMedia), [S.l.], n. 2017/2, p. 51-56, jan. 2018.
ISSN 2341-5576.
Available at: <https://www.ambientmediaassociation.org/Journal/index.php/series/article/view/281>. Date accessed: 06 june 2023.
(c) International Ambient Media Association (iAMEA) [Copyright and License Transfer Agreement]
