When children sit down to write, they are often faced with two challenges: the technical aspects of writing, and their overwhelming imagination, full of ideas. This NSF-funded project suggests that by providing an intermediary medium for children to express their ideas, they can have better support at the point of writing. Towards this end, we developed an interface for children to create their stories through different means of motion tracking, which takes their enactment and turns it into a cartoon that they then use to support their writing. I was responsible for both designing and developing the core of this interface, which has been iterated upon by my colleagues as we investigate different modes of enactment. As student lead, I also led organizational efforts between the two universities hosting this work, and aided in a number of studies which led to publication.

Read more on our lab’s website!

1. Brown, S. A., Chu, S. L., & Loustau, T. (2019, November). Embodying Cognitive Processes in Storytelling Interfaces for Children. In International Conference on Interactive Digital Storytelling (pp. 357-363). Springer, Cham.Zarei, N., Quek, F., Chu, S. L. & Brown, S. A. (2020, November) Towards Design Strategies to Support Children’s Narrative Writing Through Enactment. In International Conference on Interactive Digital Storytelling (publication forthcoming)
2. Zarei, N., Chu, S. L., Quek, F., Rao, N. J., & Brown, S. A. (2020, April). Investigating the Effects of Self-Avatars and Story-Relevant Avatars on Children’s Creative Storytelling. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-11).
3. Zarei, N., Quek, F., Chu, S. L., & Brown, S. A. (2020, November). A Comparison of Children’s Narrative Expressions in Enactment and Writing. In International Conference on Interactive Digital Storytelling (pp. 125-130). Springer, Cham.

Our mental health is highly dependent on how we engage with our personal narratives, and prior work has theorized that we indeed conceptualize life in the form of narrative. Additionally, college students’ high demand for mental healthcare often outpaces supply, which combined with their familiarity with technology, has created a space for self-help digital tools to support their mental health. However, few, if any of these tools help the user engage with their personal narratives the way a therapist does in practice. I propose the use of interactive story authoring tools to support college students in reflecting on their personal narratives. This work provides a uniquely narrative approach to the realm of digital mental health support tools for college students, which seldom support narrative reflection.

So far, I have conducted a grounded theory analysis of how college students reflect on their personal narratives, as well as how counselors of college students engage with student narratives to support their mental health [1]. This was used to generate initial implications for the design of this approach.

1. Brown, S. A., & Chu, S. L. (2021, December). “You Write Your Own Story”: Design Implications for an Interactive Narrative Authoring Tool to Support Reflection for Mental Health in College Students. In International Conference on Interactive Digital Storytelling (pp. 312-321). Springer, Cham.

We embody our ideas in our gestures, and this project aims to visualize those ideas from gestures and the speech that accompanies them. Imagine being able to, as you describe a story or idea, have a supporting visualization portray your descriptions for others to see and collaborate with you on. That is the vision this project aims to achieve, starting with the complex challenge of designing a system to interpret gesture features into meaningful concepts for visualization.

We have examined iconic (image-describing) gestures for physical features that convey consistent meaning by analyzing how people retell stories from the same stimulus (in our case, a short cartoon) [1]. We found that gestures that convey dimension (the length, width, or height of an object) are the most consistent in form, and developed a prototype system that visualizes the size of objects from a person’s gestures [2]. This led to design implications for translating the different mental frames of reference a user wields through their gestures [3]. In the future, we are interested in furthering this research as a means to support designers in their work and have contemplated how multimodality and embodiment can be used to support creative work [4].

Read more on our lab’s website!

1. Brown, S. A., Chu, S. L., Quek, F., Canaday, P., Li, Q., Loustau, T., … & Zhang, L. (2019, November). Towards a Gesture-Based Story Authoring System: Design Implications from Feature Analysis of Iconic Gestures During Storytelling. In International Conference on Interactive Digital Storytelling (pp. 364-373). Springer, Cham. (Nominee for Best Short Paper)
2. Brown, S. A., Chu, S.L. & Rani, N. (2020, September). Externalizing Mental Images by Harnessing Size-Describing Gestures: Design Implications for a Visualization System. In International Conference on Advanced Visual Interfaces. (publication forthcoming)
3. Brown, S. A., Chu, S. L., & Rani, N. (2020, April). Harnessing Gestures to Externalize Discourse Ideas for Common Ground: Design Implications from a Frame of Reference Analysis. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts (pp. 1-8).
4. Brown, S. and Chu, S. L. (2020). In the Flow of Creative Practice: Multimodality and Embodiment for Creativity Support Tools. In Proceedings of Workshop on Where Art Meets Technology: Integrating Tangible and Intelligent Tools in Creative Processes. CHI 2020. Honolulu, HI. ACM.

I acted as student lead on this cross-university project for about a year, helping to investigate ways in which block-based programming interfaces can support the learning of computational thinking and science through their incorporation in maker-based activities. I aided in the design of the initial CASMM (Computation and Science Modeling through Making) application, and produced wireframes for both the student- and instructor-facing interfaces. One of our initial investigations explored how science concepts are represented inside the block-based programming language itself, to further support science learning through these activities [1].

Read more on our lab’s website!

1. Brown, S. A., Lynn Chu, S., & Yin, P. (2021, June). A Survey of Interface Representations in Visual Programming Language Environments for Children’s Physical Computing Kits. In Interaction Design and Children (pp. 268-275).