Kim – Explore, Collect, Reflect

Kim turns the classroom into an adventure: Students explore the universe of the Internet in an augmented reality world. Together with an avatar, they master the hurdles of their everyday lives. Riddles on relevant topics are solved: How does social media work? What happens when I share pictures? How do I protect my mental health? How do I handle passwords correctly?

Each action cube that is distributed by the teacher in the learning space before the game starts, represents a topic. As soon as the students find a cube, they scan it and thus they immerse themselves in the respective learning world, which connects the virtual world with the real world. After a short introduction, the students rotate the action cubes to small, motivating tasks in the subject area, which they solve together in small groups. To do this, they use different building blocks that have different meanings at each station. Each action cube ends with a more complex task that requires prior experience and learning. Once the students complete it, they receive an AR object.

Kim contributes to digital education in schools. To promote collaborative learning in class, teams compare and discuss the AR objects they have collected with the teacher. The focus is on joint reflection rather than on competitive thinking. Students in grades 5 and 6 experience with Kim interactively and exploratively how to make responsible, confident, and (self-)conscious decisions in their everyday activities as we live in an increasingly digitalized world.

Didactics

Exploration and E-I-S Method

Kids are young researchers through experimentation. They discover the gaps between their internal worlds about the world and what happens in the real world. Learning then means changing the internal models to fit more the real world. (Dehaene)

The overall concept of our experience is that the children move through the world and discover their gaps in the topic of digital literacy with today’s prominent tool for the digital world, the smartphone. This makes the learning more authentic.

Our grounding is the E-I-S-method, which stands for “enaktiv”, “ikonisch”, and “symbolisch”. It was Jerome Bruner who said that a good connection between the three types of representation supports successful learning.

“Enaktiv” means to work with the object, like creating a circle drawing the line to “feel” its roundness.

“Ikonisch” means to present facts in a picture. It means to explore the topic through its iconic representation or solve problems in this domain with iconic representations. E. g., in a square, draw a triangle whose vertices lie on the sides of the square and divide them in a ratio of one to two. What proportion does the drawn triangle have in the whole square? If you are using mental geometry to solve the task without pencil and paper, you are already using an iconic representation for the mentally visualized square.

“Symbolisch” means to work on the abstract level. A sign receives the property symbolic, if there are applicable rules for the handling of this sign or the viewer recognizes possible rules connected with the sign. If he can apply them, he has grasped the symbolic content. A non-numeric example is a symbol for the right angle. Or perhaps also a tree diagram.

Pedagogical Inspirations

Friedrich Wilhelm August Fröbel

His special merit is not only to have recognized the importance of early childhood, but to have advanced the realization of these insights by creating a system (with Froebel: „whole“) of songs, occupations, and „gifts of play.“ He founded the „General German Kindergarten“ in Blankenburg on June 28, 1840. The Kindergarten differed from the childcare institutions and infant schools that already existed at the time in its pedagogical concept. This was associated with the expansion of the range of tasks from care to the triad of education, upbringing and care. Today, there are many kindergartens around the world that are named after Froebel and take up and continue his pedagogy. Among other things, the stimulating promotion of children's play by adults and their support in the children's efforts to experience and understand the world is considered essential.

-- FRÖBEL Rahmenkonzept, 2. Auflage, URL: https://www.froebel-gruppe.de/fileadmin/user/Dokumente/Broschueren_Themenhefte/FROEBEL-Rahmenkonzeption_deutsch.pdf

Tangram

Tangram (Chinese 七巧板, Pinyin qī qiǎo bǎn) is an ancient Chinese tile-laying game that probably originated between the 8th and 4th centuries BC. The game consists of seven tiles in simple geometric shapes. The tiles are created by „cutting“ a square into two large triangles, a medium-sized triangle, two small triangles, a square, and a parallelogram. From these tiles can be laid countless shapes, which then show shadow cracked animals, ships, or other figures. Typically, to do this, you need to use all the parts, not superimposing them on each other.

Concept

Kim's concept connects the analog world with the digital one. It wants to connect the physicality of man (Leiblichkeit des Menschen) with the abstract existence of digital objects and their behavior. While we can touch, feel, grab, etc. physical objects in the physical world we cannot do so with digital objects directly. We can explore them or manipulate their attributes only through devices, like a keyword, a joystick, or our smartphone.

One of our central concepts is to help the learners connect the three levels, esp. enactive learning, by the physical manipulation of objects that has consequences in the digital world. We want to make a shift from the only iconic or symbolic level and allow to use all three levels of learning: enaktiv – ikonisch – symbolisch.

The biggest challenge was working out the students' complex problems in the digital world and breaking them down in a way that allowed us to set learning objectives and build a concept around them.

An important factor here was that in our product experience, decisions are actively made through physical interaction. Our goal was to design these actions as playful and intuitively as possible while learning.

01/02/2022

To test the overall interaction experience, we played the game several times.

30/01/2022

After thinking about the flow we decided that we should have the option to play the „How to Play“-video manually. It makes no sense that the video is played every time the game is started. So, we will have a start menu that has two options: start the game and show video „how to play“.

26/01/2022

As every station has somehow the same flow, the Flowchart shows on the left side the overall experience from the perspective of the app:

Starting the „game“ there will be a short introduction video about the app works and what the students should do to go through the experience. After that the app will go into the main game loop which offers three options: i) exploration-path, ii) show basket with the collected items, iii) sum up and end the game.

i) Path of Exploration

This is the main experience. The students explore the room, find cubes and scan them. Every cube offers a different task but will begin with an intro that puts the topic into context. Afterwards the students will do the different tasks and a final tasks. The latter one can be passed using the experiences and learned things from the former tasks. Passing this last task will give them an AR-collectable depending on their decision. After putting the collectible into the basket the app returns to the main loop and the students have the three options of exploration/ show basket/ end the game again.

ii) The Basket

The basket is like an inventory in an RPG. It shows the already collected things, so it could also be empty in the beginning. But there is no more functionality.

iii) The Sum Up

As the game cannot be coded with a fixed number of stations (cubes) and a fixed amount of time (single lesson with 45 min / double lesson with 90 min?) and as teachers need a lot of flexibility throughout a lesson (problems with the devices, fire drills, etc. can shorten the planned time frame --> teachers must able to stop the game at fixed points in the game, but also be able to collect results), the game will offer the sum-up-function in the main loop at any time. The function will give an overview of the collected items. Later on - will not be implemented yet - it should send the results to the teacher in a view that shows the results of all teams.

The right chart shows the activities for the Brute Force Attack task. It shows three lanes that split the activities into „app“, „user“ and „chars“ (the puzzle pieces).

The task has two sub-tasks that follow after each other: first, the students have to break an easy password, the second sub-task lets the students attack a longer password.

The overall idea is that the students learn and experience for themselves that longer passwords need more time/ more effort if someone wants to break them.

The Easy Password

The password has just one character (letter). The students have to choose and try in the demo 1 out of 4 pieces. So, a successful attack needs in the worth case 4 tries. In a final product, they would need to try 9 (or maybe 24) pieces. After choosing a piece the students need to tap a scan button. Otherwise, the ARKit device would scan all the time which could lead to unwanted errors, and to frustration. The scan button will trigger a decode function in the app that will present the character that is encoded with the puzzle piece. If the students want to test this character they can answer the following question with „yes“, otherwise „no“ and they can choose another piece. After showing the result of the try (success, no success) the students can choose, if they want to try another easy password or if they want to move on to the more complex password. The idea is that the students are not stuck on a task and not getting frustrated if they constantly „fail“.

The Harder Password

The overall flow of activities is the same. Now the students have to break a password that consists of 3 characters. This means that the amount of possibilities is much higher. They still have to choose out of 4 possible pieces in the demo (later out of 9 pieces). At the end when the students choose to go on (and not to try another password or another combination of pieces) an internal variable will be set to be a value that marks this task as to be done.

Afterwards the app goes back to the mode of scanning a cube plane for the next task.

19/01/2022 

For the feature freeze, we tried to break down all the decisions made so far. We updated our user journey, with combinations of researched content and interactions. It is significantly more detailed than the last state. During the process, we dealt intensively with the teaching of our learning objectives and scripted dialogs for Kim based on them.

We tried to locate the interactions to get an overview of whether analog and digital interactions were balanced. The goal was always to merge both interactions as well as possible. That's why we also chose the medium AR.

The whole experience of the product is bracketed by the common start and the common reflection with the teacher, both in person in the classroom.

It was also the first time that the analog and digital material features were roughly broken down and we could evaluate what might be achievable in the given time.

15/12/2021

For the interim presentation, we captured our current progress. The interaction here is still very unspecific. We wanted to work with action cards, not AR yet. Here, the magnifying glass is still the main element of our concept. The focus is on exploration in movement.

01/12/2021

We created a storyboard for the interaction with the magnifier to visualize the functions and components. We searched for potential mistakes and get an overall feeling of the product's use.

Digital Material

12/02/2022: App Iteration .07

in Code:

• collection of gem

• presenting reflection phase with button and slide in picture

The video below shows a complete run through the app with scanning (virtual) stones. Sadly, we did not manage to integrate the full AR scences that were created.

10/02/2022: App  Iteration .06

in Code:

• all subtasks can be done

• all videos are included

• inventory shows now correct image depending on solved status

• button back to menu works

open:

• collection of gem after station passwords

• presenting the reflection phase

The video above shows the first part of the full game loop - so far implemented. Because of reflecting on the laptop screen the app had recognition problems in the middle.

After a restart of the game it worked. See video below.

07/02/2022: App Iteration .05

in Code:

• all tasks can be scanned

• something is happening with every scan:

• -- still waiting for AR scences

• -- showing images for not implemented tasks

open:

• AR Scences + integration

• final collection

• presenting the reflection phase

06/02/2022: App Iteration .4

in Code:

• inner flow in tasks completed

• station passwords can be scanned

• inventory seems to work

• videos are played

• problem: after scanning a cube's side video is played after a long while, execution does not wait, pause() does not solve the problem, maybe we can only play one video

open

• solve the problem, if possible

• integrate the AR scences

• show images for the not implemented tasks

• final collection

• how do we present the reflection in the group?

05/02/2022: App Iteration .03

in Code: 

• Game states implemented, 

• Tasks with descriptions and status (solved/ unsolved) implemented

• Tasks cannot be done twice

• Inventory shows different content depending on status of the game (empty and one stone); for demo

• Videos are played depending which image anchor was scanned

• Different image references for anchors depending on the state of the game are being used

open:

AR subtasks, correct videos, correct inventory, images for the tasks not implemented in the demo

-- 2022/02/05

01/02/2022: App Iteration .02

Start menu works (without Reset). Added two Buttons in Exploration Mode: Scan + Inventory. Inventory-button works. Inventory pops up and is closable.

-- 2022/01/24

29/01/2022: First Video for Password-Station - Passwords

The video shows the first idea of how videos could look like. This video should be the introduction to the station passwords. Introduction video give a context and should motivate for the topic.

AR-Testvideo Password-Station #2

Testing how we can scan multiple objects with ARKit and if it works.

24/01/2022: AR-Testvideo Password-Station #1

Start a station in AR.

17/01/2022: Prototype 2.0

In the second draft, we researched….

11/01/2022: Prototype 1.0 

As Augmented Reality is one of the features of the digital material we tested different possibilities to create an AR adventure. We decided to use the tools provided by Apple as ARkit and Reality Composer, tried out its options, and created the first draft.

This is our very first AR sketch. We clarified for ourselves that scanning persons is not an option.

Haptic Material

Inspired by one of the first fast prototypes, the magnifying glass, we designed our formal-aesthetic concept. Hence, at the beginning of the design process, the focus was on the interaction with the magnifying glass. However, as the content research progressed, it became increasingly clear that other objects besides the magnifying glass needed to be added in order to solve the riddles within the AR adventure. It was important that these objects were interchangeable on the AR level in terms of content. This enhances the game- and learning experience by making it more exciting and intuitive. In addition, this eliminated the need to re-explain functions every time an action cube is found. The development of group activity was another crucial factor influencing the design of the system. Thus, there was a shift of focus from the magnifier to the stones that are designed for shared use. We made the conscious decision not to implement the magnifying glass further as felt that these competed aesthetically with the action objects.

The final design of the action cube is an iteration of the design of the 5 cubes as interaction objects. Formally aesthetically, these were very simple and did not convey the dynamic character we were hoping for. In addition, they do not invite free arrangement and did not necessarily serve the iconic component of the EIS method we use. Also, the Reality Composer hat troubles recognizing the anchors as they interfered with each other. However, a cube representing a topic area worked extremely well. The clean lines provided a strong contrast to the stones and can be easily distinguished in terms of form. This contrast is also to emphasize the content level: Cubes signify riddles on a specific topic, asymmetrical stones mean solving riddles.

Once we had our nine-piece shape, the biggest challenge was getting it recognized by the Reality Composer. At first, we hoped that they would work well when scanned, but unfortunately that was not the case. It quickly became clear that we would need to design a pattern and integrate it into the shape.

08/02/2022

Since the two orange tones were not compatible, even after several test prints no good combination was achieved. We decided to swap the colors orange and blue in the pattern so that the blue had the larger percentage.

07/02/2022

There were some misprints in this process as well, but we calculated them well into our schedule.

02/02/2022 

Now we started to 3D print the final components. We used orange PLA filament. The printing process was very time-consuming, with one part printing between 10-20 hours. In the meantime, we were looking for an elegant solution to integrate the pattern on the 3D prints. We had cardboard laminated, as the plan was to embed it in the 3D print. The problem here was that the orange of the filament was not the same as the orange.

01/02/2022

To be able to test the interaction, we printed out the disassembled circle in a smaller version and played extensively. This also allowed us to check if the edges were still too sharp and make changes.

31/01/2022

Since the new shapes seemed too abstract and were also difficult to recognize by the Reality Composer, we decided to use a deconstructed circle. The basic idea is that the circle represents absolute digital sovereignty. At the beginning of the lesson, the teacher lays out the circle, which can then be disassembled and rearranged by the students. Also, the individual parts of the circle invite to be integrated into the digital game to be collected by the pupils.

From the individual parts of the circle, we created a pattern that we could rearrange individually based on the modules.

At this point, we decided on a font and color concept to speak a consistent product language. The IBM Plex Mono was chosen to create a link to the digital world. We chose our colors with the intention of being gender-neutral.

We started the process of 3D printing to find out the optimal curves and the optimal size for smaller hands. In addition, integrating the pattern into the molds was also a challenge.

25/01/2022

Based on the first cube prototypes, we decided to try new shapes. The cubes seemed too simple and the game factor needed to be bigger. Inspired by the tangram game, we tested different shapes that could be used to create appealing pictures. The shapes had to be abstract but not too abstract.

A big challenge was also the AR scannability. The Reality Composer requires asymmetrical, non-repeating patterns with strong contrasts.

We also tried to include physical feedback in the new shapes. To test the prototypes and get a feeling for the material, we lasered gray cardboard, acrylic milky glass, and colored PVC. We worked with hall and vibration sensors and LEDs, to quickly realize that these new features detracted from our concept.

24/01/2022

Our first Hi-fi prototype was a case for an iPhone 8, with which we wanted to show our AR in the exhibition. The two handles are designed to help the students not drop the phone and to be able to interact with the stones using one hand.

Unfortunately, it proved to be very inconvenient during testing. The handles were more of a burden than a help.

20/01/2022

Inspired by the research, we drew the first options for the magnifying glass.

19/ 01/ 2022 

Building blocks in the form of cubes were our first idea. The modular use also formally-aesthetically underlined our didactic concept. The cubes are to be used for answering the tasks as they are arranged freely. Once the solution is found, the cubes snap together using magnets and finally light up.

The feedback is designed to complement the AR experience in the physical space. In the end, we decided not to permanently place the applied signs on the cubes. We found it much better to display them in the AR world. This way we have the advantage of being able to replace them station by station. Formally, we are also no longer dependent on the cube shape, which also presented challenges in the technical implementation.

16/01/2021

At this point, we looked through all possible features. From packaging to the individual parts of the game. Here, mood boards helped us to get a feeling for color and shape. Also, features of existing phone cases were very helpful to develop possible design solutions.

The first new mold for the phone case / magnifying glass was molded from plasticine to intuitively come up with a design that feels good and is ergonomic. However, the result is a shape that is too Gameboy-like. 

15/12/2021

We developed an improved version of the magnifier. It is used to solve different challenges. The needed information is discovered with an AR function that brings movement into the classroom.

We use action cards to take the students on an adventure step by step. This was not a satisfying experience for us. We realized the potential of AR, elevates the interaction.

For our first Lofi prottyping challenge we pursued mainly 4 ideas:

1. The Reminder as a companion reminding you to make good decisions in your everday online interactions.

2. The magnifier as a tool for exploring and looking closely on  different Digital Literacy connected topics.

3. The Box of secrets as game that illustrates how encryption works.

4. Reflect, Share, Social Network as a safe possibility for children to share their questions and impressions with each other and their teachers.

After we presented them we evaluated their opportunities and boundaries for our project and then decided on continuing to work with the magnifier.

Fusion of digital and haptic material

Intuitively, we started the design process with the analog material, as haptic and tactile learning was the highest priority in our concept. As soon as we had designed the formal language here and also determined the color concept, we used this as a building kit for the design of the avatar and the app, which also represents an analogy to the concept of interaction. 

All the elements of the analog components of the game are also found on the digital layer. Thus, individual parts of the disassembled circle were used to create the avatar and to design buttons. The disassembled circle represents the digital sovereignty, which is collected station by the station during the game. We implemented an inventory as components of the circle are collected at each action cube. Depending on how the players decide, the stone has a different pattern. This is to be able to reflect afterwards on how the players decided. The patterns are inspired by the sides of the action cube.

Avatar

Kim is our gender-neutral avatar, hence the name. It was especially important to us that all genders can identify with the character and be included since Kim takes on a role model role. That's why Kim has only a few human features, such as a mouth, eyes, and hints of hands and legs. Kim consists of five stones of the decomposed circle and thus fits into our color and pattern concept. During our test at school, the students tried to recreate Kim and had great fun. 

Dialogs and Sounds

As the whole experience is for primary school students Nintendo's games are our inspiration for the video dialogs. The videos are fully 2D. The text is flying in with a typewriter effect. This makes it easy to change the language later on (if needed) and also links our visual concept to the digital world.

Exhibition

Unfortunately, we didn't manage to implement the AR experience into the app. Both components work very well on their own. That's why we used a combination of disclaimers and videos for the exhibition (see Digital Material). In addition, we worked out only one riddle, the Brute Force Attack, for the presentation of the product. The other sides of the cube, showed short descriptions of the possible tasks.

Expert Interviews

To get some experts insights we interviewed Elizabeth Burrows and Katinka Lotz.

Elizabeth Burrows is the principal of the Phorms School. We wanted to know what problems children have to face online, what skills they should learn to handle these, and which of them are perhaps already taught. The other main topic was how to create an artifact that can be used in a classroom situation.

The second interview was with Katinka Lotz who works as a  Game & Process Designer at dieBaupiloten. She was our expert for creating a challenging, fun, and educating game for children. We asked how to motivate kids to participate in the interactions.

Defining The Challenge

Reviewing our mindmap and some statements of our first interview with Elizabeth Burrows, we collected the topics we felt most important to educate children about. Many could be classified as data protection issues. As this was still a too broad and unclear we narrowed it down to Data Privacy where we saw the most potential and found our interests.

We tried to narrow down our interests by mapping our very own definition of digital literacy.

In our very first meeting we tried to find a common understanding of our new project. What Potential so wee see? How relevant is our topic? Who is our user group?

Presentations