Generative Theories of Interaction Generative Theories of Interaction

5.1.1 Human Tool Use in Psychology. In his theory of affordances, Gibson [66] mentions tools only briefly but makes a key observation: “When in use, a tool is a sort of extension of the hand, almost an attachment to it or a part of the user's own body, and thus is no longer a part of the environment of the user” [66p. 40]. For Gibson, “The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill” [66p. 127, emphasis by the author]. Holding a tool thus redefines the affordances of the environment, since it changes the person's capabilities. Recent work supports this theory that a tool becomes an extension of the body schema, as if it were an integral part of the body [89]. This property provides a psychologically and cognitively sound basis for creating extensible systems, where new digital tools reveal new affordances.

Cognitive scientists have also studied human tool use to understand the cognitive processes at play when selecting and using tools. Osiurak et al. [126] contrast two competing views of tool use. The computational approach states that perceived properties of the tool are translated into physical actions for use, whereas the ecological approach states that the affordances of the tool (what can be done with it) are perceived directly. More recently, Osiurak [125] reports on experiments with patients suffering from several forms of apraxia (action disorganization syndrome) that suggest that tool use involves “technical reasoning”, a specific type of reasoning that complements the direct perception of what the tool can do.

Technical reasoning is based on abstract technical laws, which, unlike skills acquired through procedural learning, do not require constant reinforcement. For example, people do not forget how to ride a bike, even after many years without practice, but commonly forget simple procedures, such as changing the time on a digital clock. Technical reasoning relies on identifying the technical properties of objects and their appropriate combination to perform a task and reach the desired goal. It explains unanticipated uses of objects as tools, such as using a knife to undo a screw when a screwdriver is not available. Both procedural learning and technical reasoning are clearly useful in everyday life. However, given that current computer systems are heavily based on procedural learning, we focus on the ability of technical reasoning to enable appropriation.

This difference between procedural learning and technical reasoning can be a powerful basis for creating (digital) tools that are more “natural”, in the sense that they do not need constant re-learning. An interesting and open research question is to identify which types of digital tools, if any, elicit technical reasoning vs. procedural learning. A sensible hypothesis is that the digital tools found in tool palettes are more likely to involve technical reasoning while the use of menus and dialog boxes seem more likely to involve procedural learning.

5.1.2 Human Tool Use in HCI. As mentioned in the previous section, the tool perspective dates back to the 1980s with the works of, e.g., Weizenbaum [152], Winograd and Flores [154], and Ehn [57]. Ehn and Kyng [58] and Göranzon [67] see human craftmanship and skill as essential qualities in developing the tool perspective, pointing out that human tool use does not happen in isolation but is situated in human practices and materials. Surprisingly however, the role of tools in human-artifact interaction is absent from seminal works in HCI such as Card, Moran and Newell [44], Suchman [145], or Norman [123].

In their taxonomy of the concept of interaction, Oulasvirta and Hornbaek [77] include tool use as one of seven core conceptions of interaction. Indeed, several authors have focused on the mediating role of tools (or instruments) in the relation between human users and their objects of interest, addressing issues such as in-directness. Bødker [22] emphasizes how tool use is embedded in human practice, with individual and cultural/collaborative sides that mirror each other. Rabardel [130, 131] analyzes the genesis of instruments as either directed toward the subject (“instrumentation”) or toward the artifact (“instrumentalization”). Beaudouin-Lafon [13, 14] argues for the design of interaction models that can be used to describe, evaluate, and generate interaction styles and techniques, and introduces Instrumental Interaction, a model based on explicit digital tools.

In summary, while human tool use has not been a mainstream topic of research in Psychology nor HCI, it brings an enlightening perspective on the role of mediating artifacts in our interaction with both the physical and digital worlds.

5.3.1 Reification. The key principle of Instrumental Interaction is Reification: An instrument reifies an abstract command or concept, i.e., it embodies it into an object that the user can interact with to manipulate a target object. For example, a scrollbar reifies the action of navigating a document by mediating the user's actions: The user acts upon the scrollbar by moving the thumb, which acts upon the document, changing which part of the document is displayed in the window.

Reification is the fundamental generative principle for creating new instruments: When confronted with the problem of providing a given functionality, designers can imagine an instrument that reifies it, in order to make it concrete for the user. This contrasts with the traditional approach where a new functionality is typically bound to a new menu item that acts as a “magic word” to invoke the functionality, typically forcing procedural learning of which command does what and when it is available. By contrast, a “properly designed” instrument leverages technical reasoning: The tool should have a technical effect that makes it more understandable and memorable than the mere arbitrary mapping between a menu entry and a command.

Of course, not every reification of a command into an instrument will be successful. Applying the principle still requires skill, creativity, luck, and the patience to iterate. Nevertheless, as we will show in the next sections, Reification has been instrumental (so to speak) in helping us design interfaces that are both more powerful yet simpler to learn and operate than their non-instrumental counterparts.

5.3.2 Polymorphism. Reification generates new instruments, calling for an additional principle to control the potential explosion of the number of instruments in the environment. The second generative principle, Polymorphism, helps create instruments that can be applied to objects of different types.

For example, a coloring instrument can work with different types of objects, and even different parts of objects. Many drawing tools have a different command to change the border color vs. the fill color of an object, and the color of text vs. the color of text highlighting. A single coloring instrument can replace these commands, by acting differently depending upon whether it is applied to the border or interior of a graphical object, to text, or to a highlighting brush.

As in programming languages, Polymorphism can be used for better or for worse. To be successful, a polymorphic instrument must have some internal consistency that makes its behavior with different types of objects predictable. Here too, we expect to leverage technical reasoning: “I want to change the color of a folder in the file explorer, I should be able to use the same color tool as in my text editor”, rather than procedural learning: “I have to remember that to change the background color of the document I need to go to the document settings”.

5.3.3 Reuse. The third generative principle of Instrumental Interaction is Reuse: the ability for users to reuse previous actions (Input Reuse) or the results of these actions (Output Reuse). This principle is based on the observation that computer tasks are often repetitive and users prefer starting from existing content and modifying it rather than starting from scratch. Historically, many physical tools have been created to facilitate reuse and make repetitive tasks more efficient. For example, pins were replaced by staples to bind documents, and then paper clips were introduced to facilitate reuse and to easily unbind documents without damage [128].

Output Reuse is already present in many interfaces through the “copy-paste” and “duplicate” commands. However, designing with Output Reuse in mind can help extend the power of these commands, for example with the ability to copy the tools themselves in order to create variants, e.g., brushes with different patterns. Similarly, Input Reuse is often present in the form of a “redo” command, and sometimes with the ability to access and selectively re-execute past actions. Combining Input Reuse with Reification leads to creating instruments that embody a sequence of past actions, empowering users to create their own tools through use.

5.3.4 Power in Combination. Beyond their individual generative power, the three principles are even more powerful in combination: Reification creates more objects that can be targeted by Reuse, Polymorphism expands the types of objects that can be targeted by a given instrument, including groups of heterogeneous objects, and Reuse facilitates the creation of new objects and instruments by end users.

We now illustrate how these principles directly guided the design of novel interaction techniques in two recent research projects. We describe the analytical, critical, and generative power of Instrumental Interaction with two examples drawn from our own research: StickyLines [51], for graphical editing, and Textlets [71], for text editing.

5.4.1 Analytical and Critical Power. Our studies of users of CPNs showed that they use graphical alignment extensively to create diagrams that are both visually pleasing and semantically meaningful. The tool they used, Design/CPN, featured alignment commands similar to most graphical tools (Figure 6): a set of menu items or buttons in a tool palette to align the selected objects horizontally or vertically, along their centers or one of their sides.

The fact that these commands are not reified into a manipulable object means that users spend up to 25% of their time aligning and re-aligning objects, or selecting a group of aligned objects before moving them together so as to keep them aligned. While Design/CPN, like most graphical tools, let them group a set of objects to manipulate them as a single object, they seldom used this feature because many objects are part of multiple, overlapping alignments. We observed similar issues with professional designers using tools such as Adobe Illustrator. Moreover, they also often needed to adjust the alignment or distribution of shapes to be visually correct: When aligning logos, for example, the visual center is rarely the geometric center of the bounding box. Similarly, when distributing objects, the visual size is often not the bounding box of the object. Graphic designers therefore routinely adjust alignment and spacing manually. Unfortunately, the system does not keep track of these adjustments, which users need to perform again every time they re-align or re-distribute the objects.

The alignment commands of tools such as Design/CPN and Adobe Illustrator are also insufficiently polymorphic. While they can be used to align different types of shapes, they are limited to horizontal and vertical alignments. Some Petri Nets and graphical designs lend themselves to, e.g., circular layouts, but users can only create those coarsely, by hand. Most tools also do not support aligning the control points of polylines or curves, but only the entire object. According to the principle of Polymorphism, the “align” and “distribute” commands should be applicable to any object with a position and size and to any shape, not just horizontal and vertical lines.

Finally, the align and distribute commands result in poor Reuse. For example, it is not possible to extract the alignment and/or distribution properties of a set of objects to apply them to another set.

5.4.2 Constructive Power. StickyLines is the result of applying Reification to the align command: A magnetic guideline is an object that can be created, moved, and deleted like any other object in the diagram.¹ The user can attach an object to the guideline by dragging the object and snapping it to the guideline, and can detach an object by dragging it away from the guideline (Figure 7). Crucially, the objects attached to a guideline move with it when the guideline itself is moved, maintaining the alignment.

StickyLines also supports the even distribution of objects along a guideline: When objects are added to or removed from it, or when the endpoints of the guideline are moved, the objects move along the StickyLine so that they stay evenly distributed. Finally, StickyLines reifies the concept of adjusting the alignment of an object: When using the arrow keys to move an object attached to a StickyLine, a purple line linking the object to the line, called a “tweak”, shows the visual offset applied to the object (Figure 8, left). The object is still attached to the line, and moves with it. The tweak itself is an object (Reification) that can be selected, copied and pasted (Reuse), or deleted. The same principle applies to the bounding box of an object when distributing objects (Polymorphism): The box can be made visible and directly modified, changing the system's notion of the object's bounding box and therefore altering the distribution accordingly (Figure 8, right).

Another use of Reification is the ability to change the distribution function that maps the rank of objects on a guideline into their position. StickyLines can show this mapping as a curve, which can be edited to create a layout where objects are progressively closer or further away from each other (Figure 9).

StickyLines support Polymorphism by allowing any object to be attached to them, including geometric shapes, text, and the corners of multi-segment lines and arrows. In addition, any shape can be turned into a guideline, including shapes that are part of the diagram itself (Figure 7, right). We also found it useful to let guidelines be the target of other commands, such as setting the fill or border color of shapes. Rather than applying these attributes to the guideline itself, it applies them to each shape attached to the guideline, as if they were a group.

Since guidelines and tweaks are objects that can be directly manipulable by the users, they naturally support Reuse, such as copy-paste and duplicate. Copying a guideline copies the objects attached to it, but it is also possible to copy just the guideline, with its attribute. This is particularly useful when the guideline is used for distribution, e.g., to copy the distribution function.

We found that StickyLines blur the distinction between tools and objects of interest: A StickyLine is used as a tool when the user moves it to move the aligned objects, or when it “grabs” objects as the user moves and releases the guideline over them; but it is also the object of interest when the user snaps an object onto it, or when a new StickyLine is created automatically when an object alignment is detected. A StickyLine is therefore not just an object “ready-at-hand” that works only when explicitly manipulated, it is an object “present-at-hand” that affects the behavior of other objects as they come near it.

Our performance evaluation of StickyLines showed that it was up to 40% faster than traditional align and distribute commands [51]. This is due mainly to the fact that StickyLines keep objects aligned, and that users can manipulate aligned objects as groups even when individual objects are attached to multiple StickyLines.

We also observed that graphic designers used StickyLines in creative and unexpected ways. While tweaks were intended as small adjustments to an object's position, we observed some users creating giant tweaks so that they could move a set of non-aligned objects together by moving the guideline, reifying the notion of a group. A user also asked if the concept of distribution could be generalized to 2D, to automatically lay out (and tweak) objects within an area instead of along a line. We also observed that Petri Net designers started to create guidelines before creating the diagram itself, as a way to plan the overall layout. This shows how the generative principles not only help generate creative and powerful solutions, but also how they inspire users to take them even further. It also provides evidence that users were employing technical reasoning to appropriate the properties of the guidelines in unexpected ways.

5.5.1 Analytical and Critical Power. We found that while all participants used Microsoft Word extensively, they did not use built-in features that could address their needs, even if they knew these features. For example, many participants do not use global search and replace because they cannot check its effects easily, and they hesitate to use incremental search and replace through fear of missing a change. Some prefer numbering items in a contract by hand, even if it also means renumbering them and their references by hand every time an article is added, moved, and removed. They rarely use styles, as these tend to pollute documents as they circulate among users, and they often manage versions and variants by hand, e.g., by using colored text rather than Word's review mode.

Our interviews showed that users prefer these manual operations because it gives them a sense of control. The fact that the effect of Microsoft Word commands such as search-and-replace are not directly visible and editable, i.e., reified into objects that they can inspect and revise, makes them wary that “it did not do the right thing”, with possible dire legal consequences. They find that checking that the command worked is more tedious than performing the changes manually.

The lack of Polymorphism for Microsoft Word features that are seemingly identical, such as numbering list items, section titles, references, and footnotes leads to cognitive overload: Users must rely on procedural learning rather than technical reasoning as each command works differently and has different settings.

Finally, the lack of Reification limits Reuse. For example, styles are not first-class objects that can be directly copied and pasted, e.g., across documents. The participants in our study sometimes create their own templates, and use copy-paste extensively, but fight with the side effects such as “style pollution”.

5.5.2 Constructive Power. In order to address some of these issues, we applied Reification to the most common operation in text editing after typing text: selection. In current text editors, selection is transient: The user selects some text, invokes a command that affects the text, e.g., turns it bold, and the selection is lost as soon as the user clicks elsewhere. By reifying text selection into a first-class object, called a textlet [71], we turn the transient selection into a persistent object that users can return to at any time. Textlets are highlighted in the document and displayed in a side panel. Clicking a textlet reselects the associated text.

The power of Textlets derives from the ability to assign behaviors to them. For example, the word count behavior adds a counter that displays the number of words in the textlet in real time, as the content of the textlet is edited (Figure 10). Similarly, a variant behavior lets users record multiple alternatives for the content of the textlet, and pick the one being displayed. A more sophisticated behavior is search-and-replace (Figure 11): A search textlet dynamically creates textlets for each occurrence of the search text and lets users replace occurrences one by one, in the order they want, or all at once. Replaced occurrences are displayed in a different color, and the user can undo and redo any change. Another example is numbering (Figure 12), which creates automatically numbered textlets according to a pattern defined by the user; each such textlet can then manage references to it as separate textlets.

Behaviors represent a different form of Reification than that of StickyLines or the basic textlet introduced above. Rather than turning a command into an object, they turn a command into a dependency that is re-evaluated and updated when the content of the surrounding document changes, as in a spreadsheet. Attaching the command to the textlet makes it more concrete, observable, and manipulable. For example, a word-counting behavior turns a textlet into a real-time word-counting instrument.

Behaviors make textlets Polymorphic: Different behaviors can be attached to a textlet, possibly in combination. For example, combining the variant and word counting behaviors lets users try alternatives of, e.g., an abstract limited to 150 words, and see the word count in real time as they edit the abstract and switch between the alternatives. At a more abstract level, the notion of textlet as Reification of selection is polymorphic in that it can be applied to other types of content than text, e.g., the graphical shapes of a drawing editor or the file icons of a file manager.

Textlets support Reuse at a very basic level: By remembering the transient state of a selection, the user can instantly re-select the text by clicking the textlet (Input Reuse). At a higher level, textlets and their attached behaviors can be copied and pasted, and attached to a different selection (Output Reuse). Turning transient selections into persistent objects also makes it possible to, e.g., manage multiple simultaneous searches whereas the traditional approach only supports one search string at a time. This makes it easy to reuse a previous search.

The participants in our study found Textlets particularly easy to use and efficient for their tasks, giving them a sense of control and appropriate feedback on their actions [71]. As with StickyLines, we observed appropriation of Textlets by users. For example, some participants created search textlets for forbidden words, to track words that they had decided should not be used, e.g., in a patent. Typing such a word would immediately highlight it, warning the user in a non-obtrusive way. We also brainstormed many other behaviors that could be attached to textlets, such as indexing words for a glossary, creating spreadsheet-like formulas for calculated text, hiding, summarizing or translating text, and so on.

6.1.1 Co-Adaptation in Evolutionary Biology. The biological phenomenon of Co-adaptation specifies how species are both affected by and affect the environments in which they live. The more familiar term co-evolution describes how the actions of certain species reciprocally affect each others’ behavior over millennia. Each species in a co-evolutionary relationship exerts selective pressures on the other, thus influencing how the other evolves. For example, anaerobic bacteria released oxygen into the atmosphere, which created a new, oxygenated habitat where aerobic bacteria evolved that can breathe oxygen [108]. More relevant to HCI are the changes that occur within an organism's lifetime. In The Origin of Species [52], Charles Darwin defined co-adaptation as “the usually beneficial relationship between different species”, where the “skills, habits, actions, or form of one species matches and uses the skills, habits, actions, or form of another species”. Darwin explains that co-adaptation refers to the interactions among individual organisms, whereas co-evolution refers to their evolutionary history.

The key idea here is that organisms do not simply adapt to their current environment and the other organisms within it. Rather, organisms also actively participate in the evolutionary process: Their actions adapt and modify the environment, such that they and other organisms within that environment evolve and change together. Co-adaptation emphasizes this ongoing, potentially asymmetrical process of mutual influence between organisms and the environment, where “survival of the fittest” is not simply a matter of an organism adapting to a changing environment, but also of it physically changing that environment to ensure its survival.

6.1.2 Co-Adaptation in HCI. The concept of Human–Computer Partnership builds upon Mackay's [109] observations of Co-adaptation with interactive systems, where users do not passively react to a technology's features, but also intentionally appropriate it to meet their needs. Some systems are designed to reduce this potential for customization, such as the purposefully limited options available on automated teller machines. However, other systems encourage exploration and innovation, such as the underlying structure of spreadsheets that supports far more innovative activities than simply adding up columns of numbers [119]. Co-adaptation takes the user's perspective, highlighting the dual relationship users establish with any interactive technology, where they discover how it works but also modify it for new purposes. When we take the system's perspective, especially with an intelligent system, we see a similar, although not identical, relationship, where some systems adapt to the user, e.g., smartphones that suggest words based on the user's earlier input [156, 157], while others adapt the users behavior, e.g., to teach users new skills. In summary, Co-adaptation offers a lens for examining the dual role of the user's interaction with technology, which shifts the focus from what Bill Buxton calls “getting the design right” to “getting the right design” [43].

6.2.1 Reciprocal Co-Adaptation. The above examples involve interaction with “non-intelligent” objects that have no agency of their own. An artist can hone the edge of a pencil to achieve a particular effect, but the pencil will not suggest which edge will produce the best result. By contrast, intelligent systems can act independently, with their own agency. This suggests that such systems should be able to:

1. adapt to users, by learning from their behavior; and
   
2. adapt users, by changing their behavior.
   

Here, we see the same basic co-adaptive relationship as before, although the specific natures of human and system intelligence are very different. The third property asks the system to capture patterns of the user's behavior over time, so it can adjust its responses to meet the specific needs of that individual. However, this also implies that users should be able to discover what the system has inferred from their behavior, and change it if it is unwanted or incorrect. The fourth property asks the system to modify the user's behavior, ideally with the user's consent. This can have major benefits for the user, as when educational software [8] improves the user's skills. More controversially, some systems, e.g., persuasive technologies [61, 113], modify users’ behavior without their explicit knowledge or permission: The challenge remains how such systems can offer users “informed consent” [40].

To be effective, a Human–Computer Partnership with an intelligent system requires careful thought about how users and the system share agency, especially how to allocate control. Users reason about tasks differently than computers, and may or may not be aware of the system's agency, which makes intelligent tools significantly more complex to design and understand. We use the term Reciprocal Co-adaptation to describe how these four relationships cross-influence each other to form a Human–Computer Partnership.

6.3.1 Discoverability. When first approaching a new system, users usually have an idea of how it works and what they seek to accomplish with it. Even so, they need help figuring out where expected commands are located, and how to invoke them, as well as finding out which new commands or features are available. They need feedback about their previous actions to understand how the system interpreted them, and also feedforward, to discover how they can achieve particular effects. When we consider Reciprocal Co-adaptation, the system should also learn from the user's behavior, and use that information to inform the user or achieve specified goals.

6.3.2 Appropriability. When people try accomplish a particular task, they look for appropriate tools in the system. If they cannot find them, they should be able to modify or re-purpose commands or features to meet their needs. Ideally, as with physical systems, users should be able to re-purpose various properties of the interface to create new tools. This may be as simple as remapping gestures to new commands, or taking advantage of certain features to accomplish new tasks. When we consider Reciprocal Co-adaptation, the system should explicitly influence the user's behavior, either through teaching new skills or subtly modifying their actions.

6.3.3 Expressivity. Each person generates individual variation every time they interact with technology. For example, physical handwriting reflects both unconscious variation, identifiable as an individual's handwriting style, and conscious choices, such as carefully lettering a love letter. Systems should be able to capture this input variation and transform it into correspondingly rich, expressive output, under the user's control. When we consider Reciprocal Co-adaptation, the system should reflect individual aspects of the user's behavior, such as generative design tools that create dynamic patterns guided by the user's input.

These three principles address different aspects of Co-adaptation in order to better support Human–Computer Partnerships, and can be summarized as follows:

1. Discoverability: reveal how the system interprets the user's recent behavior (feedback) and which commands are now possible (feedforward);
   
2. Appropriability: modify the system's behavior by customizing its characteristics for new purposes; and
   
3. Expressivity: create rich, personalized output generated from individual user-controlled input variation.
   

6.3.4 Power in Combination. Each principle addresses users’ specific needs as they interact with the system over time. Although effective individually, as with Instrumental Interaction they achieve greater power when combined. Discoverable systems encourage exploration, which can lead to new insights and ideas for appropriation. Appropriable systems offer user-modifiable properties that users can manipulate to create more expressive commands or mappings between user input and rich output. Expressive systems reveal details of how the system interprets the user's behavior, which can lead to discovery of new features (Discoverability), that can, in turn, be modified for new purposes (Appropriability).

6.4.1 Analytical and Critical Power. Gesture-based interaction allows users to express a huge vocabulary in a limited space, and advances in machine learning make it possible to reliably recognize a wide variety of gestures. Because the “nervous system has the capacity to form multiple long-term (> 24 hours) motor memories” [97], experts rarely forget these learned gestures, resulting in highly efficient interaction.

Given the speed and expressive power of gesture-based interaction, why is it so rare? One major difficulty lies in the transition from novice to expert, which requires first discovering, then mastering each gesture. Users struggle to remember mappings between gestures and commands, and rely on separate “cheat sheets” with pairs of command names and gestures [12]. Most gesture-based commands lack in-context Discoverability —users usually only receive binary feedback as to whether or not a particular gesture was recognized, although a few also provide a confidence level or suggest possible alternatives.

Gesture typing addresses some of the problem by drawing gestures over a soft keyboard. Here, users trace from letter to letter of their desired word, which is “visually guided, closed-loop, and relatively slow [but] easy because it does not require any prior memory” [157]. However, over time, experts develop memory-driven gesturing which is “recall-driven, open-loop, efficient, and fast” [157]. However, despite being up to 40% faster than tap-typing [99, 156], adoption rates are low. Although gesture-typing is easier than learning free-form gestures, it still takes time, usually about 40 hours: “If you [make the swipe gesture] enough times, that gesture sticks in your motor memory. It becomes automatic and involuntary, and by repeatedly doing that single motion for a single word you become a shorthand expert” [98].

Some systems with gesture-based commands offer limited Appropriability, usually through a dialog box that lets users change the mapping between a gesture and a particular command. But most systems simply offer users the choice of turning default gestures on or off. For example, Apple iPad users can traverse a series of menus to toggle whether or not using four or five fingers will “pinch to the home screen”, but cannot specify that this command should work with only three fingers. Of course, users can use technical reasoning to re-purpose gestures, beyond their “official” purpose. For example, we observed graphic designers draw free-form lines that were not intended as part of the final drawing, but rather served as scaffolding for positioning other graphical elements. Systems such as Knotty Gestures [149], SketchSliders [148], and Knotation [50] build on this idea by allowing users to draw free-form lines that become interactive controllers of music, video, or other data.

Many creativity support tools explicitly encourage Expressivity, by capturing user-based variation and reflecting it in the output. For example, with a pressure-sensitive pen and the right software, users can control the thickness or the fuzziness of the line as it is drawn. Similarly, systems designed to support musical expression can dynamically change the sound based on the user's movements [21]. Users should be able to choose between creating uniform “professional” output, or revealing the nuanced details of the gesture in the form of personalized results. Including Expressivity as a generative principle allows us to capture the individual richness of the user's input, and treat it as a resource for creating varied output, under the user's control.

6.4.2 Constructive Power. CommandBoard [2] expands gesture typing on a soft keyboard, taking advantage of earlier techniques that support Discoverability, Appropriability, and Expressivity to provide users with the power of a command-line interface.

Discoverability: When novice users need to perform a gesture-based command, they need to know both which commands are currently available, and how to perform the current gesture. We designed Octopocus [12] as a dynamic guide that helps novice users discover these mappings, without interrupting expert users. We borrowed the Marking Menu [100] technique of only reacting when the user indicates uncertainty by pausing the gesture. At that point, the system displays feedforward (Figure 13, left) to show which currently available gestures will produce which commands, as well as feedback to show the initial part of the user's gesture. Pausing at the start of the gesture displays all possible commands (Figure 13, center); whereas pausing partway through displays only the remaining possible gesture-commands (Figure 13, right). The variation in the width of each gesture reveals the likelihood that the system will successfully recognize the gesture. Displaying too many gestures creates visual overload, since users can only differentiate about 16 gestures at a time. However, Octopocus accounts for the current context by displaying only the most relevant gesture-commands, which reduces visual clutter.

Appropriability: Users sometimes want to create their own personal gesture-commands, ones they find easy to remember. However, the system must be able to reliably recognize each user-generated gesture command, and distinguish it from others in the gesture vocabulary. We created Fieldward [114] (Figure 14), a dynamic guide that provides both feedback and feedforward as the user draws a proposed gesture. The Fieldward heatmap changes color: If the user stops in a red area, the gesture is not unique and the user should either extend the gesture by moving toward a blue area or try again. If the area is purple, the gesture is ambiguous and cannot be recognized reliably. However, if the user stops in a blue area, the current gesture is both available and easy for the system to recognize. This strategy helps users discover the acceptability of each newly proposed gesture, while revealing if it is similar to one that already exists. Users can also discover features of the recognition algorithm, such as that upward and downward strokes are treated as distinct gestures, despite leaving the same trace. Our studies showed that some users appropriated the system. For example, instead of creating unique gestures for compound commands, such as “Message Mom” and “Skype Ann”, one participant developed his own personal syntax with a set of gestures for communication apps (phone, text message, video chat), and another set for key contacts (Mom, Dad, girlfriend) which resulted in an easy-to-remember grammar.

Expressivity: Although users can pre-select particular fonts, gesture-typers cannot vary them as they write, as they would with physical handwriting. Sophisticated machine learning algorithms capture the details of each gesture, but focus only on identifying the “correct” word, and discard individual variation as noise. We created Expressive Keyboard [3] (Figure 15, left) to transform this variation into user-controlled, personalized output. We first collected samples of how people gesture-type under different physical conditions—walking, sitting, standing—for different purposes, and extracted three common features—curviness, speed variation, and size. We then mapped these to red, green, and blue, the three components of RGB colors. This lets users generate the full spectrum of RGB colors by simply modifying how they draw from letter to letter, without losing the benefit of word recognition.

The Expressive Keyboard is clearly expressive, but is also discoverable. Although intuitively it sounds easier to control an explicit feature, such as explicitly increasing the gesture's size to increase the level of red, users actually learned more quickly when they explored gesture variations to see if they could “make it red” or “make it green”. We also mapped the characteristics of each gesture to variations of professionally designed fonts (Figure 15, center) and emoji expressions (Figure 15, right). In each case, users enjoyed having fluid control over their output, as well as the option of later adjusting the level of variation. It would be interesting to make Expressive Keyboard more explicitly appropriable, by letting users remap its expressive output features to different input characteristics.

CommandBoard [4] offers users a dynamic Human–Computer Partnership that combines the three principles, Discoverability, Appropriability, and Expressivity. It supports gesture typing as well as providing a free-form interaction space above the soft keyboard, where users can draw gestures that invoke commands. If the user is unsure of how to perform a gesture command, she pauses to display an Octopocus dynamic guide. Alternatively, she can gesture-type any currently available command name, followed by the v-shaped “execute” gesture above the keyboard, to invoke it. She can create personally memorable, but recognizable gestures with Fieldward, and map them to available commands, or commands with parameters. For example, a single, continuous gesture can invoke the “color” command, which displays a color bar she can choose from. Users can modify the shapes of their gestures as they gesture type to produce a range of expressive output, with varying fonts, colors, and emoji expressions. Users enjoy several methods of discovering its various capabilities, but also appropriate it and generate expressive output, adapting it to meet their individual needs.

6.5.1 Analytical and Critical Power. Mood board design is an iterative process that involves three key phases: collection, composition, and reflection. Designers first seek visual inspiration by collecting images from physical magazines or online image depositories. Here, Discoverability refers to the problem of discovering appropriate images, as well as understanding how the system searches for those images. Next, designers crop, resize, and transform images, adapting them to correspond to their design concept. Here, Appropriability refers both to the ways they modify these images, as well as how they appropriate other objects, such as fabric swatches or dried flowers, to convey their ideas. Finally, designers find words that convey the mood board's meaning, so they can explain it to external stakeholders [47]. Both latter phases involve Expressivity, as they compose images and text to communicate their personal vision. The overall process is highly iterative, requiring fluid transitions across phases, and also highly co-adaptive, as designers adapt their ideas to the images that inspire them, while also adapting those images to better express their ideas.

When we apply a critical lens to mood board design, our studies of designers suggests that Discoverability poses the most problems. Few current tools support serendipitous image discovery, beyond browsing through images curated by other designers, e.g., on Pinterest. Computer-based search forces designers to transform vague, yet-to-be-articulated visual ideas into concrete, text-based search terms: They struggle to find the right words to describe images they have not yet seen. Search engines do, of course, return many images that the designer can choose from, but this poses an additional discovery challenge, since the system never reveals which criteria were used to select those particular images. Worse, the designer cannot fine-tune or adapt future searches to achieve more suitable results: Subsequent searches may return different images or more of the same, depending on hidden machine learning algorithms—The choice is never under the user's explicit control. Finally, search engines often generate large numbers of highly similar images, whereas what the designer really wants are related images that differ from each other as much as possible.

Appropriability is also an issue, since designers must adapt features from different creation tools to achieve desired effects. For example, Adobe Photoshop modifies and transforms images, whereas Adobe Illustrator is better for experimenting with different layouts, and pen-based drawing programs are better for making quick sketches. Designers sometimes adapt less obvious tools, such as using Powerpoint to crop images, because the designer already knows the tool. Designers also appropriate characteristics of certain images, such as generating a color palette from a favorite image, or using an existing color palette to modify other images to match.

Expressivity is clearly fundamental to the whole process, since each mood board reflects the individual designers’ choices, influenced by the design brief, other collaborator's opinions, their access to appropriate images, and their own individual taste. However, since current digital mood boards support only one phase of the design process at a time, designers must switch tools, which risks breaking their creative flow. Many designers also have difficulty expressing their visual concepts with words, and struggle to articulate a story that explains the semantics of clusters of images or the mood board's overall meaning [47]. Although designers can choose from a wide variety of search and creativity support tools, none offer intelligent support for discovering appropriate images and their meanings (Discoverability), nor do they offer integrated tools for adapting images and text (Appropriability) to express their ideas (Expressivity).

6.5.2 Constructive Power. Our studies show that mood designers’ needs vary: Sometimes they can articulate what they are looking for; sometimes they are stuck and need inspiration; and sometimes they want to successively refine their ideas, alternating between offering and receiving suggestions. We designed ImageSense [93] as a Human–Computer Partnership that offers all three types of intelligent assistance across the collection, composition, and reflection phases. The key challenge is how to let designers control the level of user- and system-based agency, such that the system's actions benefit designers, without interrupting them. ImageSense achieves this through three embedded systems, ranging from fully guiding the process with ImageCascade, to alternating control with Semantic Collage [94], to reacting to suggestions from May AI [92]. This creates a Human–Computer Partnership where designer and system each discover and react to relevant information from the other, and adapt images and text of the mood board accordingly.

Figure 16 shows the ImageSense collaborative mood board, with a shared central canvas ⓐ for composing images and text, and a “maybe” area ⓑ for storing potentially interesting images. It supports Discoverability by letting designers search with combinations of images and text ⓒ, to discover new images ⓓ. Designers can also proactively upload their own images ⓕ. When the designer seeks inspiration, the ImageCascade (Figure 17, left) supports serendipitous discovery of new images drawn from human-curated collections. They can select from their own images or relevant sets of high-quality curated images, which they can drag to the main canvas.

When the designer wants to actively refine a search, Semantic Collage (Figure 17, middle) lets users combine multiple images and text, for example, combining an image of a restaurant table and the keyword “office space”. In addition to producing relevant images, the system also generates semantic labels extracted from the images and text in the search box, in descending order of importance (Figure 17, right). The designer can respond to the system's suggestions, e.g., by removing the word “room”, or adding a new term, e.g., “cozy looking coffee house”. In this form of mixed-initiative Discoverability, the user and system each interpret the other's actions, then produce output that the other interprets, in turn. Finally, if the user is stuck, May AI offers system-initiated interaction via the “suggest-o-matic” dial ⓖ, which suggests new images, search terms, weighted tag clouds, and color palettes ⓗ. Designers can choose among different agents, e.g., asking for “less of these” images, or adjust the type of semantic analysis, e.g., to emphasize emotional content rather than identifying physical objects.

The various tools for adapting and modifying images and other mood board elements support Appropriability, including a tool palette ⓘ for cropping, resizing, recoloring, and transforming images and text, an interactive color palette derived from selected mood board images ⓗ. The user's ability to modify both the type of semantic analysis and the resulting semantic labels, color palettes and tag clouds generated by the system, as well as the immediate reanalysis of each image as it is cropped, resized, or otherwise transformed, allows users to adapt the content to meet current needs. Here, the system alternates agency with the user, adjusting its interpretations of the images as the designer changes them, while the designer reacts to the system's updated semantics and suggested images.

Finally, ImageSense supports Expressivity, with the emphasis on human agency for composition and reflection tasks. Designers retain full control when choosing and arranging images, text labels, and other decoration, but may also seek system-initiated information, such as word clouds based on their preferred semantics, or color palettes derived from chosen images. In summary, ImageSense supports a highly iterative design process, that lets the user move fluidly through collection, composition, and reflection phases, while fully controlling when and how to access intelligent assistance, creating an effective Human–Computer Partnership.

7.1.1 Activity Theory in Psychology, Anthropology, and Sociology. Activity theory has led to a wide and thriving interdisciplinary research community focusing often on learning and development, with a specific focus on collectives and the development of situated practices. This community has strong member groups all over the world in psychology, anthropology, sociology, organizational theory, and more (see, e.g., [135]).

7.1.2 Activity Theory in HCI. Since the mid 1980’s, activity theory has been explored as a basic perspective on HCI [22, 118]. In an attempt to break with cognitive science-based HCI, a theoretical platform has been established, using dialectical materialism [59, 116] to focus on human beings acting with technology in real-life situations [145, 154] (see also Star [142]).

Human activity is the analytic starting point. Human beings undertake actions and operations together, mediated by objects used as artifacts. For example, a group of people convenes in the Zoom video-conferencing system to plan their daily work. They may show each other what they are doing by sharing spreadsheets or documents through screen sharing. Operations from earlier generations of activity are crystallized, or reified into next generation artifacts. In Zoom, one participant is an owner of the meeting and all other participants can raise their hand to participate. In addition, artifacts are representations of certain modes of acting in the activity [22]. The Zoom example illustrates that, no matter how people actually collaborate in Zoom, it is based on a model with an owner who has a different status than the other participants and, e.g., cannot/is not supposed to use the raise-hand function. How this reification happens is illustrated in Bærentsen's analysis of the historical development of the hand gun [7]. The analysis points out how new organizing principles (war formations), new technologies (e.g., gun-powder and cartridges), training of soldiers, new affordances, and so on, all have developed in a dialectical relationship with the hand gun, and hence have left traces in hand guns as they have developed historically. In a similar way, online meetings are developing alongside their tools, including, e.g., shared documents and spreadsheets. People collaborate and this collaboration is mediated through technology; more generally human activity is mediated by artifacts. Human beings are embedded in praxis, in routines and activities that they learn, do, and change, together and through their interaction with material objects and artifacts [22]. Technological mediation of human activity is hence the primary principle and focus of this perspective.

Artifacts mediate human relationships with technological objects, and artifacts are themselves such objects. They have the attention of human users in breakdown situations [19, 22, 24, 129] or in more deliberate design and building situations [24]. Objects are outcomes of building processes but they are also artifacts, or mediators that help users act on other objects, in ways they could not without using the mediator [13, 22]. Bødker and Klokmose [29] focus on the plurality of objects and artifacts and talk about artifact ecologies. In addition, Nicolini et al. [121] propose to understand objects as performing at least three types of work: Objects motivate and allow collaboration; they allow participants to work across different types of boundaries; and they constitute the fundamental infrastructure of the activity. They are also multiple, heterogeneous, and potentially conflictual. Bardram [10], Sørgaard [146] as well as Bødker and Lyle [33] and Larsen-Ledet et al. [101, 102] have characterized cooperation as happening through interactive objects, as coordination around the object and co-construction of future use with the object.

Bødker and Klokmose [29], in developing the argumentation for mediation as a key principle, emphasize dialectical thinking as a way to understand things concretely in all their movement, change, and interconnection, in unity with their opposite and contradictory sides. Since movement and change are essential parts of dialectical thinking, this approach focuses on development of use. Bødker and Klokmose address the development of artifact ecologies beyond singular artifacts in use by communities. This perspective is further developed by Bødker et al. [32, 34]. The focus of the Communities & Common Objects perspective is on the social, addressing both the shared practices that keep communities together and develop over time and the fact that people are always collaborating. Hence we talk about common objects (Figure 18).

In other words, common objects must be understood as always changing, and existing in a field between the well known and the epistemic, i.e., how they seed the future. Accordingly it is important to always understand and address practices and technological objects historically, by looking back as well as looking toward the future and what they may become, i.e., how technological objects may change human practice. Addressing technological objects in use means focusing on breakdowns that happen in use [22, 129], when the common object stops supporting the activity. It is equally important to understand and address the seeds of future use embedded in the objects, which is really the core of a generative theory. These epistemic objects embody what is not yet known, and provide motivation for the creation of new knowledge [121].

Technological objects never exist alone, and ecologies and their dynamics are important, both analytically, critically, and constructively. Technological objects exist in ecologies together with other objects used by people, not in singular and isolated activities. They participate in webs of activities and act as boundary objects [143] across activities.

However, the perspective does not only provide an analytic framing. By virtue of its emphasis on dynamics, it also provides a framework for critically apprehending the breakdowns and challenges of an empirical situation with or without the deployment of a particular technological design, and for constructing such designs with an emphasis on their future use.

7.3.1 Mediation through Common Objects.. The underlying principle of this perspective is mediation. Common objects mediate human activity, as artifacts, in accordance with the basic principles of activity theoretical HCI. Common objects stand between the human users, they are material that gets turned into outcome, and they serve as mediating artifacts of the collaborative activity.

7.3.2 Material and Communicative Mediation.. Common objects are simultaneously material, social, and communicative, because common objects stand between people, both materially and communicatively, and mediate their joint activity. A particular focus of common interactive objects (CIO) is the language and concepts related to cooperation and transfer of experiences across a community. Bødker and Klokmose [30] point toward the use of conceptual blends to constructively focus on this aspect.

7.3.3 Malleability over Time.. Interaction needs to be understood as it unfolds over time and in the collaborative activities between people using technological artifacts and objects. The unfolding over time is a matter of development of community practice, individual routines, technology in use with respect to the history, the present, and the epistemic, i.e., the not yet known, possible, and desired future. Appropriation relates to development in use of both the common objects and the collaborative use activities. Objects can be collaboratively tailored and appropriated in use [11], and hence common objects should be understood as changeable and malleable over time.

7.3.4 Multiplicity of Artifact Ecologies. Human use of particular artifacts and objects happen across activities, configurations of people, applications, and devices, and it is important for the interaction to embrace such transitions and substitutions [29].

7.4.1 Analytical and Critical Power. With the general concern for mediation, the principle of Material and communicative mediation focuses on the parental leave planning process as a product of the negotiation between the parents and the surrounding stakeholders, in particular how the parents may use their rights for parental leave, as determined by national legislation and managed by the municipality, while making the most of the payment that either parent gets from their employers. The legislation constrains this planning by the fact that the rights of the parents are interwoven with one-another. The municipal office is an important source of advice as well as the body that approves the final plan.

As a critical lens, the principle of Material and communicative mediation specifically raises issues regarding establishing the best possible solution in terms of total leave-time, split between mother and father, total income during the leave period, the possibility of spending leave-time together, and saving leave-time for later. Conversations and thoughts about “what if” scenarios are mediated through common objects, shared only by the parents. Parents often do not have all the information necessary to calculate these scenarios, and the legislation is difficult to work with due to its inherent flexibility. As part of the counseling process, these “what if” scenarios can be explicitly shared with a caseworker.

The principle of Malleability over time specifically focuses on the fact that the planning period for parental leave is up to nine years per child. Parents may desire to change their plan, e.g., when the time comes for the child to start daycare. Over the nine-year period, many events may happen that make a change of plan necessary: new jobs, additional children, and so on. Consequential changes need to be reiterated with all stakeholders and it is, from a critical lens, quite complicated to keep track of these changes since both the parents and the municipality mainly documents them as time slices, with documents that formalize the here and now at the time when a version of the plan was agreed.

Due to the complexity of plans and their economic consequences, sharing of experiences is attractive among friends and groups of parents. However, since the actual plans are highly dependent upon specific details, it is difficult to apply plans across pregnancies, and the municipal workers caution against such replication of plans. The actual care plan can be changed regarding the future, but time and money spent cannot be taken back. This balance illustrates the critical use of the principle of Malleability over time.

Multiplicity of artifact ecologies emphasizes how, when parents have decided on the best solution, it has to be communicated to and negotiated with the respective employers and added, e.g., to their HR systems. The manifest objects of the activity typically consist of a number of rule-books, and a number of documents that contractually specify the actual plan/agreement for each parent. While the parents may have discussed among themselves some kind of informal calculation of their options, typically the “what if” scenarios are explored orally over the telephone with a municipal caseworker. Sharing is therefore complicated and oftentimes the municipal office gets questions from expectant parents who cannot understand why and how their situation differs from that of their friends and relatives. This is also true for parents who search for information in the limited available Internet sources.

7.4.2 Constructive Power. We used Material and communicative mediation to address the many challenges of joint parental leave planning, resulting in a new concept and early prototypes of CaseLine [37]. CaseLine is a timeline/calendar-based planning and overview tool to be shared between parents, and toward employers and public authorities (Figure 19). CaseLine enables citizens to help themselves and each other in understanding, planning, and applying for parental leave funding through a timeline design. CaseLine is designed as a Common Object to facilitate communication and collaboration between citizens and municipal caseworkers, and ultimately also between the parents and, e.g., their employers and unions.

CaseLine helps families manage and visualize shared time resources by introducing two mediating Common Objects: The parental leave plan, which was until then rather implicit in the interaction between the parties, and the timeline itself, which maps the plan and its possible changes. Sharing these objects with authorities and employers is made explicit in the tools/instruments available. The parental leave time chunks can be placed on the timeline in order to explore particular plans, and the time chunks that are not yet accounted for remain visible. The timeline has a calendar and can be zoomed in and out. In some versions of the design, we experimented with attaching the legal contractual documents to points on the timeline.

Applying the Malleability over time principle, CaseLine was designed to support and explore interaction in collaboration between the parents and with their municipality, in ways that could extend to other communities, from sharing parental plans with friends to sharing income agreements with labor unions and employers. CaseLine served as an epistemic object in that it seeded discussions among parents about, e.g., how they could share their plan or the principles behind it with their relatives, or with wider circles of friends. These discussions also related to the possibility of sharing CaseLine plans, e.g., on Facebook groups or on a Labor Union or company website.

CaseLine supports open exploration between parents, as well as controlled sharing with the municipality, when parents allow access to their plan. In addition, the municipality can provide general plan elements online. The CaseLine elements may be tailored by, e.g., employers and unions, and used by parents as a basis for their agreement with the municipality, or by each parent in their agreement with their employer. However, a shared timeline raised a number of concerns even among the parents: “Do parents of a child wish to share with each other all information about their interaction with employers, and government? Even if this is the case, can such openness also be assumed if the parents are divorced, but share joint custody over the child?” [37]. Consequently, each citizen needs to be able to make a clear distinction between exploring possibilities and sharing information with other stakeholders such as employers and the municipality. CaseLine tools, hence, provide both an experimental sandbox and a more formal part where the actual agreements are made visible. CaseLine builds on malleable objects such as plans and rules, but also works with the notion that some objects need to be frozen as decisions are made, so that time and money spent do not come back when one tries to change the plan. Shared overview and mutual awareness were important for the cooperation between parents as well as when it came to sharing with municipal case-workers.

Multiplicity of artifact ecologies. CaseLine introduced the plan as a Common Object to mediate the parents’ planning process as well as the communication with the municipality and the parents’ employers. This object keeps track of the leave time spent and the one planned, and can be manipulated (and explored in sandbox mode) by the parents together or one at a time. The plan can be shared with the municipality, and also serves as a record of the formal contracts that have been made between the parties, something that is currently difficult for parents to do over an extended timescale, leading to numerous phone calls to the municipality.

In this presentation of the case study, we have made the generative principles more explicit than they were at the time of conducting the empirical research. Yet the basis of the theoretical concepts was present from the outset, and the notion of CaseLine was due to a critical analysis of the municipal interactive setup where documents, rather than Common Objects, were the driving force. Designing CaseLine as a Common Object helped investigate both the ability for parents to explore and optimize parental leave plans over the years, and the ability to share plans with other expectant parents as well as employers and municipal officers. The principles further helped critically appraise, e.g., how boundaries change when parents change employer or get divorced.

7.5.1 Analytical and Critical Power. The principle of Material and communicative mediation is used to analytically address how, in a traditional lecturing situation, the first step in achieving an adequate understanding of the subject matter requires exposure to the common basis for the lecture with objects such as books, concepts, models, and lecture slides and notes on the blackboard produced by the lecturer. Even abstract models and concepts are made commonly available through diagrams, examples, and metaphors. Modern lecturing activities are mediated by more complex sets of artifacts and objects, for good or ill. Lecture theaters are typically equipped with projectors that the lecturer uses to project her presentations, made with, e.g., Microsoft Powerpoint or Apple Keynote. Secondary screens and blackboards/whiteboards may be available as well. Lecturers often share materials and course plans with students through online learning platforms. The students bring a number of technological devices to the lecture, to have access to such material, to take notes (for themselves or for sharing with their class or study group), or even to entertain themselves if bored (Facebook and YouTube are well known in this capacity, but Bødker and Christiansen [27] even mention a more “exotic” case of poetry reading on an iPhone).

The starting point for InPlenary is largely based on a critical appraisal of the authors’ own lecturing experiences. The critical analysis is summarized as follows [95]: When entering a lecture hall or a classroom, students have some common understanding of how to behave and what is about to happen. In case they have forgotten, the physical space gives strong indications toward its purpose and serves as an implicit common information space. In contrast, when seated and opening a laptop, the realm of possibilities is much more disconnected and detached from the primary activity. InPlenary critically tackles this detachment by focusing on the physical space as the scope for designing a digital system that supports co-located learning activities. Based on this, the principle of Material and communicative mediation has been key for the critical analysis of how to understand and change the possibilities for future lecturing.

Analytically, it is often the case that the class materials are predefined, and in that sense static, and that neither lecturers nor students have a chance to share their notes and other material during or after the class. This emphasizes the principle of Multiplicity of artifact ecologies.

Modern lecturing activities are characterized by Multiplicity of artifact ecologies: Projectors, the lecturer's PC, Powerpoint or Keynote presentations, additional screens, blackboards and whiteboards for everybody in the room to view, as well as materials and course plans shared through online learning platforms, student laptops and devices for note-taking, social media, and so on. A critical investigation of Multiplicity of artifact ecologies available in and around the lecturing situation is a good starting point for further development.

Analytically, it is interesting to note how much Facebook and other tools are used for entertainment and as back-channels through which students interact with each other during the lecture. This has not been investigated specifically for this case study, although it points to elements of Malleability over time among students. In contrast, none of the above technologies offer much possibility for collaborative appropriation and object malleability, in particular across students and lecturer. How to hand over control between lecturer and students and the role of the physical place can therefore benefit from an analysis through the Malleability over time principle.

7.5.2 Constructive Power. InPlenary [95] is a prototype system (Figure 20) built to support and develop university lectures to be actively mediated by the students’ and lecturers’ personal computers and software. It is based on the idea that it is possible to activate students in collaborative activities focusing on the lecture topic through their various brought-in devices in lectures and classes. InPlenary uses existing infrastructure and personal devices to distribute the lecture presentation across multiple devices, embed learning activities within the lecture, use personal devices as an entry-point for active participation, co-develop the lecture presentation as a common lecturing artifact throughout the lecture, and couple users to common lecturing objects based on their connection to a wireless access point.

The users of InPlenary are teachers and students. Its Common Objects are designed to mediate human activity by supporting existing practices of both lecturers and students, as illustrated in Figure 20. By activating students’ personal devices as part of the primary activities, InPlenary applies the principle of Mediation through common objects. It leverages the duality of Material and communicative mediation to encourage active participation: InPlenary requires physical presence in the lecture for participation and subsequent access to the common objects produced during the lecturing activity. Participation in InPlenary is anonymous, but physical co-location serves as social moderation.

InPlenary integrates with existing teaching and note-taking practices to support broad participation, both from the perspective of the lecturer and the students, thereby applying the principle of Multiplicity of artifact ecologies. Using the system should add something extra to the lecture, without destroying traditional lecturing and use of slides. This also implies supporting as many personal devices as possible to avoid exclusion by incompatibility. InPlenary is set up to support the process of building a mutual understanding during the lecture and sees the slides and the information generated in the lecturing activities as common resources, following the principle of Malleability over time. As mentioned above, the collaborative appropriation and object malleability has yet to be constructively explored further.