Portfolio of team ExSitu
EXtreme SITUated interaction

Article: Fallacies of Agreement

T. Tsandilas. Fallacies of Agreement: A Critical Review of Consensus Assessment Methods for Gesture Elicitation. ACM Transactions on Computer-Human Interaction, 25(3):1–49, June 2018. DOI: 10.1145/3182168 HAL: hal-01788775

• PDF version of the article (from HAL)
• HTML version of the article (from ACM DL)
• Web site with additional material

Context

A fundamental problem of interaction design is how to effectively map computer actions to user input, or gestures. A method for dealing with this problem is to seek intuitive mappings between gestures and computer actions that naturally emerge. Gesture elicitation studies are commonly used for this purpose based on the highly cited work of Wobbrock et al. (2005, 2009). The analysis of the data extracted from such studies has predominantly relied on agreement measures introduced by Wobbrock et al. (2005). Later, Vatavu and Wobbrock (2015, 2016) introduced additional measures, formal statistics, and two significance tests.

Our TOCHI article revisits these analysis methods and shows that they present serious problems:

1. The measures employed fail to address systematic biases, such as users consistently selecting the same gestures regardless of the presented referents;
2. The statistical tests are built on flawed assumptions, resulting in substantial Type I error rates well above nominal levels; and
3. Interpretation guidelines provided in previous papers rely on problematic probabilistic reasoning, leading to overly optimistic conclusions.

The article proposes concrete solutions to these challenges, drawing on measures and statistics commonly used in inter-rater reliability studies.

Challenging established results and questioning prevailing practices is not typical in HCI research, making the publication of this work a lengthy journey. The author had to construct a compelling argumentation, blending intuitive examples with robust experimental findings to clearly illustrate both the identified problems and their solutions.

Contribution

The article investigates the following three problems: (i) measuring agreement among participants in gesture elicitation studies, (ii) evaluating the magnitude of agreement, and (iii) supporting statistical inference based on agreement measures. For each of these problems, it exposes the shortcomings of existing methods using three distinct approaches:

1. Illustrative examples that highlight the flaws in the authors' probabilistic reasoning.
2. Monte Carlo experiments that assess errors across a large number of iterations. To conduct these experiments, the article introduces distributional models that account for systematic biases.
3. A reanalysis of data from several gesture elicitation studies.

In addition to the article, we provide supplementary materials to enable other researchers to replicate our findings and deploy the proposed methods.

Impact

The article is cited 56 times according to Google Scholar and has sparked discussions. For instance, Oulavistra's 2019 Twitter post discusses the implications of our findings regarding natural gestures.

Vatavu and Wobbrock (2022) published a 70-page response article, citing our article 74 times. They conducted a new series of experiments, which overwhelmingly support our conclusions regarding the shortcomings of their statistical tests. However, the authors do not agree with the necessity of employing chance-corrected agreement measures in cases of open-ended gestural vocabularies, despite our detailed analysis and case studies on this matter.

In a subsequent paper (Tsandilas and Dragicevic, 2022), we demonstrated that chance-corrected agreement indices better align with the design objectives of gesture elicitation studies, as they penalize biases that could potentially lead to conflicts among referents.

References

T. Tsandilas and P. Dragicevic. 2022. Gesture Elicitation as a Computational Optimization Problem. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’22). ACM, 13 pages. https://doi.org/10.1145/3491102.3501942. web site

J. O. Wobbrock, H. H. Aung, B. Rothrock, and B. A. Myers. 2005. Maximizing the Guessability of Symbolic Input. In Extended Abstracts on Human Factors in Computing Systems (CHI EA ’05). ACM. https://doi.org/10.1145/1056808.1057043

J. O. Wobbrock, M. Ringel Morris, and A. D. Wilson. 2009. User-defined Gestures for Surface Computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09). ACM, pp. 1083–1092. https://doi.org/10.1145/1518701.1518866

R.-D. Vatavu and J. O. Wobbrock. 2015. Formalizing Agreement Analysis for Elicitation Studies: New Measures, Significance Test, and Toolkit. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ’15). ACM, pp. 1325–1334. https://doi.org/10.1145/2702123.2702223

R.-D. Vatavu and J. O. Wobbrock. 2016. Between-Subjects Elicitation Studies: Formalization and Tool Support. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, pp. 3390–3402. https://doi.org/10.1145/2858036.2858228

R.-D. Vatavu and J. O. Wobbrock. 2022. Clarifying Agreement Calculations and Analysis for EndUser Elicitation Studies. ACM Transactions on Computer-Human Interaction 29, 1, Article 5 (January 2022), 70 pages. https://doi.org/10.1145/3476101

creARTathon summer schools

We ran two creARTathon summer schools in 2021 and 2022 that included Master classes in Human-Computer Interaction (HCI), Artificial Intelligence (AI) and Design plus a "creative hackathon'' where multi-disciplinary groups of students brainstormed and implemented a "creative, interactive and intelligent" work of art that was displayed to the general public at an art gallery in Paris.

• 2021 Program Schedule
• 2021 Making-of Gallery
• 2022 Program Schedule
• 2022 Making-of Gallery

Context

As researchers in Human-Computer Interaction, we are interested in how the rapid advances in artificial intelligence will affect users, especially how to design effective intelligent systems that augment human capabilities rather than deskill or replace users. We need to train the next generation of researchers and practitioners to take a multi-disciplinary design approach that values the contributions of each discipline and benefits from cross-collaboration. We decided to create a summer school format, which we call the creARTathon, that brings together graduate students in HCI and AI with masters students in art and design and challenge them to create innovative artifacts that explicitly require the expertise of all participants.

Organization

We received support from Inria, the Université Paris-Saclay, Societies, the Ministry of Culture as well as design schools in Paris (École Boule, ENSCI, ...). We ran two versions of the creARTathon in the summers of 2021 and 2022, with 70 students from Masters and Ph.D. programs in HCI, AI, art and Design. Each year, 35 students took Masters Classes offered by faculty from Inria and the Université Paris-Saclay, and were coached by four professional artists as they brainstormed, designed and implemented an original artwork that was "creative, interactive and intelligent".

After developing their work in a FabLab (FabLab Digiscope, FabLab Centrale/Supélec), each of the seven multi-disciplinary teams presented their work to a jury during a preliminary exhibition at the École Normale Supérieure Paris-Saclay, and then participated in multi-day exhibit at a galery in Paris (Galerie Joseph, Galerie ENSCI) open to the general public.

We organized the projects so that no participant could succeed without the contributions from students from other disicplines. Most said they were inspired by their collaborations with students from extremely different backgrounds and several said it "changed their life". Interestingly, the eight professional artists were also inspired by their particiption, which opened up new design directions and demonstrated not only the potential of AI, but also the power of creating interactive art works that require bi-directional participation by both the human and the intelligent system.

The creARTathon was organized by Wendy Mackay and Janin Koch, with faculty presentations by Michel Beaudouin-Lafon and Baptiste Caramiaux. ExSitu students Téo Sanchez, Robert Falcasantos and Junhang Yu participated.

Impact

In addition to the public exhibitions at ENS Paris-Saclay, attended by local researchers, the public exhibits at the galeries in Paris drew a large number of visitors from the general public, including over 250 participants at the opening vernissage on the first day.

The exhibits were attended by members of the ministry of culture and the head of the Villa Médicis, who has since launched a collaboration with Inria between researchers and artists on the plateau de Saclay. The winners of the 2021 creARTathon were asked to present their work at a 2-day colloquium at the Centre Pompidou on teaching across arts, design and sciences. One group was approached by a manufacturer who was interested in pursuing a sofa with interactive materials, other students submitted their works for subsequent art exhibits, which were presented in London, Paris and Berlin. One student wrote his Masters thesis at ENSCI based on his project, and another student from the École des Beaux Arts and Centrale/Supélec has begun a Ph.D. at Université Paris-Saclay.

National project: PEPR eNSEMBLE

PEPR eNSEMBLE is a 7-year, 38.25M€ national program on the future of digital collaboration that started in 2023. It gathers over 80 research teams across France, from multiple disciplines: Computer Science, Ergonomics, Cognitive Psychology, Sociology, Design, Law, Economics. Its goal is to fundamentally redefine digital tools for collaboration, by making collaboration and sharing native features of computer systems, by promoting interoperability across collaboration infrastructures, and by fostering a rich ecosystem providing collaborative services.

The original project proposal provides an overview of the program. eNSEMBLE is organized into 6 projects contracted separately with ANR, the French National Research Agency:

• Governance: Project management and transversal actions (Technology development, Communication and dissemination, Valorisation and transfer, Education and training, International strategy);
• PC1 CATS: Collaboration spaces, addressing co-located, remote and hybrid real-time collaboration;
• PC2 PILOT: Long-term collaboration, addressing synchronous and asynchronous collaboration across time;
• PC3 MATCHING: Collaboration with intelligent systems, addressing the use of AI to both support collaboration and as an agent in a collaborative activity;
• PC4 CONGRATS: Collaboration at scale, addressing technology support for large collectives;
• PC5 TRANSVERSE: Cross-cutting themes, addressing issues around research and evaluation methodology, ethics, and sustainability.

Organization of the eNSEMBLE projects 1 to 4

Context

PEPRs (Programmes et Équipements Prioritaires de Recherche) are part of France 2030, the French Government's program that funds a variety of research and transfer activities.

The idea of proposing a research program rooted in Human-Computer Interaction came up when the first call for PEPR was published in 2021. Michel Beaudouin-Lafon (Ex-Situ) and Laurence Nigay (Université Grenoble Alpes) started to brainstorm a proposal around Ubiquitous Computing. With the support of their Universities, they submitted a project proposal called DIFFUSE (Interaction avec l'informatique diffuse). The project was not accepted but a resubmission was encouraged with the condition of enrolling national research organizations.

Based on this feedback and on further brainstorming with other colleagues, including Wendy Mackay, the focus of the project shifted from the techno-centric approach of DIFFUSE to the human-centered topic of digital collaboration, with an explicit goal of establishing a multi-disciplinary program. We organized a conference call to see if there was a critical mass to establish the project, and were astonished that over 100 people joined, from a variety of disciplines. It is important to note that, despite the fact that the topic of computer-supported collaboration is well-established internationally, with conferences and journals such as ACM CSCW, ECSCW and the CSCW Journal, it is not an established community in France. Very few research teams focus on this topic, and a few others, like Ex-Situ, address it as one of their research themes.

As the proposal matured, CNRS and Inria were approached and, after a period of reflexion, agreed to be co-pilots of the program with Université Paris-Saclay and Université Grenoble Alpes. Together with the large number of interested research teams, this created momentum that led to the successful submission of the project. The project was officially accepted in July 2022, with very good reviews. After a long phase of negotiation with ANR, the 6 projects of the eNSEMBLE program officially started between June and December 2023.

Contribution of ExSitu and LISN

Ex-Situ members were instrumental in spearheading, establishing and running PEPR eNSEMBLE. Michel Beaudouin-Lafon is one of the four co-directors of the program. Wendy Mackay is one of the co-coordinators of the TRANSVERSE project.

Moreover, Stéphane Huot from Inria is another co-director of the program. He was a member of ExSitu until he became Research Director at Inria Lille in 2014. Aurélien Tabard is co-coordinator of the CONGRATS project. He is a former Ph.D. student of ExSitu, now Assistant Professor in Lyon. Finally, Chris Sandor, Professor at LISN in the VENISE team, is co-coordinator of the CATS project, which is managed by Université Paris-Saclay.

PEPR eNSEMBLE, and especially project CATS, benefits from another France 2030 project funded as part of the Equipex+ program: CONTINUUM, a 13.6M€ equipment project gathering 30 large interactive visualization platforms across France, including immersive CAVEs and high-resolution wall-sized displays. The goal of CONTINUUM is to expand the capabilities of the existing platforms, create new ones, and interconnect them to support remote collaborative work. CONTINUUM is recognized by the government as a National Research Infrastructure. Michel Beaudouin-Lafon is the scientific director of CONTINUUM (and was scientific director of the former Equipex project DIGISCOPE).

The project manager, Sébastien Dubos, is part of Ex-Situ, as is the project manager of CONTINUUM, Alexandre Kabil.

Impact

PEPR eNSEMBLE was inaugurated in Grenoble on October 5, 2023, and the kick-off meeting took place in Paris on November 17, 2023. The project is therefore recent but has been very active:

• The first eNSEMBLE Doctoral School, organized and taught by Wendy Mackay, took place right after the inauguration, with 19 Ph.D. students.

• The first call for Ph.D. theses took place in April-June 2023 and resulted in the funding of 15 Ph.D. theses (8 men, 7 women). LISN received funding for two Ph.D. students: one in the Ex-Situ team, the other in the VENISE team. eNSEMBLE will fund over 100 Ph.D. theses across four calls, one per year. The 2024 call received 40 applications.

• A call for post-doctoral fellows is ongoing. So far, LISN has received funding for one post-doctoral fellow, in the CPU group.

• A call for expressions of interest (AMI, Appel à Manifestation d'Intérêt) was issued in late 2023. Based on the results, a call for project (AAP, Appel à Projets) with 5M€ of funding was opened in the first trimester of 2024. Project proposals are currently being evaluated by an international jury, with decisions to be made by the end of summer 2024.

These actions are supported by the close collaboration of the management team: the four co-directors meet once a week, and so does the Executive Committee (co-directors + 3 co-coordinators of each of the 5 projects). All decisions are taken collegially so that the actions of each project are coherent with the overall program.

Inauguration of PEPR eNSEMBLE

First eNSEMBLE doctoral school

Timeline of the project