Short version (April 2023)

In my PhD, I distinguish syntactic interaction techniques, in which the user manipulates the concepts of the language, and semantic interaction techniques, in which the user manipulates concepts expressed with the language.

I focus on semantic interaction and explore how we can augment textual programming languages with alternative and interactive representations of concepts that may benefit from being visualised and manipulated as something else than plain text. For example, this includes manipulating a colour code using a colour picker, and manipulating the code of a LaTeX table using an interactive grid interface.

Long version (Spring 2022)

On this page, I first explain what is the problem I am trying to solve, and why it matters. I then explain what are some possible solutions, and what is the direction I've chosen to focus on during my PhD. I then briefly review the work I did during the first half of my PhD, and what are my plans for the second half.

The problem

When we use a natural language such as English, we exploit the properties of the language to express ourselves. When we write or talk, we choose certain words and syntactic constructions rather than others because we believe they best express the idea we want to convey. Similarly, when we read or listen, we interpret the words and the way they are said and connected to understand the message that is being communicated.

Using a programming language is not very different. Just like (most?) natural languages, programming languages have rules (such as a grammar) that constrain what is considered a valid piece of code. Incidentally, one of the key aspects of software engineering is to choose between the many different ways to name something, encode a piece of data, or perform an action. These choices depend on factors such as the language that is being used, the public that will read the code, the difficulty of writing it, and the impact it will have in terms of performance, and can result in very different-looking solutions.

We thus see that no matter whether we talk with another human being or with a computer, we can express the same idea in multiple ways, and in fact, in multiple languages. As an example, anyone with a sufficient background in mathematics would quickly understand that even though “the D-wide circle centred in point P”, “the set of points at distance D/2 from point P” and “{p | distance(p, P) = D/2}” are three different sentences (the last one even being written in a mathematical language!), they all refer to the same circle. This, for me, is a good illustration of what a concept is: the reification of an idea, independant from the way it is represented.

At that point, we can observe that (1) while using a language requires instantiating each concept we refer to in one specific way, (2) each concept may (theoretically) be represented in different ways, with different advantages and drawbacks.

We very naturally exploit observation (2) in the real world: if we talk to someone about a circle, we may not only switch between different languages and notations depending on what we want to express or focus on, but may also draw it with more or less details, point to an object with similar properties, etc.

Yet, as often, numeric environments tend not to be as flexible as the real world we are used to. Once a concept such as a circle is encoded in a certain way, in a certain programming language, it is usually hard or impossible to visualise or manipulate it in another way, even though the representation used in the code may not fit our mental model or be adapted to the way we want to interact with it.

The potential solutions

What can we do to improve this situation?

One solution may be to switch to a completely different kind of language, or a completely different kind of editor (to read and write this language). Over the last decades, this approach has been explored with more or less success.

The results include non-textual programming languages such as data-flow graphs (e.g., Max/MSP, Blender's node editor), as well as non-text-centric editors such as structured editors, which were never adopted, and block editors, which have became the de-facto tool to teach programming to children (using environments such as Scratch). While these solutions may offer more adequate representations of certain concepts, they do not solve the problem mentioned above: they simply opt for different representations of certain concepts, which may be more adapted in certain situations or to a certain public (usually with no training in programming, such as children and artists). Furthermore, this approach is not adapted to support programmers who must or prefer to use existing textual languages—which may sometimes date back from the 80s or earlier and still be in use today.

In answer to the limitations of these approaches, several hybrid textual/block environments (e.g., Droplet, Pencil Code, frame-based editors) and textual/graph environements (e.g., Nodes, Enso) have been developed. Yet, these approaches are still limited by the fact they use one or two generic representations to display all the code, without considering the fact that specific concepts within the code may benefit from more specific representations.

Another solution, which has the benefit of being compatible with (existing) textual languages, is to augment the editors used to read and write code to display specialised representations of specific pieces of code when needed. This is the approach I decided to explore during my PhD, inspired by work such as Barista, Codelets, Graphite, Envision, Sketch-n-Stech, B2, mage, Livelits and Ivy.

I am interested by several sub-questions raised by this approach, such as

First half of the PhD

In the first half of my PhD, I focused on LaTeX, a very expressive language designed to describe documents that is commonly used to write academic papers and technical documents.

Michel Beaudouin-Lafon and I interviewed 11 users of LaTeX to better understand the issues they face when they use LaTeX and derived recommendations for designing better LaTeX editors from the results. We then introduced the concept of transitional representation, which are alternative representations of certain concepts encoded in specific fragments of code that are connected with the output they generate in the generated document. We applied transitional representations to LaTeX by developing i-LaTeX, a prototypal LaTeX editor that enables to visualise and manipulate the code of mathematical formulae, tables, images and grid layouts through representations that complement (1) the code and (2) the generated document. We evaluated their effects on the performance, the workload and the strategies for performing common editing tasks via a controlled experiment with 16 participants. Overall, the results show that transitionals helped participants solve tasks faster, with a lower workload, using more straightforward strategies.

This work has been published in two peer-reviewed papers: one at IHM'21, a French HCI conference, where it was nominated best paper of the conference; and one at CHI'22, the largest and most general international HCI conference. You can find more information about them in the page dedicated to my publications.

Second half of the PhD

In the second half of my PhD, I plan to step back a little from specific environments such as LaTeX and explore how representations of common concepts (e.g., tables, style properties) could be reused across very different situations and languages. To that goal, I am currently working on uncoupling (1) the textual patterns that encode the concept from (2) the data models used by the alternative representations, and am interested in better understanding how to create bidirectional mappings between the two.