Stroke Shortcuts Toolkit (SST) is a library to add stroke shortcuts to a Java Swing graphical user interface. It includes (i) the Design Shortcuts environment to define the mappings (stroke, command) for commands of Java Swing applications and (ii) a very simple API to activate these shortcuts in the application. The designer can decide to offer several types of visual clues to make end users able to learn these shortcuts (augmented tooltips, menu preview or strokes sheet). SST also provides an API to easily implement any stroke based interaction by offering high-level events and ways of handling these events (listeners or state machines).
As a keyboard shortcut allows to invoke a command by typing a combination of keys, a stroke shortcut allows to invoke a command by drawing a symbol using a pointing device (mouse, pen, etc.). Keyboard shortcuts are usually diplayed beside their corresponding menu item as illustrated by the figure below.
|
|
|
|
| keyboard shortcut | stroke shortcut | both types of shortcut |
In this tutorial, we will use two examples: a music player and a graphical editor. The code for these two examples can be downloaded here. This archive contains the source code of these examples structured into two Java packages: example.player and example.graphicalEditor. The archive also contains the SST and SwingStates jar files and some scripts so you can directly launch the examples using the .sh or .bat files:
|
|
The central object that links the stroke input to commands of an application is a stroke shortcuts manager (class strokes.StrokeShortcuts). It manages a shape matching recognizer that outputs the name of the best matching template given a stroke drawn by the user. To enable stroke input in an application, the first thing the developer has to do is to define the mappings (template, name) handled by the recognizer. If name is the accessible name of a command in the application, a stroke drawn by the user that is recognized as the template name will invoke the corresponding command in the application. If name is a free string that is not an existing command, the developer can define the action to perform through the use of listeners as presented in the section Handling stroke events.
The developer can use the Design Shortcuts environment to define mappings (template, name) for an application. Launching this environment on the application windows for which he wants to map commands with stroke shortcuts is achieved by writing the following lines of code. In this example, we want to define stroke shortcuts for the commands found in the main frame of our music player and the commands found in the "about" dialog box of this music player.
1 DesignShortcuts designShortcuts = new DesignShortcuts(); // player and player.dialogAbout are respectively // the main frame of the music player and the dialog box // that pops up when the user clicks on the "about" menu item. 2 designShortcuts.addWindow(player); 3 designShortcuts.addWindow(player.dialogAbout);
As shown on the Figure above, the Design Shortcuts environment is divided into three parts:
To define a mapping (template, name), the developer picks a stroke in the dictionary to add it to the set of shortcuts he wants to make available in his application. Each time a stroke is added to the set of shortcuts, a window pops up as shown on the Figure below. This window contains a text entry and the list of commands found in the windows currently attached to the Design Shortcuts environment. To define a shortcut, the developer simply picks one of the command name in the list and clicks the Ok button. Alternatively, to associate a free name to a stroke, he writes down the desired name in the text entry field and clicks the Ok button.
|
|
|
|
The dictionary presents a base list of 9 predefined strokes for the developer to choose from. The transformation buttons displayed on top of each stroke can be used to create new strokes from these predefined strokes. These four buttons allow to rotate the stroke (in both directions) and/or mirror it (horizontally or vertically). They are displayed at the top of every stroke in the dictionary but also in the set of shortcuts already defined so the strokes can be modified before or after a mapping has been registered.
Several strokes can also be concatenated to create a new stroke. Click on the strokes you want to concatenate while keeping the SHIFT key pressed and click one the '+' buttons displayed on beside the selected strokes. The resulting stroke is thus computed by concatenating the selected strokes respecting the order in which they have been selected. For allowing to concatenate two similar strokes, a stroke can be duplicated in the dictionary using a pop up menu as shown on the scenario below.
|
|
|
|
|
|
|
Finally, one can add some free strokes to get more options in the dictionary. At the bottom of the list of primitive strokes, the developer can find a "Free stroke" button that opens a white frame for drawing a stroke that will be added as a primitive to the dictionary. In the example below, we define a question mark stroke for the "Help" command.
|
|
|
|
At any time, the developer can test the recognition accuracy in the testing area. As soon a stroke is ended, the list of registered mapping is displayed in the format [score] name. The list is sorted with the best match being on the top. If the recognition acceptance is too liberal, the developer can set a maximum distance recognition threshold between the input stroke and the best matching template. He can also specify the minimum length (in pixels) of a drag to be considered as a stroke. If a stroke is too short or its distance to the best matching template is too high, the stroke is not recognized.
|
|
Once the developer is done with designing the set of shortcuts and other mappings (template, name), he can save its design in a file using the menu "File" --> "Save this strokes set". In the music player example, we have saved our design in a file named "player.strokes".
Any file containing a design can still be loaded in the Design Shortcuts environment to be later modified (use the menu "File" --> "Open a strokes set").
As we mentioned above, the central object that links the stroke input to an application is a stroke shortcuts manager (class strokes.StrokeShortcuts). To build a manager for the application windows for which the developer wants to define shortcuts, he simply writes down the following lines of code:
1 StrokeShortcuts manager = new StrokeShortcuts();
2 manager.addWindow(playerFrame);
3 manager.addWindow(playerFrame.dialogAbout);
The second step consists in registering the mappings (template, name) in this stroke shortcuts manager. Remember that our mappings for the music player example has been saved in a file "player.strokes" thus the following line loads these mappings into the manager (the role of the two last boolean parameters is explained below in the section Enabling visual clues for end users).
4 manager.addShortcuts("player.strokes", true, true);
Remark: Mappings can also be registered "manually", i.e. without relying on the Design Shorcuts environment by using the different methods available in class strokes.Stroke. See the commented code in StrokePlayer to get an example.
In SST, a stroke is defined as a series of points sent by an analog input device (a mouse or a digital pen) that starts with a press event and ends with a release event. Each stroke occurring on a "strokable" component is entered into the recognizer to get the name of the recognized template.
In our music player example, we register both the main frame and the dialog "About" as "strokable" components (lines 5 and 6 below) so the user can draw on any of the two windows. The boolean value of the second parameter of method enableStrokes specifies if all the children components must also be registered as "strokable" or not. The boolean value of the third boolean parameter specifies if strokes' ink trail must be drawn on the top of the "strokable" components.
5 manager.enableStrokes(playerFrame, true, true);
6 manager.enableStrokes(playerFrame.dialogAbout, true, true);
Since press, drag and release are events that may already be used by standard widgets, SST allows the developer to either (i) associate a criterion on the mouse press event that specifies when the stroke recognition must be enabled or (ii) disable stroke recognition on specific components. For example, no criterion is required when the user wants to stroke in the interface background while one is required for a list box on which a drag is already an action dedicated to select items in the list. In this latter case, the developer can decide to accept only strokes traced when the right mouse button is pressed (lines 7 to 11). Finally, he disables stroke recognition on the sliders for adjusting the playing point in a song and the volume (lines 12 and 13).
(Criteria are presented in more details in section Criteria)
7 manager.setCriterion(playerFrame.playlist, new DefaultCriterion() {
8 public boolean startStroke(MouseEvent event) {
9 return event.getButton() == MouseEvent.BUTTON3;
10 }
11 });
12 manager.disableStrokes(playerFrame.sliderSong);
13 manager.disableStrokes(playerFrame.sliderVolume);
The last boolean parameter of method enableStrokes (lines 5 and 6) specifies if the ink trail when tracing a stroke must be drawn on the top of the interface or not (by means of the transparent overlay available in the window containing the component: the Java Swing glasspane). Once the user ends a stroke, its ink trail is smoothly morphed into the template it matches in case of recognition success while the ink flashes red in case of recognition failure (i.e. when the closest template is greater than the set threshold or the input stroke is too short). Note that entered stroke ink is only morphed to a template scaled to the same size to avoid a too great visual change. The colors used for ink trail, the starting point of a stroke, ink trail during morphing animation and ink trail in case of non recogition can be changed using methods in the class StrokeShortcuts (e.g.,setInkColorWhileStroking, setInkColorWhenNonRecognized, etc.). If the developer has already dedicated the transparent overlay to another purpose, he is free to disable the default drawing ink mechanism and implement the feedback of his choice using stroke listeners (see section Handling stroke events).
To address the visibility problem of stroke input (i.e. users do not have a way to discover the available strokes and their meaning), SST offers three types of visual clues to make the user discover and learn the mappings:
Tooltip and Menu preview visual clues are turned in or off according to the value of the two last parameters of the addShortcuts method (line 4). A Strokes Sheet is enabled on the stroke shortcuts manager by using the method enableStrokesSheet:
14 manager.enableStrokesSheet();
The stroke shortcuts manager fires notifications each time an event related to stroke input occurs: when the user begins a stroke, adds a point to a stroke or ends a stroke. When ending a stroke, the event can be of three different types:
The first type of event occurs when the user ends a stroke for which the best matching template has the name of one accessible action in the current opened windows. The second type of event occurs when the user ends a stroke for which the best matching template has been associated with a free name. Finally, the last type of event occurs when the best matching template is too far from the stroke entered by the user (i.e. greater than the recognition threshold) or when the drag is too short to be considered as a stroke.
These events can not only be used to implement custom visual feedbacks for stroke input but also to define the behavior of free commands. When a stroke event occurs, the series of points in this stroke and the recognized name (if available) can be retrieved from the event (methods getStroke and getCommand in class StrokeEvent). The coordinates of the points in the stroke can be transformed into the coordinate system of a given component (method getStrokeInComponent(Component) in class StrokeEvent). In the two following sub-sections, we show an example in which we use a free command "Lasso" which has a circular shape to handle selection in our graphical editor.
To implement behaviors in response to stroke events, one or several stroke listeners can be attached to the shortcut manager to be notified each time a stroke event occurs. Any object implementing the interface strokes.listeners.StrokeListener can be attached to the stroke shortcuts manager using the method addStrokeListener. The StrokeListener interface proposes the five methods below:
void strokeBegun(StrokeEvent event)void strokePointAdded(StrokeEvent event)void shortcutRecognized(StrokeEvent event)void strokeRecognized(StrokeEvent event)void strokeNotRecognized(StrokeEvent event)In the example below, we show how we can be notified when a "Paste" shortcut is recognized and thus use the stroke to set the location where the objects will be pasted instead of using a predefined location as many graphical editors do. Usually, the paste position is the center of the editor's window or the initial position of the cut/copied elements translated by a predefined offset. In this example, thank to the higher expressivity of strokes, we can use the ending point of the stroke as the paste location (lines 3 to 7).
|
In this example, we also show how we can use a free command to select several graphical objects. Let's assume that we have defined the mapping shown on the figure below where "Lasso" is not a command that already exists in the graphical editor. By means of a listener, we can be notified when the "Lasso" stroke has been recognized and implement a behavior that selects all the graphical objects that intersects the bounding box of this "Lasso" stroke (lines 8 to 12). |
|
1 GraphicalEditor editor = ...;
...
2 manager.addStrokeListener(new StrokeAdapter() {
3 public void shortcutRecognized(StrokeEvent event) {
4 if(event.getNameEvent().compareTo("Paste") == 0) {
5 editor.setPastePosition(event.getPointInComponent(editor));
6 }
7 }
8 public void strokeRecognized(StrokeEvent event) {
9 if(event.getNameEvent().compareTo("Lasso") == 0) {
10 select(GestureUtils.boundingBox(event.getStrokeInComponent(editor).getPoints()));
11 }
12 }
13 });
Stroke listeners can also be implemented as state machines. To that end, the package strokes.transitions proposes 5 classes of transitions corresponding to the 5 different types of stroke events (StrokeBegun, StrokePointAdded, ShortcutRecognized, StrokeRecognized and StrokeNotRecognized) and a generic class of transition, StrokeTransition, that is triggered by any stroke event. The listener we presented just above can be implemented as the following state machine:
manager.addStrokeListener(new StateMachine() {
State start = new State() {
Transition cut = new ShortcutRecognized("Paste", ">> start") {
public void action() {
editor.setPastePosition(getStrokeEvent().getPointInComponent(editor));
}
};
Transition select = new StrokeRecognized("Lasso", ">> start") {
public void action() {
select(GestureUtils.boundingBox(getStrokeEvent().getStrokeInComponent(editor).getPoints()));
}
};
};
});
To allow developers to easily implement their own feedbacks and visual clues, the stroke shortcuts manager proposes two methods (getIcon and getPngImage) to get a visual representation of a stroke: a StrokeIcon (implementing javax.swing.Icon so it can be used as any Swing icon) or a png image. For example, we could use a png image of the "Lasso" stroke to display a tip when the user selects several objects in a row in our graphical editor example (since "Lasso" is not a command that exists in the simple editor application, its preview is not shown in the menu or in any tooltip).
Stroke ink can also be retrieved using the methods getInkTrail and getStartingPoint in the stroke shortcuts manager. The ink and the starting point are SwingStates' graphical objects that can be easily modified (see SwingStates' documentation to get a list of all the possible methods).
In this final section, we show how to use more advanced criteria than the right mouse button filter we used above to control whether or not stroking is enabled on a given "strokable" component.
A criterion implements the interface strokes.criterion.Criterion which contains two methods:
startStroke(MouseEvent event): It is called each time a mouse press event occurs on the associated "strokable" component. If this method returns true, stroking is enabled. If ink trail mechanisms are on, events are intercepted and not dispatched to the "strokable" component: the user is in the stroking mode. If ink trail mechanisms are off, events are still dispatched to the "strokable" component concurrently with stroking.
cancelStroke(MouseEvent event): It is called each time a new point is added to a stroke by a mouse drag event. If this method returns true, stroking is disabled so the current stroke is canceled. When ink trail mechanisms, the events that have been intercepted since the beginning of the current stroke are dispatched to the "strokable" at this time.
The class DefaultCriterion that implements these two methods by always returning true for startStroke and always returning false for cancelStroke. This class is useful when the developer needs to define only one of these two methods for his custom criterion.
To illustrate how we can define such criteria, we detail here how is implemented the predefined ClickCriterion already available in SST. This criterion accepts strokes only when they are preceded by a click. It listens for clicks on the "strokable" component and starts a timer as soon as a click occurs. Until this timer expires (1 second after its triggering), the user is allowed to start a stroke.
public class ClickCriterion extends MouseAdapter implements ActionListener, Criterion {
private Timer timer = new Timer(1000, this);
private boolean timerOn = false;
private JComponent strokableComponent;
private int nbClicks;
public ClickCriterion(int nbClicks, JComponent strokableComponent) {
this.strokableComponent = strokableComponent;
// register this criterion for listening clicks on the "strokable" component
strokableComponent.addMouseListener(this);
this.nbClicks = nbClicks;
timer.setRepeats(false);
}
// A stroke is accepted only when the 1 second timer is running
public boolean startStroke(MouseEvent event) {
return timerOn;
}
public boolean cancelStroke(MouseEvent event) {
return false;
}
public void mouseClicked(MouseEvent event) {
// a click occurs
if(event.getClickCount() == nbClicks) {
inTimer = true;
timer.restart();
// The background of the "strokable" component is highlighted in yellow to indicate that stroking mode is enabled
strokableComponent.setBackground(Color.YELLOW);
}
}
// The timer has expired, stroking mode is turned off
public void actionPerformed(ActionEvent e) {
timerOn = false;
// The background of the "strokable" component is turned back in white
strokableComponent.setBackground(Color.WHITE);
}
}