🕹 Joy-Con Clickers

This page is playable as slides!

Press p to play as slides, then j and k to page.

Turning Switch's Joy-Con 🕹 into a clicker is non-trivial effort because you need to poll keypress events by yourself with the Gamepad API.

Yet Another Reason to Buy A Switch that Has Joy-Cons

Some contexts

Ever since I started building the slides features, I also from time to time toy with the ideas of controlling my slides in some funny ways.

One of the silly ideas is to brew in a few magic words such as, "next page", or, "ok, next" - the words that you will likely say during your talk when you're about to slide to the next page.

  • I'm building my own slides feature ("do chefs cook at home?")
  • use one page that serves as resource page, an article to read, and slides to present
  • good talk scripts are also good reads
  • (initial version on rcs site)

Then my appetite grew bigger

  • how do I control the slides ah?
  • so I built keyboard control first (which is quite normal)
  • normal clickers by default can use wan

omg what are these

  • Making things fast in world of build tools by Surma & Jake Archibald | JSConf Budapest 2019

Here another one that I discussed with my friend Huijing is to turn her favorite toy, a mechanical key tester that she keeps clicking like a maniac, into a bluetooth connected key(board). My feature request is to have two keys, j and k, respectively, because that's how my slides page forward and backward.

Then we were both at JSConfBP earlier this year. And during the closing keynote by Surma and Jake on the first day she texted me:

Jake and Surma are each holding a switch controller 😁

My thoughts then went a round trip from that point back to first hearing of the idea from Jinjiang who talked about this as he prepared for his talk on keyboard accessibility (slides in Chinese / 挖掘鍵盤的潛能) for Modern Web 19, and brilliantly recognizes accessibility issues as extended explorations of how we interact with devices and hence web pages. Considering the fact that any time I give a talk I consider myself half disabled (brainless and one hand holding on to a mic, likely), I find this idea a perfect fit.

Plus, the idea of holding on to a Joy-Con to control your slides is simply too cool and fun to miss.

The Gamepad API

Before I played with this, my naïve imagination was that I write something similar to keyboard event handlers and be done with it.


By the way, what's your expectation?


// some kind of key press events?
document.addEventListener('keydown', event => {
  // do things?
});

If typing describes our excitement with games

  • key presses
  • press and holds
  • combination key strokes
  • poses
  • joystick

There are only two APIs

  • gamepadconnected
  • gamepaddisconnected

But no, WRONG. The only event handlers that the Gamepad API exposes are gamepadconnected and gamepaddisconnected. What essentially we have to do is to start polling keypresses (normally done by starting a gameloop) on gamepadconnected and stop this process on gamepaddisconnected.

Browsers will receive the gamepadconnected event when the controllers connect.

You need to poll key presses yourself


dunno about you but I actually felt quite cheated.

connecting the game pad

window.addEventListener('gamepadconnected', evt => {
  // display information that the game pad is connected
  document.getElementById('gamepad-display').innerHTML = `Gamepad connected!`;
});

Inside evt

If you also have very simple brain like I do, you'll probably also like the gamepad API. There's no fancy wrapping / controller / whatever, just that evt object containing real time data regarding the status of your Joy-Cons.

I'm also brain washed by the two keywords in the React community, "immutable" and "re-render" and so for a brief moment I could not comprehend the fact that that evt object just quietly gets updated to the Joy Con's current state. There's no "re-rendering" and everything mutates.

Let's take a closer look at what's inside that evt object:

{
  "axes": [1.2857142857142856, 0],
  "buttons": [
    {
      "pressed": false,
      "touched": false,
      "value": 0
    }
    // ... more (altogether 17) buttons
  ],
  "connected": true,
  "displayId": 0,
  "hand": "",
  "hapticActuators": [],
  "id": "57e-2006-Joy-Con (L)",
  "index": 0,
  "mapping": "standard",
  "pose": {
    "angularAcceleration": null,
    "angularVelocity": null,
    "hasOrientation": false,
    "hasPosition": false,
    "linearAcceleration": null,
    "linearVelocity": null,
    "orientation": null,
    "position": null
  },
  "timestamp": 590802
}

Those fields together describe the concurrent status of the joy con. And in particular:

  • axes: array describing the joystick navigation
  • buttons: array of "buttons" object, each describing whether it's pressed or touched, value being from 0 to 1 describing the physical force
  • timestamp: can be used to decide which update comes latest
  • pose: handhold data, looking very powerful ah

What is different from our normal mindset of interactions with browsers is that, the browsers have already done a lot of work for us, such as polling key presses and exposes a declarative API where we can say onKeyPress then do this.

Essentially, what we want to do is to keep polling this object to acquire the most current information. And in this context we do so by implementing a "game loop".

Note that

  • all data is mutating to reflect the most current state of the joy con

so how to poll key presses ah

"game loop"

  • it's like a snake that eats its own tail

The main idea is game loop is that it's a snake that eats it's own tail.


const gameLoop = () => {
  // does things

  // run again
  gameLoop();
};

And if you remember the gamepadConnect we wrote earlier, besides announcing that our Joy Con is connected, we start the game loop there:

const gamepadConnect = evt => {
  // start the loop
  requestAnimationFrame(gameLoop);
};

Interopting with rAF

still not quite

let start;
const gameLoop = () => {
  // ...
  // run again
  start = requestAnimationFrame(gameLoop);
};

const gamepadConnect = evt => {
  start = requestAnimationFrame(gameLoop);
};

const gamepadDisconnect = () => {
  cancelAnimationFrame(start);
};

Using with React

import * as React from 'react';
import { gamepadConnect, gamepadDisconnect } from './gamepad';

const JoyConController = () => {
  React.useEffect(() => {
    window.addEventListener('gamepadconnected', gamepadConnect);
    window.addEventListener('gamepaddisconnected', gamepadDisconnect);
  }, []);
  return <div className="gamepad-display" />;
};

And then, we define the actual interactions inside our game loop.

Implementing the game loop

const gameLoop = () => {
  // get the gamepad
  var gamepads = navigator.getGamepads
    ? navigator.getGamepads()
    : navigator.webkitGetGamepads
    ? navigator.webkitGetGamepads
    : [];
  if (!gamepads) {
    return;
  }

  // do things...
};

I think the thing we're most not used to is the fact that on every requested key frame, we need to inquire in the object that contains everything whether a key is pressed.


const gameLoop = () => {
  // ... got the gamepad already
  var gp = gamepads[0]; 
  const gamepadButtons = gp.buttons; // array of 17 buttons

  // for example, 8 is 'minus'
  if (buttonPressed(gp.buttons[8])) {
    console.log('pressed minus!');
  }
  // ... run again
};

const buttonPressed = key => {
  if (typeof key == 'object') {
    return key.pressed;
  }
  return false;
};

codepen earlier this year

I happen to have drawn a Switch using CSS roughly a year ago 😅 So convenient, let's use that. If you're interested, here is the original CodePen for the Nintendo Switch art. And once again, the original design credits to its original creator Louie Mantia.

Original design Nintendo Switch art by Louie Mantia


And since a button is either pressed or not pressed, we can then use this to toggle a "shake" UI on those buttons:

// 8 is 'minus'
if (buttonPressed(gp.buttons[8])) {
  document.getElementById('minus').classList.add('active');
} else {
  document.getElementById('minus').classList.remove('active');
}

More complex interactions can be acquired by waiting a little between actions, which I won't go into at the moment. But you can 😉

You can check out this CodeSandbox for a demo of the key pollers. Of course this is not the tersest implementation, but there's no mind burning function passes. It's good for brain health.

maybe with a bit of API

// initialize gameloop with an option
const options = {
  x: () => toggleMode(modes.presentation),
  y: () => toggleMode(modes.notes),
  a: () => toggleMode(modes.article),
  b: () => toggleMode(modes.speaker),
  // plus is on the RHS, minus on the LHS
  plus: toggleTimer,
  minus: toggleTimer,
  trigger: () => slideTo('next'),
  bumper: () => slideTo('prev'),
}

let start;
const gameLoop = options => () => {
  // ...
  // run again
  start = requestAnimationFrame(gameLoop(options));
};

const gamepadConnect = options =>  evt => {
  start = requestAnimationFrame(gameLoop(options));
};

const gamepadDisconnect = () => {
  cancelAnimationFrame(start);
};
gamepadButtons.map((button, index) => {
  if (buttonPressed(gp.buttons[index])) {
    typeof options[button] === 'function' && options[button]();
  }
});

window.addEventListener(
  'gamepadconnected',
  gamepadConnect({
    x: () => toggleMode(modes.presentation),
    y: () => toggleMode(modes.notes),
    a: () => toggleMode(modes.article),
    b: () => toggleMode(modes.speaker),
    // plus is on the RHS, minus on the LHS
    plus: toggleTimer,
    minus: toggleTimer,
    trigger: () => slideTo('next'),
    bumper: () => slideTo('prev'),
  })
);

sliiiiiiiiide

It slides all the way to the end because our "game loop" is iterating very fast.



  • 🤞 maybe we want to throttle a little bit

This is not the most thorough solution but it'll work, roughly the code looks like this:

recall that...

  • throttling will delay executing a function, it will reduce the notifications of an event that fires multiple times
  • debouncing will bunch a series of sequential calls to a function into a single call to that function, it ensures that one notification is made for an event that fires multiple times

throttling

var  throttleFunction  =  function (func, delay) {
	// If setTimeout is already scheduled, no need to do anything
	if (timerId) {
		return
	}

	// Schedule a setTimeout after delay seconds
	timerId  =  setTimeout(function () {
		func()
		
		// Once setTimeout function execution is finished, timerId = undefined so that in <br>
		// the next scroll event function execution can be scheduled by the setTimeout
		timerId  =  undefined;
	}, delay)
}

const options = {
  x: throttle(() => toggleMode(modes.presentation), 300),
  // ...
};

  • there are more proper ways to do this, but we'll leave it liddat for now

the next bug

  • getting the closure right with hooks

  • with hooks, each render cycle is bound to its own... everything

const Slides = () => {
  // ... other stuff

  // register gamepad connected
  React.useEffect(() => {
    window.addEventListener(
      'gamepadconnected',
      gamepadConnect({
        trigger: () => slideTo('next'), // << 👀
      })
    );
    window.removeEventListener('gamepaddisconnected', gamepadDisconnect);
  }, []);

  return <div className="gamepad-display" />;
};

there are multiple ways to go around this

  • instead of
useEffect(() => {
  const id = setInterval(() => {
    setCount(count + 1);    
  }, 1000);
  return () => clearInterval(id);
}, [count]);

  • make the set state fn not reliant on current state
useEffect(() => {
  const id = setInterval(() => {
    setCount(c => c + 1);
  }, 1000);
  return () => clearInterval(id);
}, []);

decoupling updates from actions

  • using reducer is a great way to do this

because then the reducer is responsible for looking at the current state and update accordingly


let me introduce you to the few things that my slides feature takes care of

  • mode
  • slide number
  • timer
  • timer running / paused

so i took the chance to refactor the whole component and used useReducer


const options = {
  trigger: throttle(() => dispatch({ type: 'next' }), 300),
}

getting back to the game pad

  • highlight pointer - moving sth using the joy stick

... if you recall from the gamepad object

{
  "axes": [1.2857142857142856, 0],
}



const joystickMap = {
  '-0.4285714285714286': [0, -1],
  '-0.1428571428571429': [1, -1],
  '0.1428571428571428': [1, 0],
  '0.4285714285714286': [1, 1],
  '0.7142857142857142': [0, 1],
  '1': [-1, 1],
  '-1': [-1, 0],
  '-0.7142857142857143': [-1, -1],
  '1.2857142857142856': [0, 0]
};

const stable = [0, 0];

let left = 0,
  top = 0,
  offset = stable;

const getJoystickNav = num =>
  joystickMap.hasOwnProperty(num) ? joystickMap[num] : stable;
const gameLoop = options => () => {
  // ... got the gamepad object gp

  offset = getJoystickNav(gp.axes[0]);
  left += offset[0];
  top += offset[1];

  // ...render accordingly with the offset left and top
}

Revisit game loop

This section is WiP.

are we... implementing a game?

(Game loop:) Almost every game has one, no two are exactly alike, and relatively few programs outside of games use them.

the original goal of a game loop

Decouple the progression of game time from user input and processor speed.

let's look at some pseudocode?

while (true)
{
  processInput();
  update();
  render();
}
  • original intent (in game) is to align time

what exactly are we doing

  • treat the joy con as an input device
  • observe our behavior & design the interactions

  • implement the interactions

    • clicks
    • holds
    • hold & swipe
    • joystick

Caveats

1Each browser has different mappings. In some cases, even whether you connect two Joy Cons at the same time result in different mappings.

  • the Joy-Con never turns off ah?
  • FireFox crashes >90% of the time when you connect two JoyCon to two tabs
  • different browsers and different game pads have different key mappings

Takeaways

  • learn browser APIs
  • think in terms of interactions and input devices

Links