Google’s New Project Turns Your Face into a Cursor for Android Apps

May. 15, 2024

To put it simply, this gaming accessibility tool is actually acursorthatuses head tracking and facial expressions. The tooluses your Android device’s camerato work as intended. Google also uploaded a dedicated YouTube video showcasing how thismouseworks, which you can check out:

Project Gameface uses MediaPipe’sFace Landmarks Detection APIto track those facial expressions, and ultimately, let the user control the cursor. In itsblog, Google states,

We have replicated the same idea to bring a new virtual cursor on an Android device. We are using the Android accessibility service to create a new cursor and are leveraging MediaPipe’s Face Landmarks Detection API to program the cursor in a way so it moves according to a user’s head movement.

This veryAPI contains 52 facial expressionsbrought about by 52 such face blendshape values. The tool then basically lets you map out these gestures by binding them to a mobile action. Once that’s done, you’re good to go.

Then, the tool creates a camera feed overlay to let you in on what you’re doing and how it’s working out on the screen. This, as Google states,” facilitates accurate threshold settings and a deeper comprehension of gestures.”Not just that, Project Gameface also allows you toexecute the drag function on your Android device, hands-free.

Ultimately, Project Gameface will let people with disabilities get the full-fledged Android flavor without having to use their hands. An X user who visited the Google I/O Connect event in Bengaluru last year was able to see the tool at work.

You can clearly see how, once you can get used to it, it has some solid potential. The user does not seem to face any issue with aiming as such, which is commendable as well. It’s good to see how Project Gameface has evolved from what started as acollaborationwith a quadriplegic video game streamer Lance Carr.

However, it’s not just Google that is taking such an initiative. Although Elon Musk’s Neuralink recentlyhit the headlines for the wrong reasons, its birth was a result of wanting to help paralytic patients control digital interfaces with their minds.

We are heading towards a hands-free future, for the good, and I’m certainly stoked to see the full potential of such projects. What about you? What do you think of Google’s Project Gameface initiative? Do let me know in the comments down below.

Sagnik is a tech aficionado who can never say “no” to dipping his toes into unknown waters of tech or reviewing the latest gadgets. He is also a hardcore gamer, having played everything from Snake Xenzia to Dead Space Remake.