Browsing the Web with a Gamepad

So I got bored during the lockdown and stumbled upon a strange dare. The dare was simply to use my laptop without touching the keyboard and mouse for thirty minutes.

There are some keyboard shortcuts which helped me at the beginning, like Ctrl+Tab, Ctrl+Shift+Tab, F6, F5, F11,Ctrl + +,Ctrl + - etc. There are other special hotkeys which help to manage Chrome instances like Ctrl+T to open a new tab, Ctrl+Shift+T to open last closed Chrome instances etc.

The problems which I faced while browsing

What about selecting the links? Hitting the Tab key numerous times landed me to my desired link. But it was a cumbersome task. There was a case when I hit the tab multiple times to reach the middle of the page, now moving to the top of the page would require multiple Shift+Tab.

Suppose, you are shopping on Amazon and you receive a notification from Facebook or you want to change the song played on Youtube, it will be a critical task to get hold of the notification or swipe through the tabs. There was a case when I wanted to send a picture from a webpage to my friend, but reaching to the image and then choosing the right-click options to copy the image was a difficult task. Basically, the main problems which I faced while surfing the web without a mouse was, using the right click options and navigating to a particular link using Tab key!

The difficulty of navigating without a mouse made me think of alternatives like a Gaming Controller to navigate through the computer in an easy way.

Diving into the functionalities of Gaming Controller

Almost all gaming controllers contain 6 axes which generate analog values ranging from -1 to 1. The other buttons and hats generate discrete values among {-1,0,1}.

Coming to the axes, the analog values can be used to imitate the mouse pointer because the analog values will help to define the extent of pointer movement. So the first axis may be used for pointer control and the second axis can be used to control the scrolling feature. The buttons may be used to achieve any functionality like left or right click etc.

Talk is cheap, show me the code

Okay, I am using Python for implementing the idea, but the Gamepad API powered by JavaScript may also be used for the same.

API Used: pygame, pyautogui

Code to map axis to pointer

for i in range(axes):
axis = joystick.get_axis(i) 
xx=0 
yy=0 
if i==0:
 #xx and yy are variables to scale up axis value
 xx=axis*1366
 yy=axis*768 
 #movement of pointer in X and Y direction
 pyautogui.moveRel(xx/10,0,duration=1)
 pyautogui.moveRel(0,yy/10,duration=1)

Now, we will connect the second axis to the scroll feature.

for i in range(axes):
if i==3:
 #horizontal scroll of 2 units in right side
 pyautogui.hscroll((2))
elif i==4:
 #vertical scroll of 2 units in upper direction
 pyautogui.scroll(-(2))

Now, coming to the buttons, they can be mapped to any function of user’s choice.

The trigger buttons can be used to assign left and right click.

for i in range(axes):
if i==4:
 pyautogui.click()
elif i==5:
 pyautogui.click(button=”right”)

There are other various functionalities which can be added to buttons like typing a text from voice input. One such example may be

r = sr.Recognizer() 
with sr.Microphone() as source:
print(“Say something!”)
audio = r.listen(source)
try:
 pyautogui.write(r.recognize_sphinx(audio)) except sr.UnknownValueError:
 print(“Sphinx could not understand audio”)
except sr.RequestError as e:
 print(“Sphinx error; {0}”.format(e))

╔═══════════╦═══════════════════════════╦══════════════════════════╗
║  Keys     ║      Functionality        ║ Analogous Keyboard Keys  ║
╠═══════════╬═══════════════════════════╬══════════════════════════╣
║  Axis 10  ║  Movement of mouse Pointer║           None           ║
║  Axis 11  ║  Scroll Wheel Feature     ║          Up/Down         ║
║   Home    ║  Opens a new Chrome Tab   ║         Ctrl + T         ║
║   Key 8   ║   Text to Speech Query    ║           None           ║
║Key 4,Key 5║   Left and Right Click    ║           None           ║
║ Axis 6,7  ║ Swipe through Chrome Tabs ║        Ctrl + Tab        ║         ╚═══════════╩═══════════════════════════╩══════════════════════════╝

Interested to see a demo! Go Ahead!!

Implementation of the article

Interested to try it out? You can find a demo code here.

Liked the article, Show your love by clapping and writing comments.😀