Simulating User Input Easily in Python

Image generated using Midjourney

It’s almost an eventuality for every programmer to need to somehow simulate user input for one of their projects. Whether it’s a desire to interact with a system that doesn’t have an API and only a UI, or just wanting to mess with a friend by making a script randomly press keys and jiggle the mouse. It’s useful skill to know how to do. So, today I’m going to write out the Pynput library for python, which can monitor user input and, for the main purpose of this blog, SIMULATE user input. We’ll cover reading user input with this library in a future article!

Link to the docs: https://pynput.readthedocs.io/en/latest/index.html

Simulating the Mouse

Controlling the mouse through Python is made easy using Pynput mouse Controller class.

Creating an instance of this class exposes movement, button presses and scrolling. Making it extremely easy to mimic user input.

Creating an instance of the class is as easy as the following

from pynput.mouse import Controller as MouseController
from pynput.mouse import Button as MouseButton
 
mouse_controller = MouseController()

Now that we have the mouse_controller we can start exploring the various features it exposes to us. The best place to start would probably be the current position of the mouse.

# read the current mouse coordinates from the screen in (x, y) format
mouse_position = mouse_controller.position
print("current mouse position: ", mouse_position)

The coordinates are based on screen space, the min/max values of the coordinates will be determined by a number of factors, how many monitors you have, their resolution, their orientation ect. The (0, 0) point will be the top left pixel of the main monitor, and the monitors resolution will determine the maximum point. For my setup I have a 2k Monitor with a resolution of (2560, 1440), so the coordinate (2559, 1439) will result in the mouse being set to the bottom right of my monitor. Remember the position is zero based.

If you have a multi-monitor setup the coordinate space can extend beyond the min/max values of your main monitor, for example I have another 2k monitor oriented to the left of my main monitor, a coordinate of (-1481, 1100) would put the mouse around the center of my second monitor.

From what I have seen Pynput doesn’t have a good built-in way to determine the monitor setup or resolution, so if you want to determine those values programmatically, you’ll have to find other packages, otherwise experiment with your setup and find the values you need for your use case. Always remember that the resolution and orientation will affect these coordinate values.

So, let’s demonstrate how to actually set the mouse position in code. Just like we read the position using the mouse_controller.position we can also manually set that attribute to snap the mouse to that position.

from pynput.mouse import Controller as MouseController
from pynput.mouse import Button as MouseButton
monitor_resolution = [2560, 1440]
 
monitor_center = [(x/2)-1 for x in monitor_resolution]
 
mouse_controller = MouseController()
 
# move the mouse to the center of the screen
mouse_controller.position = monitor_center

The controller also has a function called move, which will move the mouse relative to its current position. So if we wanted to move the mouse 100 pixels up from its current location we would call mouse_controller.move(0, -100). Remember that positive values trend towards the bottom right, so to move “up” towards the top of the screen we need a negative value.

So here’s a small example moving the mouse up and down:

from pynput.mouse import Controller as MouseController
from pynput.mouse import Button as MouseButton
import time
 
sleep_time = 2
monitor_resolution = [2560, 1440]
 
monitor_center = [(x/2)-1 for x in monitor_resolution]
 
mouse_controller = MouseController()
 
# move the mouse to the center of the screen
mouse_controller.position = monitor_center
 
# put some sleep statements in so we can see the mouse at it's new location
# move the mouse up 100 pixels
time.sleep(sleep_time)
mouse_controller.move(0, -100)
 
# move the mouse down 200 pixels, 100 below our starting point
time.sleep(sleep_time)
mouse_controller.move(0, 200)
 
 
# back to center
time.sleep(sleep_time)
mouse_controller.move(0, -100)

Mouse buttons are also just as easy. In the controller object we have a press and a release function, as well as a click function. The press and release do just as you expect, you signal which button you want to influence and it will either press or release the button, very straight forward. The click can be used for actions like double clicking, this way you don’t have to have multiple press and release calls to get the effect.

Here’s how you can do that in pynput:

from pynput.mouse import Controller as MouseController
from pynput.mouse import Button as MouseButton
mouse_controller = MouseController()
 
# press left button
mouse_controller.press(MouseButton.left)
# press right button
mouse_controller.release(MouseButton.left)
# double click left button
mouse_controller.click(MouseButton.left, 2)
 
# available mouse buttons: MouseButton.left, MouseButton.right, MouseButton.middle

With that we reach the end of controlling the mouse. With this you have all you need to simulate most user actions with the mouse! Next up, controlling the keyboard.

Simulating the Keyboard

As for controls for the keyboard we only have 3. press, release and type. Press and release work the same as the mouse versions, indicate which button you want to affect and it will either press or release that button. It can either accept a string indicating the key that needs to be typed, or a value from the Key values. For example, if you want to hold the shift key you would indicate that with ‘Key.shift’. The type function takes in a string input and will press and release the keys to type out the contents of the string.

Here’s how to do that:

from pynput.keyboard import Key
from pynput.keyboard import Controller as KeyboardController
import time
 
keyboard_controller = KeyboardController()
 
# press and release the a key, typing 1 instance of the letter, 
keyboard_controller.press('a')
keyboard_controller.release('a')
 
# spend some time admiring our wonderful a
time.sleep(2)
 
# Now let's type an a using shift to make it a capital letter
keyboard_controller.press(Key.shift)
keyboard_controller.press('a')
keyboard_controller.release('a')
keyboard_controller.release(Key.shift)
# alternatively, you could just use keyboard_controller.press('A') and that would also
# type out a capital letter A, but it's useful to know how to press those special keys
# for other usecases
 
# spend some time admiring our wonderful aA
time.sleep(2)
 
 
# now type out a full string
keyboard_controller.type("Hello, World!")

That’s really all there is to it for simulating the keyboard. In a way it is much more straightforward than the mouse since we only have buttons to worry about.

With those you now have the tools needed to programmatically simulate user input through python! This opens up some interesting possibilities for fun side projects, especially in the CV and AI space that don’t have a programmatic api, which is personally how I hope to use this in the future!

In a future blog I’ll return to pynput and explain how it can also be used to listen for user input, another important side to this little library!

Hope you found the article interesting, and Happy Coding!