Hi Boon Gan Ng,
You got it, that’s exactly how it works; the 5-way switch acts as arrow keys by default, and becomes a mouse on a second layer. I’ve connected the 5-way switch as if it were 5 different switches: the ground pin is the column, and each direction pin is a different row (you‘ll only need one diode because it’s only possible to activate one direction at a time).
Hope this helps! Check out my repo for more details: https://github.com/w4ilun/pocket-keyboard