You are right that when it comes to these two examples, there is little advantage to using the Tensorflow network over the simple table. The advantage from neural networks come in when you are interested in using an environment that has too many states to store in a table, or when some states are rarely visited, but are like other more frequently visited states. For example, the game of Go has roughly 10¹⁷² possible states, and as such it would be impossible to build a table in which a Q value for each is stored and updated over time. Neural networks are able to act as “function approximators” in this case, a kind of magically extensible table.
I hope that gives some intuition as to why the neural network approach is valuable. If you check out my tutorials that come after this one, I show example environments that need neural network approaches to be solved.