A little Bit of Everything

There is basically one thing that differentiates you, the aspiring data scientist, that has stumbled upon the bitwise operator, and Neo, the chosen one from the matrix trilogy: You have no choice between a blue and a red pill. In order to understand bitwise operation you have to leave the shiny surface and have to face that it’s all just zeros and ones.

Let’s go there together.

I was asked recently about the nature of & and | in pandas. A topic that is somewhat hard to google because it’s not exactly bitwise … I’ll hope this article helps, well, a bit.

It’s uses some examples in python and pandas (as this was taken from my original answer in a data science chat) but is applicable for many languages and implementations.

Deep within a computer mainboard lives an electronic circuitry named central processing unit and it looks on signals of high and low voltage. Everything in our modern digitalised world starts from here. It’s scary, I know.

We’ve come a pretty long way from the early days, when we started interpreting high voltage as 1 and low voltage as 0. This way, the computer history, can be described as a history of abstractions. Everything in IT is an abstraction and abstraction is by far the most important concept, be it in software architecture, the internet or virtualization. All abstraction layers have a shared ancestor — 0s and 1s.

Variables in python for example — things like strings, integers, lists and so on — are abstractions. The computer processor does not know about those things, it does only know about 0 and 1 (bits).

If you take strings, there are simply human-made conventions about how we arrange 0 and 1 in order to represent them. You may have heard about those conventions (e.g. UTF-8 or the older ISO-8859–1). A main goal of python 3 was to solve some kind of convention-problems once and for all (by implementing an underlying character set named unicode as default). That’s something that is harder to do than one might think, e.g. PHP Version 6 failed over this task and was never released.

Whenever you see weird characters within a text, it’s effectively a mix up of those conventions (which are named encodings). It’s bits in the wrong order and the abstraction leaks through.

To make a long story short, way below, at CPU level, everything is actually bits.

Bitwise operators work on that level. E.g. the number 201 is 1100 1001 in bits and the 15 is 0000 1111 (a full example is available here). If you compare those bitwise, it’s just walking through those bits and evaluate each bit from the first sequence against the second.

1100 1001 starts with a 1 and 0000 1111 starts with a 0, bitwise and (&) would evaluate that to 0. Both end with a 1, hence that would evaluate to 1 (similar as True and True is True while True and False is False).

If you do that for the entire sample you’d end up with 0000 1001.

However, it’s something you may never use. It’s used in network protocols and many other low-level tools you probably don’t need to touch. That’s because low-level problems have been, well, already abstracted away.

Operator overloading

I’ve said that you may never use bitwise comparison, while you may think that you’ve already stumbled upon them, e.g. in pandas boolean-vector comparisons:

2    2
4 4
dtype: int64 0 0
1 1
5 5
dtype: int64

But chances are: You have not. You’ve touched the surface of the inner dynamics of the python language.

Operators are things like * / + - (basic math operations), comparisons like < <= > >= and the bitwise operators (& for bit wise and and | for bitwise or) are operators itself. The fun thing about python is that these operators are not something limited to build-in things like integers and strings, everyone who is writing a class can implement it on their own - or prefer not doing it.

E.g. the python creators obviously implemented multiply (*) on integers, but they didn't implement the bitwise and-operator (&) on lists. That's because it would be unintuitive what it does - how to bitwise compare lists? Compare their values or bit sequences of their value? Hm, who knows. When you compare a list bitwise to another list, python will raise an error claiming no one implemented what's to do:

unsupported operand type(s) for &: 'list' and 'list'

While it may be counterintuitive at first, there are very good use-cases to implement your own operators and that’s called operator overloading.

Imagine you’ve done all the machine learning classes and came up with a self-taught poker bot that’s cracking the low-limit tables on pokerstars (you wouldn’t use it, because it’s illegal — of course). That bot would need some way of evaluating hands and you may end up writing a class that represents poker hands.

You could then compare hands doing this:

does_beat and __gt__ would have the same implementation (figure out which hand beats the other). Some people (including me) would argue that the latter is more elegant and expressive and the obvious way to do it (that's from the python zen), but that's debatable. Making that kind of decisions will allow you to create clean class design.

That’s what software engineering is about, elegance is important.

You can overload the bitwise operator as well, by defining the __and__() and __or__() methods on a class.

The Django ORM (an abstraction layer for SQL and Databases) uses the latter to implement the OR in SQL. It’s not the only way of doing it (e.g. SQLAlchemy has an or_ method to do it, but does heavy operator overloading as well).

It’s a design decision one does make in order to create expressive, readable, elegant and clean code.

I’ll give an example where this decision was taken wrong (at least in my opinion). Python strings can do things like:

>>> 'This book costs %i $.' % 112
'This book costs 112 $.'

That’s called string interpolation — in order to not have to mix up strings, one can write the string with placeholders and exchange them with the values at the end. However: Dividing a string by an integer leads to replacing a placeholder — is that intuitive?

I’d say no. Don’t use it. Use .format() instead, it is more powerful as well and the obvious way to do it.

Others say yes, use it, because it’s more performant and used in a lot of languages. Again others say that’s premature optimization and discussions like this can bring mailing lists to explode.

Which brings us to the question …

Is bitwise operator overloading in pandas boolean vector implementation good or bad design?

Pandas bitwise operator on vectors is designed for boolean vectors (simplified: lists that contain only True or False).

The bitwise operator in pandas is implemented like this: it compares every entry of the first list sequence with the entries of the latter, pretty much like the bitwise operator walks through each entries in a bit sequence. And True and False can be thought of as bits (1 and 0).

I’d say yes. It’s intuitive enough to understand what’s going on and it leads to very elegant, expressive syntax. I like that design.

There are a few drawbacks as well: it raises the learning curve for beginners who get confused about these advanced language features.

Decisions like that pop up all the time when you are designing a software library and most of the time there is no perfect match.

Creating an unbeatable rock, paper and scissor bot

We’ve done lot’s of theory, so let’s do a little practise with our new toy named operator overloading.

Here’s a class that represents an actual selection in a game of rock, paper, scissors. It takes the option one wants to play as initial argument, the game logic itself is using operator overloading exclusively:

Ok, let’s quickly do a few tests, if the implementation is working:

>>> print('Is paper beating scissor? {}'.format(
'Yes.' if Selection('paper') > Selection('scissor') else 'No.'
Is paper beating scissor? No.

And the full stack of a random game:

>>> import random
>>> c1 = random.choice(Selection.OPTIONS)
>>> c2 = random.choice(Selection.OPTIONS)
>>> s1 = Selection(c1)
>>> s2 = Selection(c2)
>>> print('{} > {}:'.format(c1, c2), s1 > s2)
rock > scissor: True
>>> print('{} >= {}:'.format(c1, c2), s1 >= s2)
rock >= scissor: True
>>> print('{} == {}:'.format(c1, c2), s1 == s2)
rock == scissor: False
>>> print('{} != {}:'.format(c1, c2), s1 != s2)
rock != scissor: True
>>> print('{} < {}:'.format(c1, c2), s1 < s2)
rock < scissor: False
>>> print('{} <= {}:'.format(c1, c2), s1 <= s2)
rock <= scissor: False

Ok, that looks good.

Since we’re already here, let’s let the computer play a few games with competing strategies.

An interesting thing about games is that it requires risk taking if you want to win. Hence, if you decrease risk — by getting rid of your exploitable spots — you can’t be defeated. A strategy that is not exploitable — even if your opponent knows the strategy — is called game theory optimal (GTO).

In complex games, such as No Limit Hold’em you could make a mathematically profit against worse opponents if you happen to be able to play GTO (that’s because errors in No Limit can be disastrous and if your opponent does only a few, it’ll yield a profit). In rock, paper and scissors, though, this leads to your opponent having no chance of winning — and you having no chance of winning either.

A perfectly balanced range, the game theory optimum, isn’t that complex in rock, paper and scissors — it’s just perfect random picking. That’s very hard for humans as they are not very good in generating random sequences. The art of exploiting human-made-random-sequences is the underlying dynamic, the game flow of many competitions.

However, let’s have a look on the bold statement that a GTO strategy can’t win or loose and eventually will tie up against any strategy in the long run. We’ll do that by letting GTO play a million times against another strategy, in that case the always take rock strategy.

Let’s see how GTO is playing out against that:

matplotlib will generate something like this:

Yeah, it’s a tie.

That’s an interesting idea behind game theory optimal. When you are a better player in any game than your opponent, you are better off without a perfectly balanced range. Finding more lucrative spots than your opponents is the way to go. In the above example it would be better to mix in more papers against that special opponent, even if it makes yourself vulnerable against scissors.

GTO on the other hand, can help you, if your opponent is better than you.

There was an interesting moment in european soccer a few years ago, when the special one, Inter Mailand Coach Jose Mourinho, managed to get very close to GTO, maybe as close as you can get in such a complex game with infinite variables.

Mourinho defeated the immense favourite FC Barcelona with his team of good defenders by basically teaching them a very destructive way of playing soccer. They only focused on not being exploitable at all, which ended up of them having little chance of exploiting the opponent either. So, the tactic was focusing on playing 0:0 and hoping for a penalty shoot out — if not an individual error would determine the outcome. Hence, the games against bigger faves went 0:0, 0:1, 1:0 or so, they got lucky on top and managed to win the champions league.

I hope you could take something away from this.