Ignite using Python

Photo by Hitesh Choudhary on Unsplash

Previously I talked about accessing Ignite using Python via Spark. It has advantages but it’s also a little convoluted. In the most recent version of Ignite we might be able to do better since it now comes with a native Python API. Let’s see how that works.

The Python client is a thin client, which means it connects to the grid using a binary protocol, but doesn’t join the cluster as a peer (unlike thick clients, including the existing Spark integration). One plus is that it’s easier to install.

pip install pyignite

If you’re not familiar with it, PIP is the Python package manager, like Ruby GEM or CPAN for Perl.)

And, oh dear:

Collecting pyignite
  Using cached https://files.pythonhosted.org/packages/db/95/32338b940d6b38089476bd1e2c71d8f493950c0b56c6176fe47dd41d868d/pyignite-0.3.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/vf/vv29cdg10wn1t8ts9bj869y00000gp/T/pip-install-pCx6rf/pyignite/setup.py", line 20
        def is_a_requirement(line: str) -> bool:
                                 ^
    SyntaxError: invalid syntax
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/vf/vv29cdg10wn1t8ts9bj869y00000gp/T/pip-install-pCx6rf/pyignite/

If I’d read the documentation I would have known that you need Python 3.4 or higher. I’m using a Mac, so I can use HomeBrew to install the new version of Python and then use that to install the Ignite client.

brew install python3
pip3 install pyignite

Then we can start, just like it says in the documentation:

>>> from pyignite import Client
>>> ignite = Client()
>>> ignite.connect('127.0.0.1', 10800)
>>> new_cache = ignite.create_cache('ignite')
>>> new_cache.put(1,'Arthur Dent')
>>> new_cache.put(2,'Ford Prefect')
>>> new_cache.get(1)
'Arthur Dent'
>>> for k,v in new_cache.scan():
...   print (k, ' -> ', v)
...
1  ->  Arthur Dent
2  ->  Ford Prefect

Nice.

But what happens if you try to talk to an existing cache?

Let’s create a table using sqline:

create table ignite2 (id integer, name varchar, primary key(id));
insert into ignite2 values (1, 'Arthur Dent'), (2, 'Ford Prefect');

And then use Python to access the same data:

>>> existing_cache = ignite.get_cache('SQL_PUBLIC_IGNITE2')
>>> existing_cache.get(1)
>>> print (existing_cache.get(1))
None
>>>

What? I literally just put it there!

This took some digging to get to the bottom of, but the problem is data types. On the Ignite server, the field is of type java.lang.Integer but the Python client seems to default to a Long. Luckily we can give the client some hints.

from pyignite.datatypes import IntObject
>>> existing_cache.get(1, key_hint=IntObject)
SQL_PUBLIC_IGNITE2_1a083dcd_199b_4501_8156_ff805f4ef27f(NAME='Arthur Dent', version=1)

(I found this a bit confusing. You can import the mapping for date with DateObject and the mapping for long and string with LongObject and StringObject. But for… reasons… you abbreviate integer to IntObject.)

Anyway, we can now find the correct key and value. The bad news is that it’s in weird format. (If you look on the Ignite-side, you’ll see the same thing. When you create a table without specifying a POJO this is just what happens, so it’s entirely expected, but you can’t tell me that the name isn’t weird!)

There is a way of working with these structures but for today I’m going to give you a cheat: use SQL.

>>> result = ignite.sql('select * from ignite2')
>>> for x in result:
...   print (x[0] , ' -> ', x[1])
...
1  ->  Arthur Dent
2  ->  Ford Prefect

Stay tuned for how to access the “weird” structured data.