You know that thing where you’re trying to debug a problem and you just know that you’re the culprit, that past-you did something stupid, and you just can’t figure out what?
Welcome to my day.
Anyway, I’m documenting my stupidity so you don’t have to suffer as long as I did.
The background: in order to debug an application that runs on a cluster can be a challenge. The way I tend to do it is to run a server in debug mode in my IDE and connect to that same server using a client. The server is super-simple:
In part 1 we talked about how to connect to Apache Ignite using Python and do some basic operations, like connecting to a cluster and running SQL queries. Today we’ll build on that and see how we can use the key-value store APIs.
To make this piece fairly stand-alone, we’re going to start from an empty cache. Let’s create one:
>>> from pyignite import Client
>>> ignite = Client()
>>> ignite.sql("create table ignite (id integer, name varchar, primary key(id));")
Then we’ll use SQL to add a couple of rows:
>>> ignite.sql("insert into ignite values (1, 'Arthur Dent'), (2…
Previously I talked about accessing Ignite using Python via Spark. It has advantages but it’s also a little convoluted. In the most recent version of Ignite we might be able to do better since it now comes with a native Python API. Let’s see how that works.
The Python client is a thin client, which means it connects to the grid using a binary protocol, but doesn’t join the cluster as a peer (unlike thick clients, including the existing Spark integration). One plus is that it’s easier to install.
pip install pyignite
If you’re not familiar with it, PIP is…
I’ve seen a few people struggling with integrating Spark and Ignite in the mailing lists. As with so much of this stuff, it’s actually quite straightforward if the stars align. But getting those stars to align can take some trial and error. Here’s my experience so, hopefully, you can skip some of it.
First, the caveats. My use case here is mainly just playing around with data. I’m not setting up a cluster of Spark nodes (though the principles apply). One of the strengths of Spark is the ability to use a DataFrame from Python, making it easy do lots…
The last couple of posts have been about using Apache Ignite on Kubernetes, and this one follows in that vein.
We’ve got our cluster up and running; we’ve got the ability to scale it using StatefulSets; we’ve activated it using a job. How do we check that it’s still up and running?
Kubernetes has a feature called a “liveness probe” that we should be using, that way it can restart the pod if something untoward happens.
I’m sure there are all kinds of clever ways of making it work. We need one that’s a balance between being quick to call…
While I’m writing about running Apache Ignite on Kubernetes… it turns out that there’s a bug in the latest version (2.7) that came out last week. Starting Ignite from the standard Docker image with
ignite-kubernetes enabled will result in a NoClassDefFoundError exception.
There’s a simple workaround: you can add
ignite-rest-http to your list of optional packages. That doesn’t commit you to using them; they just happen to include the missing dependency.
- name: ignite-node
- name: OPTION_LIBS
There’s a ticket for this. Hopefully it will be fixed soon.
A couple of weeks ago I did a talk at the OpenStack conference about deploying Apache Ignite on the Kubernetes container orchestration system. As with any talk, with only so much time to work with, some things ended up on the cutting room floor.
One of those things was how to activate the cluster when using Ignite’s native persistence support. I mentioned that you needed to do it — since it’s one of the things that complicates using a distributed database in these environments — but I didn’t really say how. In the demo, I just logged straight into one…
The background story: I’m preparing to deliver a training course. Each module of the course is in a different PowerPoint deck and, once I finished, I need to export each of them as a PDF to share with delegates. There are about twenty modules so doing this wouldn’t take that long but I would probably consider using the word “mind-numbing” to describe the process.
In hindsight, I’m not sure that writing VBA code to automate it was significantly less mind-numbing, but I’m sharing it here so you don’t have to.
Sub printAllPresentations() For Each pres In Presentations pos =…
This was going to be my first thoughts on the new Apple Watch, posted about a day after I unboxed it. But then life got in the way and I’ve now been using it for a week. What I’d planned to be a little more than a “hot take” is now probably a little closer to an actual review!
From the picture above you can see what I was using before. It’s not an Apple Watch, it’s not a competing smart watch. It doesn’t even have a battery. (To be clear, my new watch isn’t a replacement. …
Ignite has all kinds of helpful APIs to make it simple load data in, most notably Data Streamers. But, sometimes you don’t want to start your favourite IDE and start bashing out code just to load in a text file.
The rest of this piece shows a number of ways that you can achieve this. One caveat I want to share before we get started: you probably don’t want to use any of these solutions in a production environment. These are mostly for experimentation. Maybe you want to play with another feature and need to quickly load in some reference…
Started coding on a Sinclair Spectrum in 1985. Thinking about upgrading soon. I write about my experiences in the software industry.