How to Make Keycloak Start Up Faster When There Are a Lot of Offline Sessions

If you have offline sessions enabled in Keycloak and there are lots of them stored in the database (e.g. more than 300k) you will notice that the time it takes to start up can be up to more than 20 minutes. In this article I will share some tips we implemented to make it go faster. In our case, we managed to start up with 750k offline sessions in 2 only minutes 58 seconds, instead of ~15 minutes!

I will also explain how to programmatically change the value of the property sessionsPerSegment, both in the XML configuration and using the JBoss CLI, which is crucial to improve the startup times.

This issue has already been reported in the following JIRA tickets:

• KEYCLOAK-11019: Offline sessions preloading still slow in some cases
• KEYCLOAK-7275: Pre-loading offline sessions from DB can timeout in slow environments (DefaultExecutorService timeout)

The first step is increasing the property sessionsPerSegment from the default 10 to a value around 512.

Setting sessionsPerSegment to 512

This property seems to determine the amount of offline sessions that will be loaded up in every “batch”; at least that’s my understanding from debug level logs during startup.

Option 1: in the standalone(-ha).xml configuration

Simply add the following to your standalone(-ha).xml configuration file. You can place it next to the other <spi> nodes.

<spi name="userSessions">
    <provider name="infinispan" enabled="true">
        <properties>
            <property name="sessionsPerSegment"
                      value="${env.SESSIONS_PER_SEGMENT:512}" />
        </properties>
    </provider>
</spi>

The reason we set the value to ${env.SESSIONS_PER_SEGMENT:512} is so that we can override our (new) default value of 512 using the environment variable SESSIONS_PER_SEGMENT.

Option 2: programmatically, using the JBoss CLI

Although I recommend setting the variable using the XML configuration, it is also possible to programmatically set it using the JBoss CLI.

The approach will depend on whether the property sessionsPerSegment exists (because it was set in the configuration file) or not.

If the property was already never set before:

1. Open a terminal to your Keycloak instance and go to the folder where the binaries reside. In case you are using an official Docker image, it will be /opt/jboss/keycloak/bin.
2. Run the script ./jboss-cli
3. Connect to Keycloak using the command connect.
4. Add the userSessions SPI with: /subsystem=keycloak-server/spi=userSessions:add()
5. Add the infinispan provider to the userSessions SPI: /subsystem=keycloak-server/spi=userSessions/provider=infinispan:add(enabled=true)
6. Set the property: /subsystem=keycloak-server/spi=userSessions/provider=infinispan:write-attribute(name=properties.sessionsPerSegment,value=512)
7. Reload with reload

In case the property already existed before, you can skip steps 4 and 5.

The output should look something similar to this:

// Add the userSessions SPI
[standalone@localhost:9990 /] /subsystem=keycloak-server/spi=userSessions:add()
{"outcome" => "success"}
// Add the infinispan provider to the userSessions SPI
[standalone@localhost:9990 /] /subsystem=keycloak-server/spi=userSessions/provider=infinispan:add(enabled=true)
{"outcome" => "success"}
// Now you can add the property :)
[standalone@localhost:9990 /] /subsystem=keycloak-server/spi=userSessions/provider=infinispan:write-attribute(name=properties.sessionsPerSegment,value=512)
{
    "outcome" => "success",
    "response-headers" => {
        "operation-requires-reload" => true,
        "process-state" => "reload-required"
    }
}
// Reload!
[standalone@localhost:9990 /] reload

Add extra indices to the database

The JIRA issues mentioned in the beginning of this article mention that adding some extra indices might be helpful to speed up startup times. Maybe because we were not using the version 11 of PostgreSQL, which comes with some optimizations for queries to use indices in IN clauses, we did not see much of an improvement. But they might be worth adding nevertheless.

On the table offline_user_session, add an index on the columns user_session_id, created_on and offline_flag:

CREATE INDEX offline_user_session_user_session_id_idx ON public.offline_user_session USING btree (user_session_id, created_on, offline_flag)

Just in case, on the table offline_client_session, add an index on the columns user_session_id, timestamp and offline_flag:

CREATE INDEX offline_client_session_user_session_id_idx ON public.offline_client_session USING btree (user_session_id, "timestamp", offline_flag)

Upgrade to PostgreSQL version 11

It was briefly hinted in the previous section, but it is recommended to use at least the version 11 of Postgres, since it comes with an optimization on IN clauses to use indices.

Our startup speed improvement

Photo by CHUTTERSNAP on Unsplash

We went from ~15 minutes startup time to just 2 minutes and 58 seconds, without using PostgreSQL version 11 and without having the extra indices.

Hopefully these tips will help you as well!