Sometimes, we need to create complex indexes on a database, like creating a GIN index using trgm extension for Postgres, etc. Currently, we can’t generate this kind of index definition using doctrine index mapping configuration.

For this purpose, the best way to create such indexes is to add them manually inside a doctrine migration. But by doing this, the table mapping is no more synchronized with the database, so command like this one (using Symfony console):

bin/console doctrine:schema:validate

Will fail, and creating new migrations will be bloated with the DROP INDEX SQL statement.

To make it properly work again with doctrine, a pertinent solution will be to ignore such indexes from doctrine dbal schema parser.

How to

I’m using these packages versions which are the most recent when I wrote this article :

• doctrine/dbal: 3.6.5
• The Symfony doctrine/doctrine-bundle: 2.10.2

On latest doctrine dbal versions, listening to schema related events is deprecated, and will be removed in doctrine dbal v4.

The proper way is to override the SQLSchemaManager. Depending on your Database, you need to extend the good one. On my side, using Postgres, have to create a class that extend : Doctrine\DBAL\Schema\PostgreSQLSchemaManager

Example (don’t forget to add your namespace / rename if needed) :

<?php

class CustomPostgreSQLSchemaManager extends \Doctrine\DBAL\Schema\PostgreSQLSchemaManager
{
    private const INDEXES_TO_FILTER = [
        'idx_custom1',
        'idx_custom2',
    ];

    protected function _getPortableTableIndexesList($tableIndexes, $tableName = null)
    {
        $indexes = parent::_getPortableTableIndexesList($tableIndexes, $tableName);

        foreach (self::INDEXES_TO_FILTER as $index) {
            if (isset($indexes[$index])) {
                unset($indexes[$index]);
            }
        }

        return $indexes;
    }
}

Here, indexes named “idx_custom1” and “idx_custom2” will be ignored. To configure ignored indexes we just need to update the INDEXES_TO_FILTER const array.

Now that we set up the index ignore/filter system, we still need to tell doctrine to use our CustomPostgreSQLSchemaManager. Looking at doctrine dbal documentation, it tells us that we need to set a custom SchemaManagerFactory.

You can follow the doctrine documentation, but if you are using the Symfony doctrine-bundle, you can do that from the config/doctrine.yaml file.

Simply add the “schema_manager_factory” config key in your connection configuration with your own factory service, or if you already override the doctrine platform service with your own, we just need to override a method and don’t need a custom SchemaManagerFactory. We can simply use the doctrine DefaultSchemaManagerFactory instead of creating our own.

Note : if you don’t configure the SchemaManagerFactory, with Symfony doctrine-bundle, the SchemaManagerFactory used is the LegacySchemaManagerFactory. This one get the SchemaManager through the doctrine driver, it’s deprecated and will be removed in doctrine/dbal v4. So better use the DefaultSchemaManagerFactory right now 👍

If we go back to our config/doctrine.yaml:

platform_service: Namespace\To\My\CustomPostgreSQLPlatformService
schema_manager_factory: 'doctrine.dbal.default_schema_manager_factory'

Here we use a custom platform service, and use the doctrine default SchemaManagerFactory. Now we just need to add this method in our CustomPostgreSQLPlatformService:

class CustomPostgreSQLPlatformService extends \Doctrine\DBAL\Platforms\PostgreSQLPlatform
{

    public function createSchemaManager(Connection $connection): CustomPostgreSQLSchemaManager
    {
        return new CustomPostgreSQLSchemaManager($connection, $this);
    }

}

Again, extend the platform matching your database type (PostgreSQLPlatform here).

If you want to know how it works, I let you look at the Doctrine\DBAL\Schema\DefaultSchemaManagerFactory class.

Spoiler : it calls the overridden method above from our platform service👆

And that’s it, generating new migrations no longer delete custom indexes.

Ignore custom indexes on doctrine dbal was originally published in Yousign Engineering & Product on Medium, where people are continuing the conversation by highlighting and responding to this story.

We all have this fear, to break our production because of these upgrades. A lot of companies postpone them because of this, until they face the inevitable.

Waiting and running a deprecated version of PHP (or any other dependencies of your code) is way worse. It can lead to vulnerability exploit, and can severely damage your whole business.

It’s a mandatory step that cannot be neglected. And the longer you wait, the harder it will be to upgrade.

At Yousign, we often face this situation and we wanted to share with you how we deal with major upgrade in PHP 🐘

▶️ Define your upgrade plan

Before starting our upgrade migration, we have to prepare some stuff. Let’s define the steps required for this upgrade to succeed. It depends on how your product works, but in our case we identify some steps, that can be transposed to almost any major dependency upgrade.

📑 Read the upgrading guide

The first thing to do is to read the official upgrading guide.

For Symfony major version upgrade: https://symfony.com/doc/current/setup/upgrade_major.html

If your major upgrade concerns a composer dependency, you can take a look to the matching Github repository, most of them have a UPGRADE.md in the repository root path.

Take a look at the doctrine UPGRADE.md

🪝 Release blockers

We begin the journey of a major version upgrade by identifying blockers, for PHP update we can rely on composer to tell us which dependencies are blocking the targeted update :

# If we want to update PHP to 8.1
composer why-not php 8.1
# If we want to update symfony to 6.2
composer why-not symfony/framework-bundle 6.2

It’s a first step to identify packages that need to be updated.

Occasionally, a blocking package needs to be updated to a new major version, you can apply this guide again, release blockers of your blockers 😅 and roll out the guide.

📩 Update blocking package(s)

Fix each blocking package one by one in a dedicated merge/pull request.

Update if needed the versioning constraint for the package in the composer.json file to be certain you can reach the wanted version (latest is better)

Update the package by doing a :

composer update the-blocking/package --with-dependencies

You can “force” a specific version of a package, if you want to downgrade it, or freeze its version.

You can do it by using the “with” parameter, for example if you intend to fix friendsofphp/php-cs-fixer to 3.13.0 and don’t want the last update (that change too many things in the codebase) :

composer update some/package --with friendsofphp/php-cs-fixer:3.13.0

Note that it should be a temporary workaround, as any future composer update will update the fixed package version.

🩹 Fix deprecation

Find a way to log and fix all deprecation issues related to the part you want to update (PHP, Symfony, Doctrine or anything else) before doing the major upgrade.

Deprecation issues can be printed by launching PHPUnit test, or through the logs. You can see them too in the Symfony profiler page 👍

It’s really important here to have a good test coverage, to catch as many deprecation issues as possible when launching your PHPUnit tests.

🎳 Update all composer dependencies

It can be a tricky part, but if your composer.json versioning constraints are good, it can be good to do a last composer update in a dedicated merge/pull request :

composer update

🔄 Iteration step

Once you think you have released all blockers, you can do the major update. It consists of trying to do the update, and roll out a checklist until everything is ✅.

The changes have to stay on its branch (don’t merge it to main branch, read until the end 🙏)

Each time you have an issue, it’s better to fix it in another dedicated merge request, before doing the main update (if possible).

Once new blockers are fixed, rebase and repeat until everything is green! The goal is to not have a long living branch with a lot of change, which is a pain to maintain (conflicts with main branch, etc).

The more you can fix and merge before doing the major upgrade, the better it is!

🛠️ PHP major update

It consists of updating local env installation, and Docker image if you have one. You need to update the PHP version used in composer.json too. Once done, a composer update should be needed to update packages according to this new PHP version.

🛠️ Dependency major update

Update the wanted dependency related packages.

With symfony, update allsymfony/* packages version constraint to the desired version in the composer.json

You can try to update them with a composer update, then use —dry-run to simulate the update and see if it’s okay or if you have other blocking package(s) (that you should update in a dedicated merge/pull request before doing the main update)

☑️ Checklist example

The checklist really depends on your workflow, tooling, CI/CD, etc. In our case:

• Symfony cache clear run without issues (or Laravel or whatever)
• PHP code style and static analyzers run without issues (PHP-CS-Fixer / PHPStan / Rector / etc)
• Your PHPUnit Testing suite run and is green
• Your existing merge / pull request CI/CD have to be ✅

Check that your observability integration(s) still works well (you can still read logs and metrics).

Test that monolog is still doing his job if you use it, or that your log aggregator properly retrieves your PHP logs in the updated Docker Image, etc.

🏁 Challenge the update

Once the checklist is okay, you can test the update to look for any performance regressions. It can be achieved by launching a dedicated Performance Benchmark.

Actually, the best is to reuse any existing, previously made performance test to have something to compare with.

Another thing to take care of when you often deploy, is zero downtime deployment compliance. A major update can lead to zero downtime issues, for example if you rely on object serialized in a message broker like RabbitMQ. For further information, I let you read this excellent article.

🚀 Release process

Because it’s a major update, we need to take precautions. So, deployment in production needs some additional care.

The major upgrade should be done in a dedicated release, and should include only the major upgrade commit (nothing else!). It’s of course not mandatory if you feel confident enough, but highly recommended.

🗨️ Communication

Communication is crucial here, so once everything is green, have a meeting with stakeholders to agree about a release date. This date should be communicated some days before.

🥵 The release day

First, it’s best to do a dedicated release for your change. Because you upgrade a major version constraint, it’s best to not ship other stuff in same time so in case of issues, you are sure it’s related to what you did, and not another commit.

Then, isolated this release allow you to rollback more easily.

My advice is to start your normal daily release process before. And wait, that all deployments are ended, and that any post deployment tests / tasks are finished.

• Prepare a dedicated release from the just deployed version. It depend on your CD workflow, but you can rely on your hotfix process if it’s easier for you.
• Only ship your major update merge/pull request changes
• Wait the merge/pull request CI/CD pipeline, and follow your deployment flow to the production 🚀 .

🧹 Post deploy tasks

You can now merge your branch into your main branch. Pay attention if it breaks your CI, you can have breaking changes with a commit done between the release and your change.

Keep an eye on metrics and logs after and during deployment, and on production post deployment tests.

Issues may still happen, even with all the care we put to avoid them. Having the proper observability tools, monitoring / alerting system, and the ability to quickly rollback things, is the key. You can consider doing another release, or hotfix if needed too.

Another nice thing to do is to communicate with your backend team about new changes that the major updates bring. It may be the occasion to do a presentation, or a blog post like this one 😅.

New features are always appreciated by developers and allows us to appropriate and learn new ways of coding, and get the opportunity to modernize the codebase.

You can also create tasks to handle new deprecation issues that the major update could bring. Anticipating is the key to ease the next major releases.

🎉 Congratulation!

Deal with major upgrades in PHP was originally published in Yousign Engineering & Product on Medium, where people are continuing the conversation by highlighting and responding to this story.