Checking technical options, we discovered…
In the past few weeks we, a great mix of 30+ people of Greenpeace volunteers, staff and I have been exploring the technical possibilities for Planet 4 as part of the Discovery phase.
The Technical Investigation is now complete. We had tons of inputs and here’s a glance at what we have learnt.
Technical discovery scope
The technical discovery did not have the objective to solve any technical problem, nor to answer any question. Actually, our aim was exactly the opposite: raise a maximum of questions, create debates and invite people to suggest options and solutions.
We only had a certitude: WordPress, selected at an early stage for a series of reasons, the most important probably being the size of its community. There are many nice open source CMS on the market and we could talk for hours, maybe even days, about why one is better than another, but well… WordPress has many qualities; its community is both giant, amazing and dynamic, and we love it. Not to mention, WordPress powers 25% of all the websites of the internet.
Starting from there, we defined the following scope for our technical discovery quest:
- A first set of options for key features: multi-domains and forms management, translations, custom fields, SEO. Even without full functional specifications, some of these key features are functionalities that we are pretty sure we will need on Planet 4, because we currently need them all on Planet 3. Our objective was to explore what are the different options available and how we may go in implementing them.
- A feasibility study for systems’ integrations: There are currently a lot of engagement systems adopted throughout the Greenpeace ecosystem: Greenwire, GreenpeaceX, CRMs, analytics platforms and so on. We explored which ones are the most used and started imagining strategies for their integration.
- Technical key performance indicators: What we can’t measure, we can’t improve! The objective here was not to define the final KPIs. We mainly wanted to define at a high level what is important for us to measure, and what we want to keep our eyes on. The detailed list of KPIs will be defined during the concept phase.
- Infrastructure and hosting options: A lot has happened in that field also called “sysops” in the last few years, with new technologies such as Docker and Kubernetes gaining serious traction. In this particular stream, we wanted to explore and list technologies that can help us deploy and scale faster, as well as define how we will measure key performance indicators.
- Development processes: Here, the aim was to explore what high level processes we would like to implement, including development cycles, testing strategy, continuous integration, and how to integrate these processes with WordPress. The result was quite impressive.
- Content migration: Planet 4 is not only about developing a new system. We needed to start talking about strategies on how to effectively and safely migrate existing content to Planet 4.
Some of the most interesting open questions
Our own distribution of WordPress?
That’s probably the most exciting part of our discussions. We agreed that we need to:
- Use a version of WordPress fully compatible with our tools and software ecosystem.
- Use WordPress modules that have been reviewed, known to work well with the other selected modules, and proven secure.
- Make sure that no vulnerabilities are introduced while updating community modules. Security is one of our main focus, and we want to avoid bad scenarios such as this one.
- Modify easily community modules if needed, without having to patch their code. We want to ensure an easy forward compatibility.
- Be able to contribute to the WordPress community. The new features we will implement and the automated tests we will develop should be shared back as much as possible with the module developers and wider WordPress community.
One solution could be the development / maintaining of Greenpeace’s own distribution of WordPress, as shown below:
WordPress would be forked, if necessary (to be determined), and kept up-to-date with the master branch. Community modules would be audited, selected with a democratic process and forked in our own modules’ repository, to ensure a constant level of security and that no updates can introduce vulnerabilities or breaking changes. A continuous integration process coupled to an automated test suite (unit tests, functional tests) would take care of constantly ensuring that the build is stable and that it can be deployed safely.
This is just an idea in which we’ll need to dig deeper during the Concept phase. Its complexity and feasibility will need to be assessed, and we are quite sure that it’s not going to be as easy as it sounds. Exciting isn’t it?
A multi-domain approach?
Multi-domain vs one site per domain? That’s the million-dollar question! Given that each individual Greenpeace website will have its own specificities but will also implement features shared with others. Putting all the websites in the same instance would make the maintenance easier, while splitting the websites in multiple WordPress instances would provide more freedom for local customization. Each solution has its pros and cons, and we couldn’t find two people to agree about what is the best approach. Choosing one strategy over the other will massively impact the overall architecture of Planet 4, that’s why the answer to this question is at the top of our list.
How to approach Globalisation?
There are various ways of handling this, and we should be cautious with multilingual compatibility issues between WordPress modules. We learned that the translation process should probably be integrated inside our editing workflow, for which some custom developments might be required. This could be a good opportunity to share our work with the WordPress community and improve inter-modules collaboration.
A default master theme and a series of child themes?
Theming is at the center of our preoccupations. We confirmed here what we already had in mind. We need to develop theming tools accessible, robust and flexible at the same time. Accessible because we want to be an inclusive community. Robust because altering the themes shouldn’t break easily their responsiveness or accessibility. Flexible because they should be easily extensible to match all different needs. The basic idea is to create a parent theme containing all the fully tested basic components, templates and tools, ready to use as-is when needed. This parent theme should be easy to extend to match specific NROs and campaigns’ needs.
What kind of roles and permissions?
As in Planet 3, Planet 4 will need a granular roles and permissions management system. The hundreds of editors, translators and content managers who will create the amazing P4 content will need to be given the editorial permissions allowing them to work fast and access the WordPress features they need. It was pretty clear to us that the investigation on this part will have to be pushed much further in the next phase, and that existing available modules might not be enough to cover all our needs. Maybe another good opportunity for us to contribute to the community with a new module?
How to approach 3rd party integrations?
Planet 4’s main goal is to “drive people into action, be relevant and innovate”, not simply putting content on a web page. Planet 4 has to integrate with external platforms, but also give the possibility to be integrated into other platforms or websites. For this, we imagined to develop a Restful API for the most important content types, and a system of javascript-driven widget that will present data in other systems by consuming their own APIs.
We also had a first look at several options for tighter integration with specific system such as the globally-used one for both petition and donation pages, which is already using WordPress modules.
How to approach content migration and archiving?
We figured that we have a huge amount of content across all Greenpeace websites, and that probably only a small fragment of it is worth migrating. Our approach is to only migrate content that is actually relevant, and for the rest we will simply archive it. The archived content will remain available in case of need, but won’t be part of the Planet 4 primary stack.
A good solution to archive old content while minimizing maintenance costs could be the transformation of existing Planet 3 (or other) websites into fully static websites, using crawlers for instance. In the next phase we will need to define the selection process for both content migration and archiving.
Talking about content migration, we have some simple content types, such as blog posts, for which it should be easy to automate the migration process.
But when coming to more complex content types, such as campaign pages, (which typically include custom styles and javascript) an automation would simply be impossible. In the Concept phase, we will also need to explore how we can approach the migration of all complex types of content.
How to select which modules to use?
Modules selection and pre-validation: the modules we will use need to reach a certain level of quality. We cannot afford to implement modules with hidden defects, that can’t scale or that are not actively maintained. To tackle this, we figured that we will need to implement a rigorous validation and selection process to define what modules are safe to use. This validation process will be automated as much as possible, and could be based on the following criterias.
Modules security: security is a primary concern for us, and a critical aspect of our websites. To ensure a constant and optimal level of security throughout our modules (both community and custom), we need to design a continuous security auditing process which will ensure that most of the potential risks and vulnerabilities are detected upstream. To do this job, we naturally considered the use of open source tools such as the OWASP tools (zap, for example), or vulnerability scanner like PHP vulnerability hunter. In the upcoming weeks, we are planning to consolidate a list of tools, test them in depth and define a strong implementation process.
How should our development process look like?
Agile / Scrum: Agile / scrum development processes are already implemented in some of our projects such as Greenwire, and we are absolutely fond of it. They have proven their efficiency and we have proven our capability to implement them successfully. This is why we want to make Agile / Scrum a pillar of our development processes and follow an open user-centered design.
What is our test strategy?
To ensure the stability of our platform, we are considering to implement 4 level of tests which will be part of a continuous integration process:
- Unit tests can be used to test our PHP and JS custom code at a function level. Our aim would ideally be to have a code coverage close to 100%, at least for all the custom code that we will produce. As much as possible, we also want to contribute to community modules by increasing their code coverage whenever necessary.
- Functional tests can be used (in combination with selenium for instance) to test the interface by mimicking user interactions. For all Planet 4 main features, we would like to implement a functional test suite which results could be shared openly through public test servers such as saucelabs who give free resources to open source projects.
- Visual regression tests can be used to test the visual parts of Planet 4, in the form of tests running on a default style guide.
- Manual tests can be run by our QA team at each release to test whatever couldn’t be tested automatically and provide formal user acceptance.
What will hosting look like?
- Hosting providers — Since we implemented Planet 3 the hosting solutions available in the market have evolved quite a bit and there are now a few exciting options we would like to test for Planet 4. We are committed to work exclusively with green hosting providers. Google Cloud Platform is one of them, and the next phase will help us to do a proper comparison of several solutions, which will include flexibility, costs impact and stress resistance.
- Performance and caching — There are many possible setups, some based on CDNs, some on reverse caching proxys, and even some very interesting WordPress modules that could help us with that. We will have to test the different solutions available, and answer a few questions such as how we are going to handle pages that are dynamic by nature (comments, for instance).
- Containers (Docker / Puppet / Kubernetes) — Finally, we couldn’t investigate the infrastructure possibilities without considering the use of modern container technologies such as Docker, or orchestration tools such as Kubernetes. During the Concept phase, we will dig deeply into the possibilities offered with the challenge to find the right balance between using the right technologies for our needs while avoid over-engineering our infrastructure and tools.
Where do we go from there?
That sums up the Discovery phase for the Technical Investigation track. There are thousands of other points that are worth discussing but we couldn’t mention everything here, so if you want to have more details feel free to check the technical discovery document directly. Let us know what you think, or if you have any other suggestions or recommendations! You can drop comments in the section below, or you can reach me out directly at kevin.muller@greenpeace.org
Next is Phase 2 : the Concept phase in which we will dig more deeply into each one of these points and gather enough information to make data-driven decisions. Keep an eye on it, and keep giving us your feedback!
I would like to thank all the awesome people who participated in the technical workgroup and provided hundreds of inputs out of which the possible ingredients for the Planet 4 recipe have been shortlisted, and most importantly helped us identify future bottlenecks. We are now ready to move on with the positive feeling that we are on the right track, thanks to you.
Special thanks to: Renaud, Anselm, Michiel, Ryan, Jan, Konstantinos, Danny, Kritsana, Osvaldo, Alessandro, Manoj, Manuel, Asmund for your thousands of valuable inputs. The Greenpeace infrastructure team : Bart, Larry, Alen for pointing out possible improvements. The Planet 4 core team: Kelli, Laura, Lilian, Luca, Natasja, and Remy; and Nadav, Cody, Michael and Pasquale for your support and hard work.
