A sneak peek at the upcoming version 3.0 of Mayan EDMS NG
It started as a hobby. Help keep a project active while its main developer dealt with some real-life issues due to a natural disaster.
My first commit was on February 15, 2018. It was only to upgrade the version of Django used by Mayan EDMS from 1.10.7 to 1.10.8. That was it.
Based on my experience using Mayan EDMS for work, I started looking at the code to see if I could fix some other things. I found myself fixing APIs, writing tests and looking at uncompleted features the main developer had been working on. I was hooked.
Soon after I was joined by Eric Riggs. Eric uses Mayan EDMS as part of his 8–5 job, therefore he was knowledgeable about the code too. With Eric’s help and twelve days since my initial commit, we releasing version 2.8 of our Mayan EDMS fork. We called it Mayan EDMS NG (NG for next generation).
We became more and more thrilled by all the possibilities for improvements. We took the plunge, and we are now working on Mayan EDMS daily.
Two weeks after that initial release and here we are today, about to release a new version with some major improvements and changes.
We are very excited about this upcoming release. Here what’s new. I hope you get you excited too.
Turning Mayan EDMS into a single page app
Historically, Mayan EDMS has steered away from adding too much Javascript in its code. Roberto, the main developer often explains his rationale for this decision. His goal has been to maintain a robust, backend-based page rendering method that will be as future-proof as possible. This approach comes at the cost of some page loading speed, and reduced user interface interactivity.
The Javascript ecosystem is an untamed jungle of competing “standards”. Between JQuery, Angular, React, Ember, CoffeeScript, ES2015, ES2017, Babel, Grunt, Gulp, Bower, Webpack, Vue, Webpack, Rollup, Parcel, among many others, is easy to see why that approach was the correct one at the time. Luckily things have improved thanks to HTML5 and CCS3. The browser can now do many things that often required convoluted Javascript code to achieve.
One of the most common complains about the user interface is that it is slow, heavy, requires users to move around too much. That it requires “too many clicks” to achieve things.
Single Page Applications (SPAs) rewrite the current page dynamically rather than loading the entire page on each click of the mouse. This makes the web application feel and behave more like a desktop application. Because the majority of the styling and Javascript code is loaded only once, there is also the added benefit of less data down the wire. Thus the application becomes lighter and provides a faster response time to user events. Because the style is loaded and interpreted at the beginning, the browser is also able to apply it to the new content faster.
In order to strike a balance with Roberto’s backend heavy philosophy, we stuck to using just HTML5 and jQuery. Aside from two additional jQuery libraries, there are no extra framework dependencies. With the conversion to an SPA, many other petitions for user interface improvements are now possible. We will work on these once the community confirms that all the user interface changes made to create the SPA are working as expected.
Upgrading to Django 1.11
The move to Django 1.11 proved to be a real challenge. Even though Django 1.11 is a minor release, it breaks compatibility and interfaces in several key areas. Among these were templates and form widgets.
Mayan EDMS uses a complex template, form and widget system. The system mimics object-oriented concepts like inheritance at the rendering stage. This allows the more than 300 views to be serviced with just a handful of forms classes and base templates. Testing and auditing all the views and forms after the upgrade was a lot of work. If any member of the Django project is reading this, please adopt stricter version numbering convention, something like Semantic Versioning.
Along with the upgrade to Django 1.11, we fixed many deprecations warning in preparation for an eventual upgrade to Django 2.0.
Notification improvements
In version 2.8 we introduced event notifications. These work by allowing users to subscribe to a particular event like Document Uploads or to an event of a particular document like when an invoice is edited. If these events occur, the user gets a reminder next to the bell icon in the main menu bar. This feature had one important limitation. The notification reminder would only update if the main page updated. That meant that if the user didn’t interact with the user interface they would not get a notification reminder.
We implemented a simple mechanism that checks for notifications periodically without user intervention. We avoided the use of webworkers, websockets and push notifications. The result is almost instant notifications without user interaction and without adding any extra dependencies.
Resources and services base API
The API was previously organized with the same core philosophy of Mayan EDMS. The philosophy of separation of concerns and compartmentalization. This meant that the API URLs is first divided by app, then by object type.
For example:
The URL to get a list of documents was:
/documents/documents
The URL to get the list of tags for a document was:
/tags/document/<ID>/tags
The URLs for each app were isolated in their own namespaces, even if those URLs were meant to add data about a related object like a document. The API has been updated be based around resources and services. Resources like documents, tags, cabinets are now top-level entries. The same goes for services like authentication.
This means that the URL to get a list of documents is now:
/documents/
And the URL to get the list of tags for a document is now:
/documents/<ID>/tags
Migration squashing
Database base migrations are a must to ensure your data remains coherent after each upgrade. They need to be committed sequentially to simulate the evolution of the database structures over time. This is true even for new installations with no existing data. As Mayan EDMS matures and adds features it also adds migrations. Each of these migration add up to the amount of times it takes to install Mayan EDMS.
Django recently added support for merging migrations. Because migrations are sometimes interdependencies between each other, this is a delicate procedure. Instead of getting wild with merges, we used the feature in moderation. We merged the migrations that have proven to be stable over the course of months or years. Additionally, these merged or “squashed” migrations are only used for new installations. Any upgrade of an existing installation will use the normal migrations. After some time, if not issues are reported with the merged migrations, the individual migrations will be removed and the merged migrations will take their place.
These are the apps for which migrations were merged:
- acls (ACLs): 1 and 2
- checkouts (Checkouts): 1 and 2
- common (Not shown in UI): 1 to 8
- converter (Not shown in UI): from 1 to 12
- django_gpg (Keys): from 1 to 6
- document_indexing (Indexing): from 1 to 5
- document_parsing (Parsing, “Content”): 1 and 2
- document_signatures (Signatures): from 1 to 6
- document_states (Workflows): 1 and 2
- dynamic_search (Search): from 1 to 3
- events (Event): from 1 to 4
- linking (Smart links): from 1 to 5
- lock_manager (Not shown in UI): 1 and 2
- mailer (Mailing): from 1 to 5
- metadata (Metadata): from 1 to 8
- motd (Message of the day): from 1 to 5
- permissions (Permissions): from 1 to 3
- sources (Sources): from 1 to 16
Dependencies upgrades
Most of the requirements, dependencies and libraries were upgraded to their latest release.
- Pillow: 5.0.0
- django-activity-stream: 0.6.5
- django-compressor: 2.2
- django-cors-headers: 2.2.0
- django-formtools: 2.1
- django-qsstats-magic: 1.0.0
- django-stronghold: 0.3.0
- django-suit: 0.2.26
- furl: 1.0.1
- graphviz: 0.8.2
- pyocr: 0.5.1
- python-dateutil: 2.6.1
- python-magic: 0.4.15
- pytz: 2018.3
- sh: 1.12.14
- rest_framework_swagger replaced with drf-yasg: 1.5.0
FancyBox was upgraded to version 3, Font Awesome to version 5, jQuery to version 3.3.1. ajaxForm version 4.2.2, URI.js 1.19.1 and pace 0.7.8 were added as part of the conversion to single page app.
Search syntax
Searching without using a specialized search database is difficult. Most software just take the lazy route and force the user to install a dedicated search engine. Early on we noticed Mayan’s design called for avoiding a separate search engine at the cost of some missing search syntax. The OR and the negative term support is our first attempt at adding special syntax to Mayan’s search code. If we get good feedback, we plan to add more.
By default now, search terms are routed to an “AND” query. That means that a search for:
Tag1 Tag2
will only return documents with both tags attached. To offer the opposite choice we added an “OR” syntax. Searching for:
Tag1 OR Tag2
will return documents with either tag attached.
We also added support for literals terms.
Searching for:
blue car
will return documents with the words “blue” and “car”, even if they are not together. That means getting documents with the phrases “blue sky” and “slow car”. To search for exact terms enclose them in quotes:
“blue car”
This will return only documents with the exact phrase “blue car”.
Running multiple instances of Mayan EDMS
If you've ever tried running two instances of Mayan EDMS, you would have noticed that they both try to create a lock file in the /tmp directory with the same name. Only the first instance will be able to run.
The lock filename needs to be unique to each instance, yet predictable so that the workers of an instance can also access the same lock file.
We solved this issues by using a hexadecimal hash representation of the installation’s unique SECRET_KEY setting. The use of a hash makes reversing the value in order to obtain the SECRET_KEY impossible for all practical purposes.
Display resolution settings
During the template work required to upgrade the version of Django, we noticed that display sizes (display, preview, thumbnail) were specified as a string that included the horizontal and the vertical resolution separated by the character “x”. Using an “x” character to separate resolution elements is not standard and didn't felt as well planned as the rest of the project. The converter in Mayan EDMS also allows specifying only the horizontal resolution. This design created some ambiguities in the code. We decided to split the settings for specifying resolutions into two settings for each size. One setting for horizontal resolution and another for vertical resolution.
The settings are now:
DOCUMENTS_DISPLAY_WIDTH, DOCUMENTS_DISPLAY_HEIGHT, DOCUMENTS_PREVIEW_WIDTH, DOCUMENTS_PREVIEW_HEIGHT, DOCUMENTS_PRINT_WIDTH, DOCUMENTS_PRINT_HEIGHT, DOCUMENTS_THUMBNAIL_WIDTH, DOCUMENTS_THUMBNAIL_HEIGHT
Other changes worth mentioning
- Fix permission filtering when performing document page searching
- base.js was splitted into mayan_app.js, mayan_image.js, and partial_navigation.js.
- Cabinet detail view pagination was fixed.
- Improve permission handling in the workflow app.
- The checkedout detail view permission is now required for the checked out document detail API view.
- Add missing services for the checkout API.
- Fix existing checkout APIs.
- Update API views and serializers for the latest Django REST framework version.
- Update to the latest version the packages for building, development, documentation and testing.
- Add statistics script to produce a report of the views, APIs and test for each app.
- Merge base64 filename patch from Cornelius Ludmann.
- SearchModel return interface changed. The class no longer returns the result_set value. Use the queryset returned instead.
- Remove the unused scrollable_content internal feature.
- Remove unused animate.css package.
- Add the MERC specifying javascript library usage.
- Documents without at least a version are not scanned for duplicates.
- Convert document thumbnails, preview, image preview and staging files to template base widgets.
- Unify all document widgets.
- Printed pages are now full width.
- Move the invalid document markup to a separate HTML template.
- Move transfomations to their own module.
- Split documents.tests.test_views into base.py, test_deleted_document_views.py,
test_document_page_views.py, test_document_type_views.py, test_document_version_views.py,
test_document_views.py, test_duplicated_document_views.py - Sort smart links by label.
- Rename the internal name of the document type permissions namespace. Existing permissions will need to be updated.
- Removed redundant permissions checks.
That’s it for now, but it’s not over. There are many more things we want to add. Head over to the “pending task” documentation chapter to see what the future holds.
