Ever wished there was a professional version of Mayan EDMS?
Yeah, we too. Sadly there isn't, so we decided to make our own.
Introducing PAPERATTOR!
Read the entire post to find out more and be sure to sign up for the newsletter, you’ll be glad you did.
What is the difference between a free and a professional version? The free version is built and used by enthusiasts of the product. It has the features that are important to them. Those features might not be the same ones you need to meet your professional bottom line.
The other big difference is support. With a free version, your questions will be answered in a forum. But no one is going to stay late on Friday night to debug the issue critical to your revenue stream.
So let’s talk features. For this first release, we focused on three areas: Security, user interface and user experience, and document retrieval.
Security
When it comes to document security there is transmission security and then there is storage security. For transmission, there’s already SSL, SSH, and other protocols widely available. You send your documents in clear text, the transmission protocol encapsulates and encrypts it, and then the document is decrypted at the destination. But your document is still stored in its original format. For storage security, nothing beats disk binary file encryption.
Many storage providers support the encryption of files. There is just one problem: You store the encryption key together with the encrypted files. That is like keeping the key to your front door under the welcome mat. Not very secure.
There is no shortage of incidents in which cloud-stored data turns out to have been exposed for weeks, months, and even years. One report found the number of companies having their cloud data openly exposed to be as high as 53%.
Wouldn't it be nice if you could send your documents already encrypted to your storage provider?
That is just what we added. We created what we call: a pass-through encryption storage driver. Pass-through because on the front end the driver exposes the same interface of a regular storage driver, the application doesn't notice the difference. But the driver doesn't actually handle the storing of the documents only the encryption. Once encrypted, the driver hands the document to the next storage driver. This next driver does the actual storing of documents and could be the default filesystem driver for a local disk or an S3 type object store driver for remote location. This gives you the benefit of having your documents encrypted while continuing to use your existing storage setup, be it local or remote, doesn't make any difference.
Together with the driver we also added management utilities to encrypt all your existing documents. This way you can start using the encryption driver even if you have an existing collection documents. Likewise, we also added a decryption utility to revert all documents to their original state if you don’t want to continue using the encryption driver. Either way, you are covered.
The encryption driver is also smart, it tracks which documents are encrypted and will protect against trying to re-encrypt an encrypted document, a practice that in other systems make the document unrecoverable. The same protection is applied when trying to decrypt a document more than once.
Since this is a pluggable driver, it is compatible will any app that stores files. That means that you can use this driver to encrypt your documents, the document signatures and even the document cache for the ultimate security setup.
User interface and User experience
Mayan EDMS is powerful. It has many features and many options for those features. That multitude of setups means Mayan EDMS can be configured for many different kinds of scenarios and deployments. Having so many options is also one the pain points for many users.
Learning to navigate the user interface of Mayan EDMS take time. The creator of Mayan EDMS often apologizes for the interface. He acknowledges that interface design is not his strong point. Nothing to worry about because it is one of our strong points!
We divided our planned overhaul of the user interface into three stages. The first stage was to modernize the existing technologies of the interface while preserving the appearance and experience. This stage will work as the scaffolding for the entire process. This first stage has been completed with the release of Mayan EDMS NG 3.0. Mayan EDMS NG 3.0 features a completely new user interface rendering based on partial templates. The interface now works as a single page application. All visual changes are now triggered and loaded via Javascript events.
The second stage of the overhaul is a visual one. The existing interface is adequate but not up to the standard the profession document sector is expecting. This creates an emotional barrier of entry.
The existing interface template system is well designed from the programming point of view but is limited in the kinds of widgets and controls it can display. There are no perfect templates, so we picked the one with the least limitations for our case. We settled with PatternFly.
The goal for this stage is to replace the templating system, improve the visual components of Mayan EDMS while keeping the views workflows almost intact. Changing the user experience and the way things get done will come later in the third stage.
The existing template system is horizontal based. Elements’ widths are deterministic. We took things further and started making some elements heights deterministic too. For example, the sidebar in the document preview window is too tall and causes the scroll bar to appear even on small documents. The same problem is exacerbated in small devices. To alleviate this we moved the document view sidebar to the top and turned into a toolbar. This freed up three columns that are now used to increase the viewable area of a document. Having the scrollbar at the edge of the page is also more intuitive and visually pleasing.
The conversion to the new template system also improved the layout for very small devices. By converting the sidebar into a top toolbar and hiding the actions dropdown in a “kebab”, all the visual data of the document as well as the usability of the features now fit on a single screen pane. Scrolling is only needed to view additional pages, not to access basic features.
The existing template relied on text truncation at the backend. This worked most of the time but stopped being reliable once the frontend fonts and word spacing were changed. To further control the horizontal harmony of the new template, hidden overflows were added in areas controlled by back-end data.
CSS has a feature that allows truncating text and we put it to good use. Together with white space handling and better word breaks we managed to create a solution that works for all use cases. Long file names or user names no longer break the user interface.
Document retrieval
Mayan EDMS is very at organizing, marking and searching documents. This means that you can find a document very easily if it is related to the data (tags, cabinets, indexes, etc) you are using as the retrieval criteria. But what if you don’t know the data you need to search for a document? Sound counterintuitive but happens all the time. You might need to find documents which have not yet been tagged. You might need documents that are not in any cabinet. Or want to list documents that don’t have OCR data. How do you search for these documents?
For these situations, we added Filters. Mayan EDMS already had a similar feature called Data Filters. The problem with Data Filters is that they need to be programmed in Python, they are not editable via the user interface, and are hard-coded into the system. Our solution is completely managed via the user interface and controlled with permissions. Users create filters by entering selection or exclusion criteria for different properties of documents. The criteria for the filters can span different aspect of the document. For example, you can create a filter to display which invoices starting with the sequence “001”, have already been processed by the OCR, but have not been tagged yet.
With Filters, you can translate your business processes into actionable objects in the system. Filters are as powerful as Indexes but without their design complexity and are much faster since they are translated into queries.
Project relationship
What is Paperattor and how it relates to Mayan EDMS and Mayan EDMS NG?
Paperattor will be a commercial product. It will fill the gap in the Mayan EDMS ecosystem for a professional, polished, supported product. It is built using Mayan EDMS NG as a platform with additional close sourced apps. We plan to merge features periodically from Paperattor to Mayan EDMS NG. This way both project benefit from each other’s advances.
Future work
That’s whats we been busy doing but there are many more things we have in store for after the initial debut of the project.
- Start stage 3 of the user interface and user experience process. Our goal is to allow managing document properties without leaving the document preview window. We plan to achieve this using panels or docks. You will be able to add and remove tags to a document using a side panel while remaining on the main document view. You can pin panels and docks to customize the interface to your liking.
- We have designed the implementation for a smart document cache. This cache system will auto prune old entries to keep its size in check. Vital to keep under the block storage quota of cloud providers.
- Remake the settings system to allow changing settings from the user interface. No more editing local.py and no more restarting the server!
- We just finished the design for zonal OCR, another must-have feature in the commercial space!
We are putting the finishing touches on the initial version and will open the selection process for private beta testing soon. If you are interested in testing the product or just want to follow up on the developments, sign up for the newsletter. It is low volume list directly tied to the product announcements only.
With Mayan EDMS NG 3.0 we proved we could take on maintaining a massive project like Mayan EDMS. With PAPERATTOR will prove we can also take it to new heights.
