Mayan EDMS version 3.3 released

The team would like to send a special thank you to the Berkeley County (SC) Government, personnel, and to David Kornahrens Jr, Chief Information Officer and Assistant to Supervisor for their big support.

Click here to read more about this collaboration and how you too can be a part of Mayan’s future.

Authentication

A new settings was added to allow disabling the password reset link in the login form. This can be done for security issues or if the authentication backend doesn’t support it, such as when using LDAP for authentication. The setting is named AUTHENTICATION_DISABLE_PASSWORD_RESET and defaults to false.

Checkouts

Support was added for checking in and out multiple documents at a time.

Compressed files

Zip files only support UTF-8 and CP437 encoding for filenames. This is not strongly enforced and some software products create Zip files with other text encodings. The ZipArchive class was updated to work with badly encoded filenames instead of raising an error.

Converter

More PDF versions are now supported.

Better page count detection for PDF and for obscure JPEG sub formats.

The image converter system was updated to support image layers. The first use of this new capacity is the redactions app.

A new option was added to the CONVERTER_GRAPHICS_BACKEND_ARGUMENTS setting to allow passing a maximum image pixel count to the Pillow library. The entry is called pillow_maximum_image_pixels and defaults to 89478485. This entry is used to increase the threshold upon which Pillow with trigger the Decompression bomb protection. (https://pillow.readthedocs.io/en/5.1.x/releasenotes/5.0.0.html#decompression-bombs-now-raise-exceptions)

Use this setting to allow uploading very large image files, such as maps or map tiles.

Database

Additional transaction handling was added to several apps to increase the protection against data corruption even on catastrophic infrastructure events.

Dashboard

The dashboard app was updated to automatically switch numeric quantities to their local format based on the user’s selected language.

Dependencies

The installjavascript command was renamed to installdependencies to better reflect is new, more encompassing function.

The version of Django used was updated to the last micro release of 1.11.28.

The way new version are checked was improved and now uses SSL for improved validation.

The way the default paths of binaries is calculated was improved to provided better defaults for FreeBSD and MacOS.

Two new commands were added to aid in batch script integrations. These are the showversion and checkversion commands. showversion will output Mayan’s version to the command line and checkversion will check for if there is new published version of Mayan.

Documents

The Dropzone widget is now used for the new version upload form. This allows the faster paradigm of dragging and dropping files when uploading a new version of a document.

A new event was added to capture when a document is sent to the trash.

The document tool usage permission was added to the ACL system and can now be granted for specific document types or individual documents as opposed as having to be granted for the entire system.

The internal hard-coded constant STUB_EXPIRATION_INTERVAL is now a user setting named DOCUMENTS_STUB_EXPIRATION_INTERVAL. This setting defaults to the previous value of 24 hours to preserve existing behavior.

Documentations

A new sitemap is automatically generated and submitted to search engines. This results in better results when using public search engines to look up for topics.

The documentation upload script was updated to delete older versions of the documentation when uploading a new one. This solves the issue of search engines returning results for old or obsolete documentation.

New content was added to the FAQ and troubleshooting sections based on the most common topics of the community forum.

Docker

The Docker image received many updates to make it easier to use while also allow it to scale up to handle bigger document counts.

The include Redis server was removed. A separate Redis container needs to be deployed. This improves the memory and CPU usage of the Mayan container and makes this usage more deterministic.

Support was added for PIP proxies. This helps reduce the build time by avoiding the download of many Python packages.

New commands were added to the Docker image. These are: run_celery, run_frontend, run_tests, and run_worker.

By default, a Mayan containers will now run the run_all command and launch frontend and backend tasks. These commands allow starting Mayan containers for specific tasks instead. This help scale up Mayan in a multi container deployment very easily from the same image.

The default Docker images for Redis and PostgreSQL now use the Alpine Linux variants for smaller sizes, faster downloads, and reduced memory usage.

Support was added to the Docker compose file and default installation to set a password for Redis.

Two new environment settings were added: MAYAN_SKIP_CHOWN_ON_STARTUP and MAYAN_DOCKER_WAIT. MAYAN_SKIP_CHOWN_ON_STARTUP is used to skip performing the initial operating system chown command on the media folder when the Docker container starts. The command chown is slow on non native block storage backends and not needed for most object storage backends.

MAYAN_DOCKER_WAIT controls a native implementation of a wait cycle to allow the container to wait until the containers it is dependent on are ready to accept connections.

File cache manager

Previously each app was responsible of managing its temporary files. The file cache manager now handles this in a centralized way. In addition, the file cache manager supports size limits for caches. Once this limit is reached, the file cache manager will automatically delete the oldest entries. This ensure file caches don’t grow infinitely. The document and workflow apps were updated to use the new file caching. The document image cache defaults to 500MB and the workflow preview cache defaults to 50MB. These values can be changed with the DOCUMENTS_CACHE_MAXIMUM_SIZE and WORKFLOW_IMAGE_CACHE_MAXIMUM_SIZE settings respectively.

With the addition of the file cache manager, the settings DOCUMENTS_DISABLE_BASE_IMAGE_CACHE and DOCUMENTS_DISABLE_TRANSFORMED_IMAGE_CACHE are deprecated and will be removed in the future. These settings were added a while back to help control the size of the cache by disabling image cache creation.

File metadata

Support was added for wildcard MIME type associations for the file metadata drivers.

The EXIFTOOL driver was updated to run for all documents regardless of MIME type. Even with non image documents, this driver still provides some useful file information.

Locking improvements were added to the file metadata app to improve concurrency and scalability during heavy loads.

Indexing

An index reset tool was added. This tool erases the entire index instance, and allows user to then rebuild a clean index from scratch. This helps when mistakes are made when updating or experimenting with index templates.

Indexes are now initialized in a predictable way and only once during initialization. This change makes them more reliable and improve re-indexing under heavy loads.

Lock manager

A distributed Redis lock backend was added to the lock manager app. This lock helps keep multiple Mayan EDMS instances synchronized, even if they are running on different hosts. This new lock requires one argument: redis_url.

The redis_url argument must be added to the new LOCK_MANAGER_BACKEND_ARGUMENTS setting.

Example:

LOCK_MANAGER_BACKEND_ARGUMENTS='{"redis_url": "redis://127.0.0.1:6379/2"}'

Usage of this backend is required on multi container or multi hosts deployments, like Kubernetes or Docker Swarm.

Mailing

The size of the password field for mailing profiles was increased to 192 characters.

Metadata

The metadata permission layout was updated. Metadata permissions are now bidirectional. The metadata add, edit, and remove permissions are now required for both the document and the the metadata type in order to add, edit or remove a metadata from a document.

The HTML and API were updated, as well as the document metadata widget to only show metadata types for which the document metadata view permission was been granted.

Mirroring

The mirroring code was updated to support slashes in index node values and document labels. Slashes now replaced with underscores.

Support for duplicate nodes values or documents labels was also added. To handle duplicates, the primary key of the object is appended to the label inside parenthesis.

For example, for two documents named sample.pdf in the index level, and with the respective primary keys 100 and 101, their mirroring entries would appear as sample.pdf(100) and `sample.pdf(101)``.

Permissions

The Role label field size was increased from 64 to 128 characters.

The initialization of the permissions is now done in a deterministic manner and on startup as opposed as on demand.

Python 3

Mayan EDMS is now fully compatible with both Python 2.7 and Python 3. This will be the last version to support Python 2.7.

REST API

The document signatures API was added to sign and verify the signatures of documents.

Additional fields were added to the checkouts API to make referencing and checking in documents easier.

A new API endpoint was added to change the type of document.

The URL layout of the OCR submit endpoint was improved and made more intuitive.

A new setting was added to disable the API documentation. The setting is named REST_API_DISABLE_LINKS and defaults to false.

Redactions

A new app was added that uses the new converter layer system. This new app is the redactions app.

This app allows censoring documents containing private information in a non destructive way. Redactions are applied interactively for a document’s page. Once applied, the area will appear obscured to every other part of the system. This includes the preview and OCR systems. Multiple redactions can be added for each page of a document.

Creating, editing, deleting, and bypassing redactions is controlled by new permissions for each action.

Search

It is now possible to disable the simple search via the new SEARCH_DISABLE_SIMPLE_SEARCH setting. This setting defaults to false.

Settings

Support for setting migrations was added. This feature will reduce the number of manual configuration file updates needed between upgrades.

Support for quoted configuration entries was removed. Quoted settings must now be specified as normal nested dictionaries. Settings affected:

• COMMON_SHARED_STORAGE_ARGUMENTS
• CONVERTER_GRAPHICS_BACKEND_ARGUMENTS
• DOCUMENTS_CACHE_STORAGE_BACKEND_ARGUMENTS
• DOCUMENTS_STORAGE_BACKEND_ARGUMENTS
• FILE_METADATA_DRIVERS_ARGUMENTS
• SIGNATURES_STORAGE_BACKEND_ARGUMENTS

However migrations were added for these settings and most users will not notice any change.

Support was added to allow user-specified location for the configuration file with the CONFIGURATION_FILEPATH and CONFIGURATION_LAST_GOOD_FILEPATH settings. This allows setting the location of the configuration file independently of the MEDIA_ROOT setting.

Multi database configuration is now supported. To specify multiple databases, use the new DATABASES setting. This new setting more closely mirrors the upstream behavior of Django.

Example, as environment variable:

MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'127.0.0.1'}}"

This is the same as passing the environment variables:

MAYAN_DATABASE_ENGINE=django.db.backends.postgresql
MAYAN_DATABASE_NAME=mayan
MAYAN_DATABASE_PASSWORD=mayanuserpass
MAYAN_DATABASE_NAME=mayan
MAYAN_DATABASE_HOST=127.0.0.1

This method allows specifying multiple databases such as when using replication or sharding. It also allows for null password entries such as when using Linux sockets.

Both methods (DATABASE_ and single DATABASES_ prefixed entries) are supported.

New Django SSL settings are now exposed. These are: SECURE_PROXY_SSL_HEADER, USE_X_FORWARDED_HOST, and USE_X_FORWARDED_PORT. These settings allow for very customizable HTTPS setups. However, improper use of these setting can compromise your installation’s security.

The default value of the DATABASE_CONN_MAX_AGE setting was changed to 0. This is Django’s default and safest value. https://docs.djangoproject.com/en/3.0/ref/settings/#conn-max-age

Smart links

Support for ACLs was added to smart links.

The ACL label field size was increased from 96 to 128 characters.

The new templating widget is added to the smart link condition form for easier inspection of model properties when building the template.

Sources

The IMAP source was updated to work using message UID instead of the message position index. This makes the IMAP email source more resilient and its email processing order predictable.

Support was also added for custom IMAP search criteria. By default NOT DELETED is used to process non-deleted messages.

The IMAP source can now also execute custom IMAP STORE commands on processed messages. The command defaults to +FLAGS (\\Deleted).

After processing IMAP message it is now possible to specify a destination mailbox. This allows keeping processed email instead of just deleting them.

The IMAP expunge command can now be turned on or off.

The Source label field size was increased from 64 to 128 characters.

The way error messages during interactive upload is composed has been improved.

The staging folder API is now feature complete. The staging folder API was expanded to allow creating, editing, and deleting staging folders.

The staging folder file sub resource endpoint now has an upload URL. Doing a POST to this URL uploads that staging folder file and creates a new document.

Task manager

The package djcelery was replaced by the package django-celery-beat.

Celery was updated to version 4.3.0. This changes some settings:

BROKER_URL to CELERY_BROKER_URL

CELERY_ALWAYS_EAGER to CELERY_TASK_ALWAYS_EAGER

The task manager was switched from librabbitmq to py-amqp. This library is used to connect to RabbitMQ and is used as the message broker for the background tasks. Although Celery’s official documentation still lists librabbitmq, maintainers are recommending users to move to py-amqp.

Template sandbox

The template sandbox add was added to help administrators when writing templates for the different automation apps like indexes, smart links, web links, and the workflow apps. The sandbox is available as a tab entry for each document.

Additionally, the template sandbox app also provides a property navigation widget that displays all the properties available of the object for which the template is being tested against. This makes writing templates much easier and faster by providing both, interactive testing of template markup and quick documentation of the properties available.

User interface

A new vertical main menu was added. The previous main menu is now split into two menus. One located on the left side for document related links and a new top bar menu for system and user links. The vertical main menu remain open even when clicking on items for faster access. Upon a browser refresh, the menu will also restore its state to match the selected view.

The icon composition system was expanded to support icon shadows.

The Select2 widget is now used for the document type selection form. Autocomplete is enabled for this usage.

Support was added to display help texts for view columns. By default the help text of the corresponding model field is used and shown as a tool tip.

Is is now possible to change the system messages position using the new DEFAULT_MESSAGE_POSITION setting. The default value is top-right.

Mobile responsive improvements were added making the interface more usable on small screen. Support for touch zoom was also added.

The Chinese translation was been fixed. The Chinese translation locale was updated from zh to zh-Hans.

The Portuguese translation was also been fixed. The “Tags” text is now correctly translated. The issue was caused not by a code error but by the way translation are prioritized by Django. Translations for apps defined first in the app list setting are prioritized. Since several Django apps form the basis of some Mayan functionality, they are defined first. They translation therefore are prioritized over translations in Mayan’s apps. In this case the word “Tags” (and a few others) were already translated by Django’s admindocs app, causing Mayan’s translation for the same word to be ignored. This has been resolved with a custom app loader until this is properly fixed upstream by Django.

Changed the gunicorn worker class to synchronous. This change was made in the Docker image and is now the worker class for the direct deployment too. Update the supervisord config file to activate this change. This solves spurious HTTP timeout errors caused by Celery’s new Redis interface not working with threaded serverlets.

Many help texts and were added to give users more hints without having to browse the documentation.

The list toolbar is now “sticky” and will remain at the top of the view when scrolling down. This new toolbar encloses the multi-item action menu and the pagination controls.

The warning messages template of the MissingItem class was finished. This class will warn users via an on screen alert if a crucial element is missing. Situations like no document types or no document sources will not go unnoticed now.

The template to calculate item and page count was optimized to used database queries. This results in faster page display under heavy loads.

Web links

The new web links app allows creating links from documents to external resources. These links are defined using the template language. The variables in the template will be substituted with document values and a final HTTP link generated. Web links provide a simple and fast to integrate Mayan EDMS with external systems.

Workflows

The workflow preview was improved to provide bigger, more readable previews. The workflow state actions are now included in the image preview.

Support was added for workflow context. Workflow context allows adding variable and values to a running workflow. These variables can then be used by the different workflow state actions.

Fields support was added to the workflow transitions. These allow adding extra fields to capture user input. The value of the extra fields is added to the workflow context and it is available to the workflow state actions.

A new workflow action to send automated emails was added. The message and destination of the action can be static or taken from the workflow context.

The HTTP POST action received several updates. The first, adds authentication and headers support. The timeout field now supports template for dynamic timeouts. It also now supports integers, floats, or empty values.

Another workflow action addition is the document sign action. This action will perform automatic detached or embedded signing of a document.

A button was added to the workflows to launch a specific one for existing documents. This is useful when changes are made that require triggering the initial state of existing documents.

The workflow app navigation was improved with adding a new links to jump between commonly used views of the app.

Removals

Database conversion. Reason for removal: The database conversions support provided by this feature (SQLite to PostgreSQL) was being confused with database migrations and upgrades. Database upgrades are the responsibility of the app and the framework. Database conversions however are not the responsibility of the app (Mayan), they are the responsibility of the framework. Database conversion is outside the scope of what Mayan does but we added the code, management command, instructions and testing setup to provide this to our users until the framework (Django) decided to add this themselves (like they did with migrations). Continued confusion about the purpose of the feature and confusion about how errors with this feature were a reflection of the code quality of Mayan necessitated the removal of the database conversion feature.

Task inspection was removed from task manager app. This is now provided by the Celery Flower project.

The library DjangoDownloadView was removed. The code to support all downloads (document, version, keys, signatures) was been implement natively using a modified port of Django 2.2 FileResponse.

Support for Magnum CI and Travis CI was removed. Only GitLabCI is supported.

These are the biggest features added but there many other small improvements and additions in many other areas. The release notes detailing all of these can be found in here: https://docs.mayan-edms.com/releases/3.3.html