Released: June 13, 2019
Work on version 4.0 continues along. Version 4.0 brings so many changes that it missed its release schedule. Therefore we decided to release an interim version, this one. This version includes bug fixes and back-ports many improvements already in the 4.0 development version. It also helps bridge the gap between the internal code and interfaces between the 3.x and 4.x code branches. The minor increment means that user-facing backward incompatible changes are minimal. Most of the changes are internal and of notice for developers.
That doesn't mean there are no changes important to users. This release brings about one third (1/3) of the user interface changes that have already landed in the 4.0 version.
Besides the usual template and style tweaks, this version adds a new contextual navigation bar. This bar is a hybrid between a list menu bar and a sidebar menu. Icons in this menu object will show in the object list view and when the object is the main object in the template. This reduces substantial mouse travel and clicks, as most views of the object now remain "open" in the sidebar instead of being hidden inside the "Actions" drop down.
The "Actions" drop-down will now divide the available actions links depending on the menu that defines them ("action", "secondary") and the object that they act upon. This is useful on views that can display more than one object at a time such as the setup views. During the workflow setup view, now the action links will be split between action for the workflow, for the state, and for the transition making setup navigation much faster and requiring less mental effort as the user no longer need to remember which link affects which view object.
Another big user interface change that landed on this version is the ability to sort lists of objects by column fields. Sortable fields will have their column heading displayed as an HTML link. Clicking on a column heading link will sort the list by the values of the column. Clicking the heading again will invert the sort order. A small arrow icon will show the sort order.
A new app was added to the core. The app is named "File metadata" and allows extracting file information. The app includes a driver by default to extract EXIF field information. This file field includes camera information for photos, authors for office documents and PDFs and other information that can now be used to search or index documents. This app was previously a separate app called Mayan-EXIF (https://gitlab.com/mayan-edms/exif). The app was generalized and is now part of the core group of apps.
This is a "minor" release only in version number scheme since it includes a big number of code changes. The minor number increase means that it will be as backward compatible with version 3.1.10 as possible.
The API documentation generation code was moved from the root module where it was out of place to the REST API app. The published URLs however remain the same and backwards compatible.
Dependencies app: A new app was added to handle all dependencies. Previously, the code to handle JavaScript dependencies and license text collection, was contained in the common app. This new app, called "dependencies" now handles both tasks. In addition, it provides checks for binary dependencies. This app's main view will allow users to know which dependencies are not being recognized and help debug installation issues faster and easier. The app supports Python, JavaScript, and binary dependencies.
The JavaScript library download and installation code was updated to provide faster hash verification using block hashing. During tests the this change cut verification time to just 28% of the previous time. Additionally, the JavaScript downloader can now detect existing installations of JavaScript libraries and skip them for even faster upgrade or startup times.
With the exception of the jQuery.ScrollView library there are no JavaScript libraries of packages in the git repository of the code anymore. All JavaScript are now downloaded at install time.
The dependencies app also has the ability to introspect packages and extract copyright and license information. This removes the need to include such texts explicitly in the code for all packages.
Dashboard app: All the code to defined and generate the dashboard was moved from the common app into its own app called "dashboards". The functionality remains the same but along with the formalized interface, the move will allow future planned improvements to the dashboard functionality to be rolled out. While only one dashboard will still be displayed, it is possible to define in code more than one dashboard using the Dashboard class.
Autoadmin app: The autoadmin app which is in charge of creating the initial admin user after the installation, has been included in the core. This app is made by the same author of Mayan EDMS and at one time in the past was part of the core apps.
File metadata: Allows extracting file information. The app includes a driver by default to extract EXIF field information (https://en.wikipedia.org/wiki/Exif). This is mostly multimedia information but the default driver can also extract information from authorship from PDF and office files. The information extracted by this app can be used to search or index documents.
Internally many interfaces and calls have been deprecated in other to move the code closer to 4.0's standard. The only feature deprecated on the user facing side is the convertdb management command. This command was added to allow existing installation using SQLite as the database manager to convert their database to one of the recommended database managers.
After many reports the consensus was reached that this functionality is not meant to be provided in the project. Software projects have little or no control of the aspects upon which they rely. Framework, environment, platform, OS, databases are such examples.
Database conversion is a task best suited for operations oriented software and professional and no Mayan's end users. For these reason the database conversion command has been deprecated and will be removed in the next major version.
Docker image updated to use Debian 9.8.
The size of the Docker image was reduced from 1.25 GB to 1.09 GB.
Support was added for setting the Docker container user's UID and GUID.
The Docker image now includes all static media like JavaScript libraries and themes. This make the initial startup of the image faster as only the database migration is execute and the restart of the image almost instant.
The events system received many updates. Now more apps register and record events. Some new events recorded are:
A new link was added under the User menu to show all the events of the currently logged user.
Existing config.yml files need to be updated manually. The prefix 'mayan.apps' must be added to any reference of an app.
All the Internet Explorer specific HTML markup was removed from the templates.
Installation seems to be the most error-prone stage of the process in getting to use Mayan EDMS. Just like configuration, installation, and dependency management are other areas where the underlying framework and language don't provide tools and it is up to the project to come up with an alternative to bridge the functionality gap.
This version includes a new dependency manager that is able to track the dependencies of each app. The dependency manager supports JavaScript, Python, and binary dependencies. This allows the dependency manager to also provide users with a centralized view to diagnose their installations for missing dependencies that may be affecting the functionality.
The dependency manager also includes improvements that will allow reusing downloaded JavaScript packages and can perform hash verification faster.
Besides the dependency check view in the Tools menu, a management command was added to show the same information from the command line. This command is called checkdependencies.
Developers can also use the new generaterequirements command to generate the Python requirement files from the dependencies declared in the apps of the project, keeping all dependency files synchronized with ease.
Big or complex app modules were split into separate modules. This includes models, views, and tests. A star (*) was added to keep current imports working but it is encourage to import from the module itself.
For example: The document apps models.py module was split into:
The module documents/models/__init__.py include them all using a start import so that any import to the path mayan.apps.documents.models still works.
Other modules that were split were:
Development URL definitions were removed from the main URL definition file. Instead, the top level urls.py file was converted into a module folder holding several URL pattern definition files. Each URL pattern definition file can be specified independently by changing the value of the ROOT_URLCONF setting.
Instead of inserting the path of the apps into the Python path, the apps are now referenced by their full import path. For example: Instead of referencing the Documents app using the string documents, the full dotted path must be used, mayan.apps.documents. This change namespaces all apps under a module folder and solves name clashes with external or native Python libraries. Once example of such name clash: The statistics app was called "statistics" and it clashed with Python's new statistics library of the same name. The solution at the time was to rename Mayan's to "mayan_statistics". With this change solutions like this won't be necessary.
This means that any reference to an app, either in the code or from a configuration file must now be prefixed with mayan.apps. Existing config.yml files need to be updated manually.
For developers, a new test mixin was added called SilenceLoggerTestCaseMixin. It allows lowering the level of loggers during tests. To use this mixin call self._silence_logger() and provide the dotted path to the module whose message level will be lowered. By default the new level will be set to critical.
Support was added for link icon path imports. Instead of importing all icons, a link can reference the dotted path of the icon. The link argument for this is called icon_class_path.
Support for link icon strings was removed. Only icon classes are allowed now. This keeps all icon definitions encapsulated in the icons.py modules.
Middleware were updated to support both, Django's old and new style middleware.
A new class named FormOptions was added to reduce the boilerplate code needed to add Meta options to custom Form classes. This class is used to create a new reusable form that will filter a selection field based on a permission. This reusable form is called FilteredSelectionForm. This class is used to simplify and unify the TagMultipleSelectionForm, CabinetListForm, ACLCreateForm, IndexTemplateFilteredForm, and DocumentVersionSignatureCreateForm subclasses.
Support was added for help text on extra fields of the DetailForm.
Language choices generation was moved to documents.utils.
The TwoStateWidget class was converted to work as a template widget. It is also now compatible with SourceColumn and don't need to be evaluated using lambdas.
The SourceColumn class received many updates to make is more versatile and reduce the boilerplate code to define model columns. It now supports related attributes using a double underscore separator. This eliminates the need of functions or lambdas for related fields. Also added was support displaying a placeholder text for empty attribute values. This eliminates the need to create a model or class method to capture empty or None values and display a translatable text. Support was added to enable an instance's absolute URL for a column removing the need to add and define a single use link for a clickable column. The SourceColumn class now supports keyword arguments via kwargs to pass to the column function.
The star import was removed from the ACL and Common app's __init__.py module. All acls.classes.ModelPermission, common.generics, and common.mixins imports have to done explicitly.
The authentication function views were updated to use Django's new class based authentication views.
The current user views were moved from the common app to the user management app. And vice versa the user and current user password change views were moved to the authentication app from the user management app. Now everything related to accounts resides in the user management app and anything relating to passwords and authentication resides exclusively in the authentication app.
The custom email form widget provided by common.widget was removed as Django now includes one.
All file related utilities were moved from the common.utils.py module to the to the storage app's utils.py module.
The navigation and authentication templates were moved to their respective apps. They are no longer found in the appearance app's templates folder.
General queryset slicing of the Document list view was removed and added to the only subclass view that uses it which is the Recently Added Document view.
A new view called AddRemove was added which replaces AssignRemove. All views were updated and AssignRemove removed from the code. This class offers a much cleaner design and more functionality like adding or removing multiple items from a single post request. This is achieve by passing a queryset of the selection instead of calling the add or remove methods for each item of the selection.
A new test case mixin was added to provide ephemeral test models. These are memory only model classes that allow tests to be performed much faster while testing all aspects as if it were any other statically defined model. Several test views that used documents models were converted for speed increases of several order of magnitude. Along with ephemeral models, ephemeral test permissions were added removing the need to use static permission for generic permission compliance tests.
Move stub filtering to the Document model manager. All subclasses of the DocumentListView class were updated to simplify their document queryset generation.
The transformation manager methods were renamed from add_for_model to add_to_object, and get_for_model to get_for_object, to reflect the fact that they can only operate only models.
The title calculation was converted from a template markup to a template tag.
The permission inherited computation was improved and now operates mostly at the database layer instead of being mostly a Python computation. A similar improvement was done to the ACL calculation system. The ACL calculation now operates mostly on the database layer freeing many resources and scaling better. The access checking method was updated to be a wrapper for the new ACL queryset calculation, reducing the amount of code and benefiting from the ACL queryset calculation improvements.
Related to the ACL calculation updates is the fact that views will now operate of a filtered queryset as returned by restrict_queryset instead of directly checking the access of the action being performed by the view. The update process of the views to this new methodology was started and will be completed in the next versions. Eventually the .check_access method will be either removed of used for special cases.
A small change with big repercussions is the rename of get_object_list to get_source_queryset. This change represents a shift in the way views will filter their queryset in the entire project. Since Django doesn't provide hooks for transforming a queryset before usage by the view, instead of being passed a model or a queryset directly, views will be passed a source queryset via get_source_queryset which will be filtered by restrict_queryset and this final queryset will be passed to the view as if from its normal get_queryset method. This allows using views as they are normally provided by Django and add Mayan's access control with minimal changes.
Uniqueness validation was added SingleObjectCreateView.
MultipleInstanceActionMixin was removed, MultipleObjectFormActionView or MultipleObjectConfirmActionView must be used instead. These two classes provide action handling for single (using pk) or multiple model instances ( using id_list) from the single view definition.
MultipleObjectMixin improvements were back ported. Selections are passed using get_object_list or self.object_list.
ObjectListPermissionFilterMixin was removed.
The license.py module in apps was removed, licenses are now defined in dependencies.py for the ones that the dependency app can't introspect automatically.
Model permission proxy models support were removed. The permission system can now detect inherited permission from multiple levels. Related to this change the related access control argument to check_access, links, API views mayan_permission_attribute_check was removed. This is all now handled by the related field registration.
The permissions system now supports nested access control checking. Self referencing models like Cabinets, must provide a custom function to filter their permission checking.
An optimization was added which removed the exception catch for the permissions argument of the check_access method. permissions argument must now be an iterable.
Signal handlers must now be prefixed with handler.
Remove .filter_by_access. Replaced by .restrict_queryset.
Rename the form template form_class to form_css_classes to differentiate it from Python form classes.
All views now redirect to the common app's home view instead of the REDIRECT_URL setting.
The project was updated to PyYAML version 5.1. The use of safe_load and safe_dump was changed to load and dump using the CSafeLoader and SafeLoader as fallback. This makes YAML handling faster and safer.
Mailing profiles were updated to allow specifying the email sender address. This change closes GitLab issue #522.
If an existing mailer profile specified a backend that is now invalid instead of showing an error it will be replaced by a new NullMailer class.
Due to the change of using the entire dotted path of apps, the backend of existing mailing profiles will be invalid and must be re-created.
The document link URL when mailed is now composed of the COMMON_PROJECT_URL setting plus the document's URL instead of Django Site domain. This was the only use of Django's Site model and was removed to an easier to configure setup.
Two new Mayan EDMS Requests for Comments were approved during version 4.0 development and applied to this release too.
MERC 5 now requires all callables to use explicit keyword arguments. This MERC is effect makes positional arguments obsolete. These are only retained for Python modules and callables that don't support named or keyword arguments.
MERC 6 introduces a security and privacy policy. This policy is a preemptive information disclosure reduction. This means that code and views in general will disclose less information than they used to when the user doesn't have the required access for an object, view, or action. Instead of displaying an "Access denied" or "Forbidden" error, a "Not found" or 404 error will be raised. This way the user will not have any information about the existence of a resource for which access has not been granted. To keep the API compatible for this minor release, MERC 6 was put into place for the views only.
If you are developing a third party app, update your non-access view tests to expect a 404 and not a 403 error.
The code audit performed during the development of version 4.0 revealed many areas where optimizations were possible. All the backward compatible optimizations were backported to this version. These are:
As a result, the memory footprint and CPU usage were lowered substantially. Memory usage was lowered to 700MB of RAM under full load. This is great news for all user but of special importance for restricted environments like low tier virtual hosts, container deployments, and single board computers like the Odroid or the Raspberry Pi.
Add new and default Tesseract OCR backend was added to avoid Tesseract bug 1670 (https://github.com/tesseract-ocr/tesseract/issues/1670, https://gitlab.gnome.org/World/OpenPaperwork/pyocr/issues/104). This bug make it impossible to use Tesseract via PyOCR under Python 3.
The role permission grant and revoke permissions were removed. Instead only the role edit permission is required to grant or revoke permissions to a role.
Dual permissions are now required for some objects. For example, when adding or removing users to and from a group, the edit permission must be granted to the user and the group. These permission changes are minimal for this version but will be rolled out to be the norm on the entire system.
Long awaited Python 3 support is here. To ensure a smooth transition only the Python package will be release supporting Python 2.7 and 3. For the next release, the Docker image will be converted to work on Python 3. And finally, on the release of the next major version, version 4.0, Python 3 will be the only Python version support. This version of Mayan EDMS, as well as future versions of the same series (3.x) will be the last version supporting python 2.7.
To recap:
If important security issues are found subsequent series 3.x releases will be done for users that can't for some reason upgrade to a version using only Python 3.
Database transaction handling was added in many more places to ensure data integrity even in extreme situations.
For each version, we identify functionality or requirements that can be removed. For this version, Django suit was removed from requirements as well as support for generating documents images in base 64 format.
The HOME_VIEW setting was defined without a namespace and as a top level setting. This configuration is reserved for native Django setting and the HOME_VIEW setting is now namespaced to the COMMON app where it is defined. The setting global name therefore changes from HOME_VIEW to COMMON_HOME_VIEW.
More Django settings were exposed and can now be modified:
New default value of 65535 for the DOCUMENTS_HASH_BLOCK_SIZE setting. This means that new documents will be read and process in blocks of 65K to determine their SHA256 hash instead of being read entirely in memory.
Removal of the MIMETYPE_FILE_READ_SIZE setting. A new method was implemented to reduce memory usage of the MIME type inspection code. Instead of limiting the number of bytes read as specified by the MIMETYPE_FILE_READ_SIZE setting, now the entire file is saved to a temporary file and the MIME type library called with the temporary file reference. This approach while minimally slower provides the benefits of lower memory usage without sacrificing MIME type detection accuracy which was a downside of the MIMETYPE_FILE_READ_SIZE setting approach.
Several improvements were back ported to the search app. One of this allows returning the search result as a queryset. Queryset are "lazy" and not evaluated until accessed. This means a queryset can represent a vast number of documents with consuming the entire memory that would be required to hold all the documents instances as a list would. This change make the memory limiting setting SEARCH_LIMIT obsolete and was removed.
Additionally the search time elapsed calculation was removed. This code stopped being used from the code several version ago.
The default value for the recently added, recently accessed, and favorite documents settings was increased from 40 to 400. Using the default pagination size of 40 documents per page than means a total of 10 pages of documents for each of one of these views instead of just one page.
The setting COMMON_TEMPORARY_DIRECTORY was moved to the storage app. The setting is now called STORAGE_TEMPORARY_DIRECTORY.
Many user interface updates were back ported some very noticeable others less so but still have a big impact on usability and navigation.
New icons classes and templates were added which allow many icon manipulations like icon composition. This allow for a more unified visual language.
The right side menu, called the "sidebar" is now a separate unit in both markup and code. This new side bar is implemented independently of the main view. This change allows for independent scrolling and more usable screen area.
Support was added for sortable columns. These allow sorting the results of a list by clicking on the column heading label. Clicking on the label again reverses the sorting order.
select2 widget is used in more places reducing data entry and providing quick search capabilities.
The markup for the invalid document image was converted into a server side template for easier customization and reuse.
The sidebar menu instance was removed. The secondary menu now perform the same function.
For every function that involved two types of objects, two views were added. For example: Associating a workflow to a document type can now be done from the workflow edit view or from the document type edit view. This eliminates jump over from the view of one object type to the other.
Form can be submitted by pressing the Enter key or by double clicking.
The full name field was removed from the user list. Instead the first name and last name fields are shows which are sortable.
Completion level and initial workflow state are now shown in the workflow proxy instance menu.
Fix translation of the source upload forms using the drag and drop dropzone.js widget.
Bootstrap and Bootswatch were updated to version 3.4.1.
Support was added for attaching form button aside from the default submit and cancel. The new add and remove views use this allow adding all or removing all with a single click.
CSS updates were added to maximize usable width of the screen.
Navigation error messages display was improved for easier debugging.
The permission list was removed from the ACL list view. This reduces clutter and unpredictable column size. The permission list can be accesses by pressing the "Permissions" button.
The default title truncation length was increased to 120 characters. Even when truncated, hovering over the title will now show the entire title as a pop over.
Previously form widgets and HTML widgets resided in the same .widgets.py module. The .widgets.py module is now reserved for form widgets and HTML widgets will be found in a new module called html_widgets for each app.
The interface for Mayan's HTML widget has been formalized and these can be used with the SourceColumn class without having to use a lambda. The widget argument was added to the SourceColumn was for this purpose.
The tag selection form widget uses some specialize JavaScript to support rendering the tag during selection. This code used to live in the main template and was loaded even when not in use. The JavaScript code was move to the tags app and is now loaded only when used. This will cause a slight visual artifact when the form is loaded. Aside from this visual side effect it continues to work as before.
Remove deprecated requirements:
$ curl https://gitlab.com/mayan-edms/mayan-edms/raw/master/removals.txt | pip uninstall -r /dev/stdin
Type in the console:
$ pip install mayan-edms==3.2
the requirements will also be updated automatically.
If you installed Mayan EDMS by cloning the Git repository issue the commands:
$ git reset --hard HEAD $ git pull
otherwise download the compressed archived and uncompress it overriding the existing installation.
Remove deprecated requirements:
$ pip uninstall -y -r removals.txt
Next upgrade/add the new requirements:
$ pip install --upgrade -r requirements.txt
Perform these steps after updating the code from either step above.
Make a backup of your supervisord file:
sudo cp /etc/supervisor/conf.d/mayan.conf /etc/supervisor/conf.d/mayan.conf.bck
Update the supervisord configuration file. Replace the environment variables values show here with your respective settings. This step will refresh the supervisord configuration file with the new queues and the latest recommended layout:
MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \ MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \ MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ /opt/mayan-edms/bin/mayan-edms.py platformtemplate supervisord > /etc/supervisor/conf.d/mayan.conf
Edit the supervisord configuration file and update any setting the template generator missed:
vi /etc/supervisor/conf.d/mayan.conf
Migrate existing database schema with:
$ mayan-edms.py performupgrade
Add new static media:
$ mayan-edms.py preparestatic --noinput
The upgrade procedure is now complete.
Due to the change of using the entire dotted path of apps, the backend of existing mailing profiles will be invalid and must be re-created.
Paths to apps must be updated in existing config.yml files. Add the prefix mayan.apps. to any app reference. Some instances:
::: system-message System Message: WARNING/2 (<string>, line 820)
Cannot analyze code. Pygments package not found.
.. code-block :: python LOCK_MANAGER_BACKEND: lock_manager.backends.file_lock.FileLock
:::
to
::: system-message System Message: WARNING/2 (<string>, line 826)
Cannot analyze code. Pygments package not found.
.. code-block :: python LOCK_MANAGER_BACKEND: mayan.apps.lock_manager.backends.file_lock.FileLock
:::
::: system-message System Message: WARNING/2 (<string>, line 830)
Cannot analyze code. Pygments package not found.
.. code-block :: python OCR_BACKEND: ocr.backends.pyocr.PyOCR
:::
to
::: system-message System Message: WARNING/2 (<string>, line 836)
Cannot analyze code. Pygments package not found.
.. code-block :: python OCR_BACKEND: mayan.apps.ocr.backends.pyocr.PyOCR
:::
::: system-message System Message: WARNING/2 (<string>, line 840)
Cannot analyze code. Pygments package not found.
.. code-block :: python CONVERTER_GRAPHICS_BACKEND: converter.backends.python.Python
:::
to
::: system-message System Message: WARNING/2 (<string>, line 846)
Cannot analyze code. Pygments package not found.
.. code-block :: python CONVERTER_GRAPHICS_BACKEND: mayan.apps.converter.backends.python.Python
:::
Check the supervisord logs at /var/log/supervisor for additional errors in the form:
ImportError: No module named ocr.backends.pyocr
To collect and compress the static media files, use the new preparestatic command instead of the traditional collectstatic. Both work the same way, but preparestatic has a default blacklist to avoid collecting files from tests, development setup, and demos.