Insane scalability
Scale up to millions of documents and hundred of millions of pages.
Released: September 30, 2020
Support for themes was added. This allows administrator to create CSS themes that modify the appearance without making code changes. The users can select what theme they want to apply from the "users" menu. The themes are sanitized using the bleach library to avoid JavaScript code to be inserted along with the theme markup.
A new action level section called "Related" was added. This section will display links to other apps that hold a relation with the current view. For example, this allows quickly switching from the roles, groups, and users views, in a single click.
Each app now has the ability to specify their own static media ignore patterns via the app config attribute static_media_ignore_patterns. This new option allowed the removal of more unused media files, lowering the static media folder size from 83MB to 51MB.
A new layer called "Decorations" was added. This layer allows adding images as overlays on top of document page images. Overlay images are added via the new Setup option named "Assets".
Two initial decoration transformations were added. The first allows displaying an asset over a document page image. This transformation is useful for things like stamps. The other transformation allows displaying an asset as a watermark. This transformation will display multiple copies of the asset of the entire document image page. Both transformations support rotation and opacity settings.
A new workflow action was added to add transformation to document pages.
The transformation view was updated to disable the "Edit" button for invalid transformations and transformations that do not support arguments.
Add support to change the dashboard widget details link icon.
A new class was added named ModelQueryFields. This class allows apps to specify which fields should be prefetched on database access, lowering memory and improving database access.
At the low level, this class interfaces with Django's select_related and prefetch_related features.
The following dependencies version were updated:
- Celery from 4.3.0 to 4.4.7
- chart.js from 2.7.2 to 2.7.3
- coverage from 5.0.4 to 5.1
- coveralls from 1.11.1 to 2.0.0
- drf-yasg from 1.6.0 to 1.17.1
- djangorestframework from 3.7.7 to 3.11.0
- django-extensions from 2.2.8 to 2.2.9
- django-rosetta from 0.9.3 to 0.9.4
- flake8 from 3.7.9 to 3.8.3
- flex from 6.14.0 to 6.14.1
- gevent from 1.4.0 to 20.4.0
- graphviz from 0.13.2 to 0.14
- ipython from 7.13.0 to 7.15.0
- PostgreSQL client from 2.8.4 to 2.8.6
- python_gnupg from 0.4.5 to 0.4.6
- pytz from 2019.1 to 2020.1
- Redis client from 3.4.1 to 3.5.3
- safety from 1.8.7 to 1.9.0
- select2 from 4.0.3 to 4.0.13
- sh from 1.12.14 to 1.13.1
- sphinxcontrib-spelling from 4.3.0 to 5.0.0
- swagger-spec-validator from 2.4.3 to 2.5.0
- transifex-client from 0.13.8 to 0.13.10
- tox from 3.14.5 to 3.14.6
The dependency settings for kombu were removed. This dependency is now automatically installed by Celery.
The Docker image was updated from Debian 10.3 to 10.5.
The image was also updated to use the headless version of Libre Office to reduce the image size.
Support was added for document type filename generators. These allow users to specify how documents are physically named when uploaded to Mayan. Two backend are provided. One preserves the existing behavior of using an unique name generated at the time of upload, and a new one that will preserve the filename of the document uploaded. The filename generator can be set for each individual document type.
Resource locking was added to the document page image generation to avoid a race condition when system are under heavy load.
Document image generation is now also retried after failure. The new setting named DOCUMENT_TASK_GENERATE_DOCUMENT_PAGE_IMAGE_RETRY_DELAY controls the retry delay of the document page image generation task.
Document stubs can now be accessed. These allow manipulating documents even when they have not yet been fully processed. These are useful when uploading many documents over the REST API.
Support was added to search document types.
The document trash empty action was converted into a background task to avoid blocking the user interface on large number of documents.
The default document file parser was updated to preserve and store the original document filename as part of the document file metadata.
The error logging code was moved from the "Common" app to a new app called "Logging". Besides system logging, this app also provides a new unified database logging and logging user interface view for apps.
The new logging app allows changing the level of the logging.
With the addition of the "Logging" app some settings were renamed. The COMMON_AUTO_LOGGING and COMMON_PRODUCTION_ERROR_LOGGING have been merged into LOGGING_ENABLE. COMMON_PRODUCTION_ERROR_LOG_PATH is now LOGGING_LOG_FILE_PATH and continues to default to the MEDIA_ROOT/error.log path. The new setting LOGGING_LEVEL controls the log level. A second new setting named LOGGING_HANDLERS controls the list of output log handlers. It defaults to console but also supports a second one named logfile. The logfile handler is the same one that previously enabled when setting the setting COMMON_PRODUCTION_ERROR_LOGGING to true.
The mailer app was updated to use the new logging app and user interface.
The paradigm of using runtime.py modules in each app was removed. Instead each consumer must instantiate the class. This change avoids keeping long lived objects in memory, and help the garbage collector reclaim memory areas easier.
Workflow actions were added to add, edit, and remove metadata from documents.
The mountindex command can now be set to run in the background.
When running the command in the foreground, a message is now displayed to avoid confusion and ensure the task is not frozen.
The OCR task was refactored to use Celery canvas. This change allows launching an OCR task for each page of the document. This increases the scalability potential of the OCR and avoid long running background tasks.
Tesseract's multi threading support is now disabled by default. This speeds up the page OCR and lower the CPU requirements.
A workflow action was added that allows updating the document OCR content.
Support was added to allow copying objects. Object supported are: document types, groups, mailing profiles, metadata types, messages of the day, workflows, quotas, roles, smart links, tags, web links.
It is now possible to change the Gunicorn worker class via the environment variable MAYAN_GUNICORN_WORKER_CLASS.
More Python 2 code removals landed in this version.
Explicit checks and conditional code for Python 3 was removed.
Conditional assignment of FileNotFoundErrorException was removed. Python 3's FileNotFoundError is now used directly.
Casting of dict_type and dictionary_type was removed.
Usage of the python_2_unicode_compatible wrapper was removed from the models.
Explicitly subclassing from object was removed
Usage of django.utils.six was removed.
Columns showing the number of groups and permissions of a role were added.
The search system refactor is now complete. On top of this refactor a new search backend was added. This new backend allows using the "Whoosh" search engine. The setting SEARCH_BACKEND controls which backend is used. To enable the use of "Whoosh", change this to mayan.apps.dynamic_search.backends.whoosh.WhooshSearchBackend.
This backend will be the default one in a future release.
The search form now sorts the search fields. Icons for the search, advanced search and search again links were added.
To speed up round trips during search sessions, the search again link is now displayed on the empty results view.
The search again link now redirects to the same search form used instead to always redirect to the advanced form.
New setting was added to limit the number of search results returned. This setting avoids runaway CPU usage on ambiguous search terms. The setting name is SEARCH_RESULTS_LIMIT and defaults to 100.
The sources app was updated to use the new unified logging.
An error will be raised if the watch folder doesn't exists or is not a directory.
The statistics queue was moved from the slow worker to the medium worker. This allows getting statistics updates even during heavy load.
The SharedUpload model was moved to the storage app. The setting COMMON_SHARED_STORAGE is now STORAGE_SHARED_STORAGE, and COMMON_SHARED_STORAGE_ARGUMENTS is now STORAGE_SHARED_STORAGE_ARGUMENTS.
Celery's BROKER_LOGIN_METHOD and BROKER_USE_SSL settings are now exposed via the new CELERY_BROKER_LOGIN_METHOD and CELERY_BROKER_USE_SSL settings. CELERY_BROKER_LOGIN_METHOD defaults to AMQPLAIN and CELERY_BROKER_USE_SSL defaults to None.
New tags and filters were added or made available by default for the templating system.
The mathfilers are enabled by default.
Add a set tag was added to allow setting template variables.
A dict_get filter was added that returns a given dictionary key.
A tag named {% method %} was added to allow calling an object's method with or without keyword arguments.
A set of regular expression tags were added. These are regex_findall, regex_match, regex_search, and regex_sub. Each regular expression tag support the flags ascii, ignorecase, locale, multiline, dotall, and verbose.
A new split filter was added to split a value by a delimiter.
A new tag/filter dropdown list was added to the template widget to make selection easier during template markup editing.
All test related code was moved to a new app called "Testing".
All view generation code was moved to a new app called "Views". Due to this change the setting COMMON_PAGINATE_BY is now named VIEWS_PAGINATE_BY. The new views app controls generic views, view mixins, forms, and widgets.
The web link navigated event is now enabled to use for subscription and as workflow trigger.
Web links were updated to enforce label uniqueness. A data migration is included to de-duplicate the labels before altering the schema.
The workflow context code was updated to only instantiate valid workflow transition fields from an existing workflow instance context. This avoids problems when a workflow had a transition field with data, and that fields is later deleted from the workflow template.
A columns was added to workflow transitions showing the fields it supports.
Workflow actions are now grouped by app and select2 support was added to allow autocompletion search.
A new workflow action was added that allows a workflow to launch other workflows.
The code that instantiates workflow has been converted to a background task. This speeds up uploading documents in bulk.
A boolean field was added to workflows to control whether or not they will launch when a new document is created. This change along with the new action to launch workflows, allows creating complex layouts where a workflow can launch subworkflows.
