Version 4.7

Released: May 07, 2024

Changes

Appearance

Added locale aware thousand separator to list header number and page details.

Added a list toolbar widget to control the sorting using a dropdown. This allows sorting item list views.

Removed URI.js. Instead the JavaScript native URL class is now used.

Removed jQuery Lazy Load library [jquery-lazyload]{.pre} and use browser native lazy loading.

Removed the library [jquery-scrollstop]{.pre}.

Added a button to refresh the AJAX view port. This allows refreshing the content without loading the entire page, the JavaScript, and waiting for the JavaScript initialization.

Added stricter UI throttling defaults. From 10 requests in 5 seconds to 5 requests in 4 seconds.

When users cancels a pending request and the backend is still processing the previous one the content is now kept in the view. The cursor icons is changed instead to show the progress state.

Updated FontAwesome from version 5.15.4 to 6.5.2.

Converter

Added a QRCode document version decoration transformation. This transformation can use workflow context for the value of the QRCode to be stamped on documents. Multiple QRCodes can be added per document page.

Updated the image converter to support loading damaged and truncated images. This behavior is controlled with the new setting CONVERTER_LOAD_TRUNCATED_IMAGES which defaults to False for backwards compatibility.

Additional errors are now captured and presented using specific error icons.

The convert workflow actions now support templating the arguments of the transformations.

Credentials

Added a credentials backend for Google Service accounts.

Credential usage is now tracked by object. Such as sources, workflow actions and mailing profiles.

The credentials used event is now committed only when credentials are retrieved via the get_credential method, not when they are loaded from the database by the backend.

Dependencies

Update dependency versions:

• mozilla-django-oidc from 3.0.0 to 4.0.0.
• django-model-utils from 4.3.1 to 4.4.0.
• importlib-metadata from 7.0.1 to 7.0.2.
• twine from 4.0.2 to 5.0.0.
• devpi-server from 6.5.0 to 6.10.0.
• django-debug-toolbar from 4.2.0 to 4.3.0.
• ipython from 8.21.0 to 8.22.2.
• pypdf from 3.17.4 to 4.1.0.
• python_gnupg from 0.4.9 to 0.5.2.
• python-dateutil from 2.8.2 to 2.9.0.post0.
• sentry-sdk from 1.40.6 to 1.41.0.
• extract-msg from 0.48.0 to 0.48.2.
• django-celery-beat from 2.5.0 to 2.6.0.
• gevent from 23.9.1 to 24.2.1.

Removed the development dependencies:

• flake8
• ruff
• pdbpp
• Remove Django Rosetta from dependencies.
• Remove tornado and sphinx-autobuild libraries.
• Remove the tox testing dependency.

Docker

The Docker Compose file now includes support for backing up PosgreSQL and for converting the PostgreSQL binary files to a new version.

It is now possible to pass more options to the database connection URL to allow enabling specialized options like disabling server side cursors and using connection pools like PgBouncer. This is done via the environment variable MAYAN_DATABASE_EXTRA_OPTIONS.

The PostgreSQL settings max_connections can be configure via the environment variable MAYAN_DOCKER_POSTGRESQL_MAX_CONNECTIONS. The default value is now 150 connections.

The PosgreSQL container supports loading custom initialization SQL via a new initdb.d volume.

The PostgreSQL image was updated from 13.14-alpine to 14.11-alpine. An update of the binary file must be performed before upgrading Mayan EDMS.

A health check was added to the PostgreSQL service.

A new Docker Compose service was added backup the PostgreSQL database.

The ElasticSearch image was updated from 7.17.17 to 7.17.20.

Documents

All document lists in any view can now be sorted by label, type, or creation date.

Stricter document creation event committing order is now enforced. This makes it more reliable when automations depend on it as a trigger.

Document file page count calculation errors are not tracked and logged.

Page count detection errors are tracked in the document file error log as reported by the converter backend.

Support was added for searching document files by size.

The action to move a document to the trash was converted into a background task. Sending documents to the trash is now much faster as a result. Via the REST API this endpoint now returns 202 ACCEPTED instead of 204 NO CONTENT.

Support searching document by ID, the ID of the document type, the ID of the tag, or the ID of the cabinet.

An API view that returns all documents of a type was added.

Document indexing

The document index instance depth and node count values are now exposed via the API.

Events

Added notification detail API view. Notifications can now be marked as read via the API.

File metadata

Added support for EML file metadata extraction. Sender, subject, and others properties are supported. The properties are extracted for the main EML file but also for any attachment or nested EML file.

Locales

Support was added for setting user locale default language and default timezone. This is done with the settings LOCALES_USER_DEFAULT_LANGUAGE and LOCALES_USER_DEFAULT_TIMEZONE.

Logging

Added support for error log domains to identify the source of the error log entry.

Messaging

Updated the messaging app workflow action to support sending messages to users by group and by roles.

Packaging

Removed the testing app from the production build.

Removed tests from Python and Docker images. Python source package size reduced from 5.4MB to 4.2MB, 22% less. Wheel package size reduced from 8.3MB to 6.9MB, 16% less.

Platform

The gevent worker is now the default for gunicorn.

REST API

Refactored the API field ordering feature. It now supports sorting for many fields and even nested sorting of sub models. The sort field detection works for any views that returns a lists of objects.

An API sort testing mixin was added to make writing sort tests easier.

Search

The ElasticSearch backend search method was updated to use the "search again" instead of the "scroll" API. This worksaround the "scroll" API limit of 10,000 results retrieval. It is now possible to retrieve an unlimited number of search results. The default limit remains at 1,000 results for backwards compatibility.

The default search result limit per search scope was increased to 100,000 results.

Generators are now used to yield the ID list of scope results lowering memory usage for heavy multi scope search queries.

Support was added of storing search results. This feature is called "Saved resultsets". The interface of this app is identical to the search feature and allows for a drop-in usage without client code changes. Saved results take the same initial time to calculate but paging the results does not perform additional search queries making paging over the results very fast event for large result sets.

The search backends were converted into folder modules to avoid them sharing an default literals module. This changes allows removing search backends in custom builds with not further code changes.

Support was added for ignoring SSL certificates for ElasticSearch. Required for ElasticSearch that only work over HTTPS like ECK (Elastic Cloud for Kubernetes).

It is now possible to support searching for 64 bit big integers and positive big integers.

A search model detail API view was added.

Storage

Added support for download backends. This allows downloading large document files directly from the storage backend. This feature is required when downloading multi gigabyte documents whose download time exceed the HTTP timeout. This is important in environment with slow network connections or where is not possible to extend the HTTP timeout such as provider managed Kubernetes clusters with no access to the ingress controller.

A normal HTTP backend with direct storage access is included as well as a backend for Google Cloud Storage signed URLs downloads.

The document download app was updated to add the new settings DOCUMENT_DOWNLOADS_DOCUMENT_FILE_DOWNLOAD_BACKEND and DOCUMENT_DOWNLOADS_DOCUMENT_FILE_DOWNLOAD_BACKEND_ARGUMENTS.

Support was added for decompressing EML files. EML can be treated like a ZIP file and its content uploaded as individual documents.

The cache manager tool now shows the name of the defined storage entry.

The file caching cache detail form was improved and fieldsets added.

Added a management command to purge a cache. The command name is file_caching_purge_cache and requires the cache's storage name.

Sources

The library flanker was removed from the source_emails app. The app now uses native Python libraries to parse emails.

The staging sources now have a default preview width value of 640 pixels. The help text was also improved.

The sources app storage cache is now under the control of the file caching app. This allows monitoring cache usage, purging the cache, and doing automatic pruning when space runs out. This adds the setting SOURCES_CACHE_MAXIMUM_SIZE which defaults to 100MB.

Improved how staging storage files are retrieved and cached. Preview errors are with a dedicated with a proper AppImageErrorImage instance for user experience parity with the document preview system.

Support setting the document language for periodic sources. Emails and watch folders documents can be uploaded with a predetermined language set to improve automate processing like OCR.

Templating support was added to the sources inclusion and exclusion regular expression fields. This allows filtering automatic uploading of document by a making the regular expression dynamic and can reference things like the current date and time.

Added support for searching documents by the source metadata content.

Added the REST API endpoints for the source metadata feature.

Made the source metadata key sortable.

Workflows

Workflows transition fields can now be sorted by their label.

The workflow instance transitioned event is now committed before the workflow template state action execute. This ensures consistent event logging even when a workflow action fails.

Refactored the workflow preview diagram generator. Each unit of the workflow now performs partial rendering based on its properties. This makes the workflow preview system modular and allows adding preview features to custom workflow components.

Workflow template fields now support lookups. Specifying a list of options automatically changes the widget of the field to a select.

Workflow API endpoints were added to return the documents attached to a workflow and the documents at a particular workflow state.

Consolidated the workflow state action code into a subset of the Workflow State model. This removed duplicated implementations and allowed for query optimizations. Compounded over 10,000,000 concurrent workflow instances tests, memory usage was reduced by 38% and workflow action time to execution start was reduced by 22%.

The workflow state action errors are now stored in the error log of the document.

The HTTP request workflow action now supports storing the response in the workflow context. This makes it possible to write feature complete API clients to third party systems using the workflow engine.

Added support for workflow transition field defaults. The default can be static or set via a template.

Added extra columns to show if a transition field has a default and a lookup.

Other

• Remove local version from the platform cache identifiers.
• Move the _save method to the document.model_mixins module.
• Workflow template state action test mixin improvements.
• Rename ._test_workflow_template_states to ._test_workflow_template_state_list for uniformity.
• Improve how test workflow template state actions are created and managed.
• Improvements to the WorkflowInstanceTestMixin class to track test workflow instances.
• Add event checking to workflow template state action tests in multiple apps.
• Moved mailing actions from the class module to their own mailing_actions module.
• Remove live documentation auto build.
• Fix MSG file uncompressed uploads.
• Disable automated menu refresh on development mode.
• Improve how the test views are managed.
• Allow the JavaScriptCatalogPublic sub class to bypass authentication and avoid JavaScript errors for non authenticated users.
• Split workflow template fields and workflow template triggers into their own modules.
• Workflow query optimizations.
• Split metadata models.
• Split Python targets into their own makefile.
• Remove [--without-gossip]{.pre} [--without-heartbeat]{.pre} from the all-in-one deployment workers.
• Unify condition field model mixin and helper methods.
• When loading a lazy load image, try to obtain the reason for the failure.
• Add AppImageErrorImage indicator for when a document file has no pages.
• Fix display of build number.
• Restore the root logging handlers after every test.
• Image converter updates:
  • Support catching AppImageError exceptions and returning the error template.
  • Add the context argument to the AppImageErrorImage class to allow passing context to the template such as error details.
  • Add a general purpose AppImageErrorImage to the converter for most image errors.
• Convert the MetadataLookup class into a general purpose template context registry. Add support for always available entries.
• Improve how detail form field labels are extracted.
• Consolidate workflow transition field code into the transition model.
• Move the workflow instance detail link from the actions menu to the list facet menu.
• Update the datetime field of the workflow instance entry list to be sortable.
• Don't run the initial_setup_or_perform_upgrade services when enabling the extra_worker_custom profile.
• Optimize and convert the batch source metadata code into a reusable coroutine with batch model creation.
• Remove the source field from the source metadata model.
• Convert the file metadata app to use the the bulk create manager mixin.
• Remove Link class argument link.html_extra_attributes.
• Fix document type model verbose name.
• Fix detached signature link object reference.

Removals

Backward incompatible changes

Deprecations

Issues closed

• None