Released: October 10, 2021
Status: Stable
The code used to detect open file and descriptor leak detection was improved and made more strict. New descriptor leaks were fixed.
File descriptors created after inspecting a Django file field (even if the file is not opened) are now explicitly managed and closed.
Context managers are now used for temporary files and directories. This ensures file descriptors are closed and freed up in all scenarios even exception errors.
The final uses of the .six
library were removed.
The performupgrade
command was updated to not hide critical errors and
instead raise any exception to obtain the maximum amount of debug
information.
Whenever applicable, the use of the sorted
function was replace with
the .sort
method to reduce memory usage.
The ACL edited event is now triggered only once when all permissions are changed.
The action object of the ACL edited event is now the content object and not the permission being added or removed.
The theme stylesheet sanitization process was moved to the .save()
method. This improves performance when themes are used.
A setting was added to allow changing to menu polling interval. The value is specified in milliseconds with None used to disable polling.
The comments API was updated to exclude trashed documents. The serializers in use were reduced by consolidation. The API permission filter was updated to return error 404 and not 403 on insufficient access for function parity with the rest of the API.
Image generation in all apps was consolidated to a common core which is
then expanded and customized by the apps. As part of this consolidation
settings and internal literals used by external app were merged or
removed. The settings
DOCUMENT_TASK_GENERATE_DOCUMENT_FILE_PAGE_IMAGE_RETRY_DELAY
and
DOCUMENT_TASK_GENERATE_DOCUMENT_VERSION_PAGE_IMAGE_RETRY_DELAY
were
removed as the image generation task now uses exponentially incrementing
retry delays. App specific image generation task queues were removed.
This includes the document_states_fast
task queue.
The setting CONVERTER_IMAGE_GENERATION_MAX_RETRIES
was added to
control how many times the image generation task will retry lock errors
before raising an error and giving up.
The unification of the image generation also allows the use of streaming responses when serving images. Streaming responses allow offloading the actual transfer of data to the application and web server. This results is faster response times, faster image transfers, lower CPU and memory usage.
It is now possible to create multiple dashboards. These are created at the app level and can be reused by other apps.
A new user dashboard was added to reflect Mayan's nature as a multi user system. The new default user dashboard tracks the documents relevant documents for quick access. The previous dashboard has been renamed as the "administrator dashboard" as it shows a consolidated view of the entire installation.
A tool was added to allow accessing all defined dashboards.
Support was also added for selecting the default dashboard for display in the home view.
A new dashboard widget was added to allow displaying a list of objects.
The dashboard widget class was interface was updated to allow subclassing existing widget classes.
Dashboard CSS was moved from the appearance app to the dashboards app.
Support for disabling the home page dashboard was added. Used to disable the home view dashboard during tests.
The link used to check for the latest version was moved to the tools menu.
A new permission was added to the view used to check for the latest version.
Project dependencies version used were update as follows:
The API was updated to exclude trashed documents from all endpoints. A new endpoint was added to serve trashed documents.
A new valid
model managers was added for recently accessed, recently
created, and favorite documents. These managers exclude trashed
documents at the model level. The objects
manager for these model
returns the unfiltered queryset.
The trashed document delete API now returns a 202 code instead of 204. The delete method now runs in the background in the same way as the trashed document delete view works in the user interface. The return code was updated to reflect this internal change.
Better user tracking for trashed document events was added.
Support was added to append all document file pages as a single document version.
The UUID field of all document child objects was added as a search field.
Stricter checks were added to ensure the name and not the path of a file is used during upload.
Compressed files can include path references, these are now scrubbed and only the filename of the file in the compressed archive is used.
Signals, hooks, and events were moved outside of the document file creation database transaction. These are only triggered when the transaction is committed.
When using the document version page reset and remap action the current user is capture for the events.
Conditionals were added to the favorite document links to ensure they only active when relevant.
The recently accessed document list now has a new column showing the data and time of the last access.
The text message sent to users to notify them that a document has been exported is now translated into the locale of the user and not of the system.
The way the document resolution settings are used has been updated.
There are four resolutions specified by the documents app: thumbnail
,
preview
, display
, and print
. The display
resolution is the
highest one meant for detailed viewing of a document's page. This view
allows interactive transformations like zooming and rotation at the
highest quality level.
Previously, the view showing the document page images was using the
display
resolution. Since document pages in this view cannot be zoomed
or interactively transformed, the extra resolution and added image
processing was not fully utilized. In this version, this view now uses
the preview
resolution.
To continue displaying the images in the document preview window at the
previous resolution, set the values of the DOCUMENTS_PREVIEW_WIDTH
and
DOCUMENTS_PREVIEW_HEIGHT
setting to match the values of the
DOCUMENTS_DISPLAY_WIDTH
and DOCUMENTS_DISPLAY_HEIGHT
settings.
The use of Docker Compose profiles added in version 4.0 was expanded to allow more deployment options. By default a single container running the entire stack. However the same Docker Compose file can launch independent workers and frontend containers.
Support was added for clearing the event list. This feature works for all the different event views.
All events are now preloaded at bootup. This allow all events to be recognized even if they are not used and does not rely on imports to activate an event.
Events can now be loaded by their name. This avoids doing direct imports when there are circular dependencies.
The events app was moved to the top of the installed apps list to allow it to preload all events.
The event clear and event export links are now only displayed for object whose events can be cleared and/or exported.
The events app API was updated to return error 404 on insufficient permissions.
The event registrations for several models was adjusted:
DownloadFile
content object registration from the
Document
model to the DocumentVersion
model.Document
model too.Document
model too.Document
model too.DocumentFile
model.DocumentVersion
model.DocumentVersion
model.DocumentVersionPage
model.Document
model
too.Document
model too.Document
model.DocumentVersion
model.These event adjustments removed some obsolete model interactions. These also allow new subscriptions and workflow triggers.
The index instance node value field was updated to be an unique field among its own tree level. This prevents tree corruption under heavy load.
The partial distributed indexing algorithm was replaced with a linear algorithm. The previous algorithm was very fast but suffered from a complete retry if there was a locking error in one of the tree levels during update. The new algorithm is slower in theory as it locks the entire tree, but in practice is more efficient because there are less lock collisions.
Locking the entire tree along with new database integrity changes, further reduces the chances of index tree corruption.
The new algorithm also allows very quick removals using a single query operation as opposed to the previous looped locking. Additions are also now performed using forward related through field managers which permit adding values to many to many fields using a single query instead of a Python loop.
The change in the algorithm required the API to also be updated. Each tree level node returns an URL to list view of its respective children. This change also enabled the switch to a paged API for the index instance.
Index instance columns were updated to be more consistent in their information. New columns were added with additional information. The ordering of the columns was also made consistent between the top level index instance view and the index instance node navigation views.
The document indexing API was updated to exclude trashed documents.
The indexing task was updated to use exponential retry delays with a maximum retry backoff delay of 60 seconds.
The mailing app was refactored to support all document features introduced in version 4.0.
It is now possible to mail document files and document versions as attachments as well as links. Documents can only be shared over email as links.
The workflow action was updated to send a link to the document or send the active version of the document as attachment.
The app icons were updated to make the more intuitive.
The API permission layout was updated to match the one of the user interface views. The edit or view permission is now required for the document type as well as the metadata type.
API serializers were unified and redundant code removed.
The metadata type form tab order was improve. The metadata type name field is now disabled via CSS to skip it during data entry when using tabs.
Support was added for deleting multiple metadata types in a single action.
This version adds the API for the messaging app.
Super users and staff users are now excluded from being message recipients.
A dedicated create message form was added with improved widgets and widget attributes.
Select2 is now used for the user selection field.
The message edit permission was added. This permission is required in order to change the message read status.
The navigation classes were refactored and several important patches merged.
Request and context resolution was unified for all navigation app classes.
Support was added for retrieving a list of SourceColumn
by name.
The SourceColumn.get_for_source
was refactored for better and faster
object resolution. Some navigation object would include child or proxy
objects in the resolution of the parent object navigation while other
excluded them and required explicit inclusion. This logic has been
unified to default to inclusion. All exclusion must now be explicit.
The views resolved_object
is now passed as an argument to the link's
display conditions callback. While the view context is still passed to
the callback, the resolved_object
argument makes determining the
active object using the context unnecessary. The context is still
provided for link conditions that are dependent on other values in the
view context.
Link de-duplication was added to links that are part of the same menu.
The helper function get_cascade_condition
was renamed to
factory_condition_queryset_access
which better reflects its purpose.
Support was added for exclusion during menu link binding.
Callback support was added to factory_condition_queryset_access
.
Support for proxy model exclusions was added.
Support for editing the document version page OCR content was added. This works via the user interface and the API.
Support was added to allow deleting the parsed content of multiple files as a single action.
Support was added for service client backends. These are clients for specialized services operating outside Mayan EDMS.
A service client for Sentry.io was added to allow production exception tracking.
The settings PLATFORM_CLIENT_BACKEND_ENABLED
and
PLATFORM_CLIENT_BACKEND_ARGUMENTS
are used to pass argument such as
credentials to the service backends and select which are to be launched
as startup.
The settings REST_API_MAXIMUM_PAGE_SIZE
and REST_API_PAGE_SIZE
were
added to allow controlling the default page size and the maximum page
size of the result responses.
The get_absolute_api_url
method was added to download files, document
versions and users. These URLs are used to determine the message
sender's API URL.
Batch API request support added. These allow sending multiple API requests inside a single HTTP request. Items of a batch API request can also share their responses to allow for advanced API automation.
Support was added for deleting multiple roles in a single action.
The TASK_RETRY_DELAY
setting was remove and exponential retry used
instead.
The query string is now sanitized and non valid search criteria removed.
The Whoosh backend received many updates. While it is still not the default search backend, this version moves that goal closer.
The Whoosh backend can now reindexing object after related many to many fields values are removed or added. Support for indexing multiple many to many field values as independent values was also added.
The default behavior of the search system was to work as a filter and remove values from the total based on the search query. This caused search results to be inclusive, adding results that were not relevant to the query. The behavior was updated to work like a search filter. Incorrect search queries will now return an empty result set.
Several operation like query processing were moved to the base class reducing the amount of code in the search backends.
The API was updated to exclude trashed documents.
When a signature file is uploaded, the current user is captured and used for the event.
A new event was added to track detached signature deletion.
The sources app was refactored. This refactor remove model inheritance, makes sources work like backends, and allow external app to create their own sources.
The refactor also allows object level permission to be applied to sources which means sources now support ACLs. A default requirement was added to sources, they required the document create permission. This permission must be explicitly granted for new or existing sources.
File locking support was remove from the staging folder uploads. File locking did not prevented incomplete files to be processed.
The sources API now work by having a common set of CRUD operations for all sources while also allowing the sources to register their own unique functionality as source actions. Example of this is the staging folder file preview and staging folder file upload actions.
The refactor also allows unifying background tasks. Sources can now also define callbacks that are executed after the document is created to allow add or changing document attributes in an asynchronous manner.
Support for recoding email Message ID was added. The email source can now record an email Message ID from the header as it is processed into documents. All documents created from the same email will have the same Message ID.
Support was added to rewind the step of the upload wizard.
The API was updated to exclude trashed documents.
The resolved smart link API and serializer were refactored to modernize the implementation.
A permission requirement was added to view resolved smart links. This permission needs to be granted for the smart link and for the document/document type.
The API was updated to return error 404 on insufficient access.
The tag color value was added as a search field.
Celery workers are now launched with the options
[--without-gossip
]{.pre} and [--without-heartbeat
]{.pre} to reduce
chatter and broker resources.
The test view mixin was updated to support multiple test views per test
case. It is now possible to passing arguments to the .add_test_view()
method.
Test view mixin subclasses can now supply their own urlpatterns
. This
is used to allow creation of API test views with minimal scaffolding.
The Spanish locale was split into Spanish (Puerto Rico) and Spanish.
Choice forms now support real-time filtering of items. These forms are used in the add/remove views of groups, roles, ACLs, indexes, and smart links.
Support was added for overriding form buttons. This is important for integrators wanting to add not just additional form button but also remove the default submit button.
Custom submit button label and submit button icons were removed from most views. These are still supported but their usage reduced for a more consistent user experience.
Changing the document type of a document will now cause the workflows for the new document type to be launched automatically.
Event subscriptions was enabled for workflow states, workflow state actions, and workflow transitions.
Using workflows as source for indexes was improved.
MultipleObjectDeleteView
for the delete view.BackendDynamicForm
, a dynamic form for interacting with
backends.mayan.apps.databases.classes.BaseBackend
.FilteredRelatedFieldMixin
to properly detect empty source
querysets. The previous boolean conditional would fail if the
queryset passed was empty as it only used an empty if
statement.
The comparison is now performed using is None
.TestModel
to silence warning.TestModel.save()
method.DOCUMENT_TASK_GENERATE_DOCUMENT_FILE_PAGE_IMAGE_RETRY_DELAY
and
DOCUMENT_TASK_GENERATE_DOCUMENT_VERSION_PAGE_IMAGE_RETRY_DELAY
.TASK_ASSET_IMAGE_GENERATE_RETRY_DELAY
.TASK_GENERATE_WORKFLOW_IMAGE_RETRY_DELAY
.document_states_fast
task queue.DEFAULT_TASK_GENERATE_DOCUMENT_FILE_PAGE_IMAGE_RETRY_DELAY
and
DEFAULT_TASK_GENERATE_DOCUMENT_VERSION_PAGE_IMAGE_RETRY_DELAY
.Using a source now requires a permission. The document create permission must be granted now not only for the document type but also for a source in order to be able to upload new documents.
To be able to use a source to upload new document files to an existing document, the new document file permission must also be granted for the source.
A permission requirement was added to view resolved smart links. This permission needs to be granted for the smart link and for the document/document type.
Trashed documents are now excluded from all API endpoints.
Superusers and staff users are now excluded from being message recipients.
The message edit permission was added. This permission is required in order to change the message read status.
Due to the changes in the index tree generation algorithm and database integrity configuration, existing index trees will deleted during upgrade and need to be rebuild after upgrade.
Environment variable MAYAN_GUNICORN_JITTER
renamed to
MAYAN_GUNICORN_REQUESTS_JITTER
.