Version 4.7.1

Added

Backend

• celery task for periodic checking of database-clickhouse synchronization was added
• automatic resolving of differences between database and clickhouse was added into the CLI command

Fixed

Frontend 

• 'unknown' state in data presence view in harvesting dialog was fixed

Backend

• incorrect state when doing resync in clickhouse after error was fixed
• some missed cases where detection of database-clickhouse difference did not work correctly were fixed

Version 4.7.0

Added

Backend

• add missing reader for IR_M1 reports in table form
• update CLI script check_report_type_dimensions to also create missing COUNTER reports

Changes

Backend

• allow deleting of user accounts from Django admin by allowing deleting of impersonation logs
• use constant memory mode when creating Excel exports in reporting

Fixed

Frontend

• do not try to translate '-- blank --' id into title name
• show import errors in dashboard SUSHI overview widget
• fix sorting by title attributes (such as ISSN, ISBN, etc.) in reporting
• fix locking of SUSHI credentials by adding the missing can_lock attribute to the API endpoint
• fix situation where it was not possible to leave impersonation of a user without an assigned organization
• fix regression in sorting by explicit dimensions in reporting
• reset pagination when report is changed in reporting

Backend

• fix celery task name for cleaning obsolete platform-title links
• fix incorrect behavior of harvest planing after credentials are verified
• fix computation of data-presence by using ImportBatch presence rather than FetchAttempt status
• properly store owner attributes when saving a new report
• skip custom platforms from other organizations when importing SUSHI credentials using CLI

Version 4.6.2

Fixed

Frontend

• fix regression which caused automatic harvesting of new SUSHI credentials to be off by default
• fix styling inconsistencies of platform and title detail pages if user got there directly without visiting a different page before

Backend

• fix planning of new intentions for last month not respecting already harvested data
• when reporting SUSHI status from the API, prefer fetch intentions with import_batch over newer ones without it.
• the code for merging titles was sped up to avoid "timeouts" in celery jobs

Version 4.6.1

Fixed

Frontend

Version 4.6.0

Added

Frontend

• SUSHI credentials are newly marked as verified on first successful use. Automatic harvesting is only enabled for verified credentials.
• user role "consortial user" was added - it has the same access level as normal user but can see data for all consortium members without them being explicitly assigned
• possibility to delete all platform usage data from the platform detail page was added
• a new tab 'Data management' was added to the platform detail page
• more options were added to the data range selection widget

Backend

• support was added for retrieving report type information from knowledgebase
• a CLI script was added to move credentials with all the associated data into a new custom platform

Changes

Frontend

• raw data export functionality was moved to the 'Data management' tab
• SUSHI credentials export now respects the selected organization and platform
• the SUSHI related dashboard widgets are no longer shown to non-admin user
• the default displayed data period was changed from 'all' to 'current + last two whole years'

Backend

• retries of failed SUSHI harvesting were modified:
• only automatically created harvest are retried
• retries are done only for credentials which were verified (successfully used for harvesting)
• retries are done for all non-breaking fetch attempt states
• partial data downloads are marked as containing no data until imported at the end of the retry period
• validity of invitations and password reset tokens was extended from 3 to 10 days
• the user detail API endpoint was extended to include the is_staff attribute
• automatically created harvests are planned a few days before month start, not whole month

Fixed

Frontend

• the platform detail page of a platform not connected to the current organization no longer displays error messages and contains more information about the platform
• fix missing metric (and potentially other objects) names in exports from the reporting module
• all import batches are shown in the delete confirmation dialog, not just the first 10

Backend

• user preferred language is always explicitly selected on the backend to ensure correct data localization
• marking SUSHI credentials as broken no longer lead to duplication of stored extra parameters
• access rights to the SUSHI credentials API endpoint was fixed to only include admin users
• the django admin list interface for import batches now shows date instead of user

Version 4.5.0

Added

Frontend

Backend

Changes

Frontend

• the v-charts library was replaced by vue-echarts (the former was not maintained anymore)
• information about interest definition was moved to the Content section of the menu
• when displaying raw harvested data, additional records (such as interest or materialized data) are not shown anymore

Backend

• empty data in SUSHI response without a corresponding exception are treated as having exception 3030
• Celus newly tries to re-harvest data from failed attempts (regardless of the error) if the credentials were successfully used to harvest some data lately
• reporting speed was optimized in case when non-zero rows are not requested, most significantly those with titles in rows
• cli commands for reimport and subsequent analysis were extended and improved

Fixed

Backend

Version 4.4.5

Changes

Backend

• lowercase 'x' as last character of ISSN is replaced by upper case 'X'
• hcube was update to speed up deleting of data from Clickhouse in newer versions

Fixed

Backend

Version 4.4.4

Added

Backend

Fixed

Backend

• merging of titles was fixed not to merge titles with the same name but without any common ID
• a cli script for merging title out-of-band was added

Version 4.4.3

Fixed

Frontend

• reporting - export of reports split by year and month was fixed
• users logging in by EduID are no longer warned about missing email validation

Backend

Version 4.4.2

Added

Frontend

Backend

• cli command for assigning reports to platforms was added
• cli command for comparison of the current database with an older version was added to help in finding differences after data reimport

Changes

Frontend

Backend

• cleaner method for recaching aggregate queries in AccessLog table was implemented
• detection of last interest change was optimized by remembering the last change time
• django admin was extended to support reimport of manual data uploads
• email about preflight errors is sent to admins asynchronously
• the reimport_data cli command was improved

Fixed

Frontend

• placeholder text for data upload was made more generic to reflect really supported formats
• when showing status of harvest, full data are preferred over partial data
• all fetch intentions are now shown when the mode is set to 'All', not just latest ones
• when validating the SUSHI URL, check /reports/ not /report/

Backend

Version 4.4.1

Added

Frontend

Backend

Fixed

Frontend

Backend

• regression of pycounter not being properly updated in 4.4.0 was fixed
• superfluous import batches created for attempts with 3030 error code when migrating from very old Celus versions were removed

Version 4.4.0

Added

Frontend

Changes

Frontend

Backend

• SUSHI exceptions 3050, 3060, 3061 and 3062 do not cause import error if data is also present in the response
• import of multi-month files was sped up considerably
• reimport script was improved with more verbose output and other changes
• obsolete code related to Dimension.type was removed
• several unused fields were removed from ReportTypeSerializer
• API for manual data uploads was optimized for speed and data size
• API for finding existing data for new harvests was sped up
• SUSHI APIKey maximum size was increased to 400 to support some strange platforms

Fixed

Frontend

Backend

• several small fixes and optimizations in Django admin
• pycounter was updated to fix error in date handling in specific situations (when the date range in header differs from the actual data)

Version 4.3.3

Added

Frontend

Version 4.3.2

Fixed

Backend

Version 4.3.1

Added

Frontend

• preflight data were extended to include:
• information if a metric was previously used for the selected report type
• comparison of imported data with average of previous year and the same month in previous year

Backend

Changes

Frontend

• reporting improvements:
• make visibility level tooltip part of the selector
• when starting new report, select something by default to make it 'runnable' immediately
• add list of organizations to metadata if organization is not in rows, columns, split_by or filtered

Backend

Fixed

Frontend

• SUSHI status dashboard and monthly overview were fixed to correctly display waiting harvests (COUNTER exception 3031)
• regression in formatting of log data in expander of list of attempts was fixed

Backend

• error in harvesting when dealing with in-memory files was fixed
• ignoring Severity (see above) fixes errors when importing data with exception 1011 which had incorrect severity

Version 4.3.0

Added

Frontend

• new view of Interest definition was added (available under Administration/Interests)
• progress bar was added to reporting when list of individual parts is loaded
• manual data uploading was improved
• it is now possible to return to data upload from the 'preflight' step
• error reporting was improved to handle some specific cases better
• progress indicator was improved

Changes

Frontend

Fixed

Frontend

Backend

• when downloading large files, make sure they are really flushed to the disk before trying to parse them
• race condition in locking of manual uploads was fixed - fixes manual data sometimes not being imported and getting stuck as importing.

Version 4.2.0

Added

Frontend

• reporting was extended to include
• splitting reports by a specified attribute into several files/sheets
• export to Excel format
• metadata about the export in a separate file/sheet
• export now has name taken from the report which created it
• show data extracted from SUSHI header (organization name and 'Created-By') together with each download
• harvest list UI was improved with a platform filter, info about currently processed downloads, etc.
• when uploading data manually more heuristics about used metrics are presented
• processing of manually uploaded files was optimized for speed

Backend

• titles newly store proprietary IDs and URIs extracted from COUNTER data
• the Platform dimension is newly extracted from COUNTER reports and stored in the database
• extract data from SUSHI header and store them with harvested data (data reimport is required to fully take advantage)
• Celus version was added to the prometheus exporter
• cli script was to check if report type dimensions match reader classes

Changes

Frontend

Backend

• processing of manually uploaded data was moved to the backend process. This fixes timeout issues when processing large files
• it is now possible to disable automatic creation of metrics on a per-installation basis
• it is now possible to limit the set of metrics allowed for a report
• the way how incoming titles are compared with the database has been changed to merge titles which only differ in some identifiers being empty (data reimport is required to fully take advantage)
• normalization is newly applied to ISBNs
• normalization and cleanup was added for ISSNs and ISBNs which ensures length restrictions
• the django-error-report module was removed

Fixed

Frontend

Backend

• synchronization with knowledgebase was fixed to prevent creation of duplicated records
• flush file to disk after it was downloaded - may fix situation when a large file is downloaded by the sushi celery worker and it appears as incomplete in the import worker

Version 4.1.1

Added

Frontend

Changes

Frontend

Fixed

Frontend

• It is now possible to reharvest data for months where the data was empty
• Very slow parsing of some manual data uploads caused by slow date parser was fixed

Version 4.1.0

Added

Frontend

• a new page for management of annotations was added
• it is now possible to delete manually uploaded data from the "SUSHI overview" view
• deleting of data was improved to show more details about deleted data
• when manually uploading data, conflicting existing data are detected and presented to the user with the option of deleting them

Backend

• encoding of manually uploaded files is automatically detected
• new model FetchIntentionQueue was introduced to bind together follow-up FetchIntentions created after SUSHI error 3031 or 3030

Changes

Frontend

• the SUSHI management page now uses the whole page even on large screens
• the "Save" button in SUSHI editing dialog is disabled when save is in progress to prevent double-clicking related issues
• the SUSHI overview dialog always shows one more year than there are existing data for

Backend

• intervals of re-checking not-yet-available recent data by SUSHI (exception 3031) were changed from 1,2,4,8,16,32 days to 1,2,4,8,8,8,8 days (for 3030 the intervals remain at 1,2,4,8,16,32 days)
• automatic harvesting ending in SUSHI error 3040 (partial data) newly leads to retries similar to 3031
• several required Python packages were updated - most notably Celery
• feature: added brin index to ImportBatch.date
• performance of the API endpoint for retrieving report-views was improved
• a timestamp field was added to the FetchIntention model
• a date field was added to the ImportBatch model
• unused system_created field from ImportBatch was removed
• detection of "broken credentials" was extended to cover more SUSHI statuses
• the create_organization_api_keys CLI management command was improved
• an empty ImportBatch object is now created after the last attempt in a queue of attempts with the 3030 or 3040 SUSHI errors
• manual uploads model was modify to use several ImportBatches - one for each month of data
• when the connected knowledgebase source no longer exports a platform, its knowledgebase related data are cleared up
• SUSHI exceptions of type Info are now correctly processed and stored

Fixed

Frontend

• no longer relevant ongoing requests to the backend are cancelled on the Harvest and SUSHI overview pages
• duplicated FetchIntentions are now properly handled on the SUSHI overview page (no more cases where successful attempts are also marked as scheduled for future harvesting)
• the "Top 10" widgets on the dashboard now properly react to change of the displayed time range
• menu is automatically expanded when the user navigates to a child page without using the menu (for example to the SUSHI management page from the dashboard widget)
• Overlap of text and widget in the test harvesting dialog was fixed

Backend

• data migration fixing several problematic ImportBatch states was added
• code to determine when interest should be recalculated when interest definition changes was optimized
• skip index for import_batch_id was added to Clickhouse to improve deletion speed
• fix cases where celery did not properly close open sushi-attempt related files
• gracefully handle situations when Item_ID is null in SUSHI C5 reports

Version 4.0.1

Added

Frontend

Backend

• a slower but more memory efficient method of synchronization between Django database and Clickhouse was added
• more information (username, etc.) was added to the error message sent to admins when an error occurs during manual data upload

Fixed

Frontend

• activation and deactivation of shown interest types on platform page was fixed (regression introduced in 4.0.0)
• active reports are properly synchronized when credentials are edited after being selected for harvesting
• fixed - manual upload of incorrect data always ended up displaying message about UTF-8 encoding (regression introduced in 4.0.0)

Version 4.0.0

Added

Frontend

• when harvesting data using SUSHI, overview of existing data is shown and months with existing data are automatically skipped
• it is now possible to delete SUSHI data from the "Overview" dialog in SUSHI management
• months with already planned harvesting are marked with a badge in the "Overview" dialog
• rows with zero usage are removed from advanced reporting output by default, it is possible to configure reports to add them
• upload of another file is offered in the last step of manual upload
• outstanding HTTP requests to the backend are cancelled when user switches pages, thus improving frontend responsiveness (especially when quickly leaving dashboard page after initial load)

Backend

• support for synchronization of log data into Clickhouse was added
• management command was added to remove titles without any associated usage data
• management command was added to remove orphaned files from sushi harvesting
• management command was added for comparing the main database content with Clickhouse

Changes

Frontend

• menu sections for reporting and overlap analysis were renamed based on user feedback
• selection of organization and platforms when adding new SUSHI credentials was optimized for space
• downloads resulting in SUSHI exception 3030 (no usage) are now shown with the "empty data" icon rather than the "error" icon
• when switching between tabs on the platform detail page, the chart data get automatically reloaded to make sure they are up to date
• more informative error messages are shown when incorrect encoding it used for manually uploaded CSV files
• javascript libraries were updated

Backend

• older data harvested using SUSHI for a period longer than one month were split to individual months for easier management (deleting and re-harvesting)
• CSV "sniffing" size was increased to improve accuracy of format dialect detection
• python libraries were updated

Fixed

Frontend

• when adding new platform, warning of potential clash with existing platforms now works correctly
• remove background request for unused chart data on the dashboard page
• manual upload of data is no longer permitted if all organizations are selected

Backend

• rare database constraint error when creating new title records during data import was fixed
• properly handle exceptions coming from the pycounter library when importing C4 data
• check for correct site.domain configuration were added when sending user invitations
• disabling of cachalot in test settings was fixed
• ensure than only enabled languages are available for user selection in Django admin
• export of empty data no longer crashes with "KeyError"