Captain's Log

This blog is a change log of what we implement in DMM. 

Always check if you are using the latest DMM version before submitting a ticket!

v.8.3.1: 2024-04-16

Hotfix related to the Migration and Export features, related to the new optimized queries being used - DMM v8.3.0 introduced this bug which is now solved in this hotfix.

v.8.3.0: 2024-04-09

• See the latest DMM QUALITY ASSURANCE module on the Forge! This module will work as a data source for DMM, with just 16 AOs, so you can deploy it on the PRD environment and have the standard DMM module in the lower environments reading information from it, when migrating data from PRD to lower environments!

• For AO (application objects) optimization, the IPA - Performance Advisor application is separated into a new independent module on the FORGE. If your OutSystems license is impact by the AO consumption and you are not using IPA and just need DMM, this latest DMM version is much smaller!

• PERFORMANCE IMPROVEMENTS: 

  • Optimization for SQL Server queries

  • Added new site property, UseQueryByRows, set to FALSE by default, if set to TRUE will use the old not-optimized queries

  • Other improvements when the query only needs the record IDs, and increasing chunk size to 200.000 (default in new site property)

• FREE Feature changes:

  • Improved validation of the FREE licenses

  • Bulk Data Import feature is now a Professional level feature

  • Site Properties Migration feature is now an Enterprise level feature

• Added support to entities/table names using database reserved keywords

• New feature in the WEB API of DMM now has a mechanism to return the mainExecutionTrackerId and a new service to see the status of the launched execution 

• New site property to choose to sync or not the users (by default yes), SynchornizeUsers default TRUE

• Added support for the site property DBCommandTimeoutSeconds in all operations (insert, merge, update, delete) instead of just the data query

• Added new error code for the DMM's WEB API when a timeout occurs when a better message  

• Increased the available size of the Filters, now a WHERE clause can have up to 50k chars

• Added a corner case where in the Export to Database feature when an entity table is empty and has defined a FK from other entities to it 

• Replaced the site property "useNoneOptimizedOracleCommand" by the new "useQueryByRows"

• Added support to the created_on and last_login information when migrating users

• Added a corner case in situations when using a direct export to database and there is an OutOfMemory error, now DMM will split the exported chunk and retry 

• Corrected an issue in situations when a table has a single record

• Corrected an issue in the Export To Database feature, affecting the filter / where clause not being correctly applied

• Corrected an issue in Oracle databases in th feature of Auto User Mapped Table

• Corrected two issues in the Export feature when using multi-tenant 

• Corrected an issue in the Bulk Data Import feature related to the static datatype  

• Corrected an issue in the Export to File feature when a table had no records 

• Corrected issue in the Migration feature in multi-tenant environments

• Corrected an issue of a possible deadlock in timer operations

• Corrected an issue in the execution dashboard, was showing a wrong elapsed time while a migration is executing

• Corrected an issue when multiple User Mapped Tables with more than one field as composed key were interconnected 

• Corrected an issue with Oracle Sequence resets related to cache

• Improved log information in: the User Mapped Table feature; when an error happens generating the execution log file

v.8.2.0: 2024-02-13

• New DMM Enterprise Feature: automatically set all User Mapped Tables! Now when you are migrating data into a database that already has data in it, and you're in Data Append mode (using the OutSystems runtime connection as destination for example), you can use this feature and automatically set up in all entities the User Mapped Table feature. This will allow DMM to instead of appending data into entities who already have data, detect (by the unique indexes on those entities) if the record already exists in the destination, and if it does, all record FKs pointing to it in the source migrated data will now point to the right record, already existing  in the destination!

• New DMM Enterprise Feature: automatically set tenant filtering! New screen to set up the automatic tenant filter in the Enterprise version of DMM, showing you all the tenant information to make it easier to choose. Also added pagination so as to allow for OutSystems implementations with many hundreds of tenants.

• PERFORMANCE IMPROVEMENTS: 

  • Delta/Incremental migration, related to how data is being read

  • Delete feature, using the OutSystems "delete_rule"; if a reference rule is set to "ignore", there is no FK and DMM does not need to "nullify" the reference preior to deletion

• Show in the Deletion feature a pop-up with the entities / fields that will be affected by the "nullification" made by DMM, when deleting the record that the FK was pointing to

• Support for the User Mapped Table feature in the Export / Import features

• Process improvement, it is now possible to restart migration after the OutSystems timer abnormally stops, restarting the timer "Timer_RunParallelWorker" the migration should resume where it left off

• Allow to use a FK field in the User Mapped Table in the Export/Import features

• Improvements in the Deletion feature algorithm, related to multiple interconnected circular references

• Support for the User Mapped Table feature when there are circular references and self-references or a chain of tables some with the mapping others mapped with natural IDs

• Support for situations when a FK is made to a PK that is also a FK and there was a real data type

• Added support to column/field names that are served names (like PROCEDURE)

• When the OutSystems delete rule is set as "Delete", DMM will treat it as if "Protected" (set to NULL the field)

• New corner case in Delta/incremental Migration, when using the User Mapped Table, and records were ignored in the migration since the tenant didn't exits there, and those records had FKs to them in other records supposed to be migrated

• Cleanup of cache prior to each execution, to avoid not detecting new fields added to entities meanwhile config was done

• Increased the internal cleanup timer timeout

• New corner case supported related to Oracle error messages (for example, ORA-03111: break received on communication channel)

• Fix in the Scheduling setup which wasn't showing up all available configs to schedule

• Fix related to OutSystems error message "index -1 Out Of Range" during a long configuration

• Fix in the EXPORT TO DATABASE feature, when using the Smart User option, and the data is to be exported of the entities OSSYS_USER, OSSYS_EMAIL, OSSYS_CALLBACK and OSSYS_ENTITY_ATTR

• Fix implemented in the BPM Activity data export

• Support in the Delta / Incremental Migration feature when using User Mapped Table with multiple columns

• New site property related to the default timeout in reading information 

• Algorithm improvements in the choice of tables/entities to start operations in highly complex situations for the Migration and Import features

• Visual improvement of the visibility of Static Entities during the configuration steps

• Fixed an issue with the filter preview in the Application level Migration, where the column / field names weren't being shown in the preview pop-up

• Better error messages in the Site Properties Migration feature

• New site property "UserMappedTableTrimOnCompare" to determine if DMM should execute a TRIM command when comparing User Mapped Table information

v.8.1.1: 2023-11-29

Hotfix related to the Migration by Module feature, where you could only search for one application - DMM v8.1.0 introduced this bug which is now solved in this hotfix

v.8.1.0: 2023-11-28

• PERFORMANCE IMPROVEMENTS: 

  • The setup/configuration of a migration, using the Migration => Application Level option, as been optimized; it is now more easy and fast to create configurations for migrations of 200+ apps in one go; the "weight" of the final configuration screen in this feature has also been reduced to speed it up and improve usability

  • For Oracle databases, changes were made in the chunk fetch pagination queries (source queries) for improved performance

  • In the Deletion feature, performance improvements in the plan order resulting in less instructions being executed

  • When using the User Mapped Table on a String type field, but only numbers exist in the records

  • When a field is setup with a Special Action (scramble, anonimize, replace, ignore) don't even read it from the Source [pending implementation for REST Conn]

• WORLD LOOKUPS

  • Added feature to allow for the selection of which lookup(s) the user wishes to update

• Improved security in the DMM license manager workflow

• Improved usability in huge configurations, where there could happen a page timeout in the DMM frontend during the configuration,  now those configuration page actions (Preparation, Continue, Select All) are set with no-timeout 

• Added support for multiple User Mapped Table fields selection when also using the Delta Migration feature

• Added support to define the Tenant field in the User Mapped Table feature

• Added support for situations where a data record information number is large (28+ digits) and the bulk insert/bulk update database library DMM used was throwing a conversion overflow error

• Added support for the OutOfMemory exception in the Export to Database feature, now DMM will split the chunk and retry

• Added support for a new corner case situation using the Smart User Migration where a column name was reserved word

• Added support when executing an Export to External Database, with the feature of Smart OSSYS, added support for multi-tenant environments

• Added support in the Export to External Database feature to destination table names that are reserved words

• Added support in the Export to External Database feature to fields that are over 2000 chars in length (now created as NVarchar in destination, like the OutSystems platform does)

• Added support for situations where different Entity Attributes have the same SS_KEY in OutSystems metadata

• Added support for situations when the PK of an entity is a Varchar, not an auto-number, and so DMM can maintain the same value in destination even when using the Runtime Connection in the OutSystems PaaS Cloud

• Added support for situations where an entity has a self-reference and the entity does not have a PK

• When saving a configuration, the name of the save is now displayed right away in the configuration page where the user is

• In the Export to External Database feature, now it is possible to change the static entity names in the destination (like it already was possible with the business tables)

• Corrected wording in the Test Connection feedback message and in the suggestions / instructions for the Data Deletion feature, where Data Migration was being mentioned

• Improved readability of the list of dependent applications, which are shown when selecting an application for Data Migration for example, removing duplicate apps 

• Added feedback when session expires and the user tries to save a configuration

• Added more information on the Offline Activation page, to make it clearer when a user is using one of the subscribed activation codes and when it is just refreshing the existing one

• Fixed an issue  when splitting a chunk was needed due to timeouts, and an uncaught exception was preventing the split action

• Fixed an issue when exporting binary data where not all records were exported

• Fixed an issue in the Migration configuration screen where sometimes an alert of "Index is out of Range" popped up

• Fixed an issue in the Preview of a Filter during a Migration configuration, where an error was being shown in the Preview window

• Fixed an issue in the Scrambling feature, in situations when a table was empty or had a byte array

• Fixed an issue in the conversion of a type String to BigInt

• Fixed an issue in the Execution Log History where loaded configurations were being shown as ad-hoc defined configurations, and another issue related to the duration column when the execution took more than one day

• Fixed an issue in the Export to External Database feature, related to NULL values that provoked a divide by zero error

• Fixed an issue in the presentation of FKs to BPM tables, now they are presented as any other FK in the configuration screen

• Fixed an issue in the Delta Migration feature when different records had the same checksum and it was throwing an error "An item with the same key has already been added"

• Fixed an issue when using the feature of keeping the Users mapped in memory (for performance) and at the same time using the Smart User Migration (to only migrate users that the selected data depends on)

• Fixed an issue on the WHERE clause preview was not being correctly used to present the preview data

• Fixed an issue on the Data Deletion feature where after successfully ending an execution, some of the visual gauges weren't correctly updated

v.8.0.1: 2023-09-18

Hotfix related to the Export to External DB feature, DMM v8.0.0 introduced a bug, the hotfix makes it so DMM can again create new destination tables as needed

Added support for the site properties ChunkSize, ChunkRecords and KeepChunkSize in the Export feature  

Fixed the UI in the Scrambling type execution, it was being presented as an ad-hoc execution even when it was run from a saved configuration

v.8.0.0: 2023-09-12

NEW FEATURES: 

• WORLD LOOKUPS - now you can use DMM, for FREE, to populate the Forge World Lookups component with data!

• Improved performance in the migration of BPT Emails

• Added filters in the Troubleshooting Details page

• Improved logging related to te license activation/renewal

New supported corner cases and bug fixes related to:

• Added a mapper for OSSYS_BPM_PROCESS when executing a direct Export to Database

• Added support in the Delta Migration for entities with the PK column named "key" (a reserved word)

• Fixed an issue when saving a configuration, the first time it was saved, if the user tried to re-save the configuration (without loading it, the use-case is with the user in the same page hitting the Save Config button again a second time) DMM would save it in a new record as a new different configuration from the previous one

• Fixed an issue when deleting Delta Migration History data

• Improved the DMM execution algorithm in the identification of the SQL instructions execution order in a highly complex corner case

• Fixed an issue when using the Delta Migration feature together with the User Mapped Table feature, when the entity mapping was done with more than 1 column in a single entity

• Fixed an issue when the OutSystems timer restarts (due to some abnormal condition), if there were instructions with a block number above the currently in processing data block, and there was a block split need identified in the current data block, DMM was starting the next block instead of enqueuing the splitted blocks (issue only on Oracle database or when migrating BPTs)

•  Fixed an issue in Oracle data migrations, when the OutSystems timer executing the migration was restarted to some abnormal condition, the issue happened only when the migration was being done without data append (full access to database was available) or the migration configuration included a User Mapped Table entity

• Added support for situations when the user has the role or user set to NULL in the OSSYS_USER_ROLE platform system metadata

• Fixed an issue in the Smart User Migration, when the mappers/FKs to OSSYS_USER were in entities in another database catalog, those FKs weren't being detected

• Fixed an issue in the Data Deletion feature, which was suspending the whole operation, and happened when bulk deleting an entity with millions of records who had to it FKs in other entities not being deleted (situation where DMM will automatically set those FKs to NULL so they aren't pointing to deleted records)

v.7.0.4: 2023-08-02

New supported corner cases and bug fixes related to:

• New OSSYS system tables supported: OSSYS_BPM_PROCESS_DEFINITION; OSSYS_BPM_ACTIVITY_DEFINITION; OSSYS_GROUP_USER; OSSYS_GROUP_ROLE; OSSYS_SITE_PROPERTY_DEFINITION

• Fixed a bug when using together the options of Smart User Migration, BPT migration, and with business data that had FKs to the user groups

• Fixed a bug in the final configuration screen for a migration execution, in entities who didn't have a PK defined, in the option of Filter Preview

• Fixed an issue in the data IMPORT feature, in the execution page where the log screen information gauges didn't correctly show the volume of imported records and execution %

• Fixed a bug in the log where it mentioned an error occurred when the migration operation actually finished OK

v.7.0.3: 2023-07-12

NEW FEATURES: 

• When doing a Direct Export to External Database, you now have the option to filter OSSYS data records so only those referenced by the business data are exported! [DMM Enterprise subscription only]

• You can now use Azure AD login with DMM!

Performance improvements in the Delta/Incremental Migration feature.

New supported corner cases and bug fixes related to:

• Tenant ID with a NULL value

• Export to external database of an entity without any PK

• Export to external database, use automated chunk size with automated re-tries in case of timeout

• Correction in the DMM Web API used for Deletion (feature used for calls executed through external scheduling systems, for example)

• Data/log cleaning timers bug fix

• When doing an Export to External Database, if OSSYS tables are to be ignored, DMM will now not validate those FKs

v.7.0.2: 2023-05-24

More performance improvements related to Oracle database operations and the use of cache for static entities mappers.

New supported corner cases and bug fixes related to:

• Static entities mapper exceptions when value not existing in the destination;

• In Oracle databases, management during migration of entities with PKs that are also FKs to an integer column;

• In the User Mapped Table Feature, issue when the mapped field in the entity is a FK to the OSSYS_USERS;

• Migration of large BLOB fields, through the REST API, where a single record can have tens of MBs in size;

• Added resilience when the OutSystems execution timer "dies" abnormally, and DMM had previously processed data chunks that had to be split due to size;

• DMMRestricted role, added to the saved configuration validation pop-up;

• Delete feature and timeout exception; also related improvements in Email migration feature;

• FORGE download issue, where instead of downloading the solution OAP it would download the Demo OAP (no changes made, but was an issue with previous deploy on the Forge; no reason found by OutSystems support, but apprently solved);

• Correction of an issue in the Import feature for Oracle databases;

• Correction of frontend error in the reporting page script;

• Reporting page changes, where the volumes are now presented in a % to better show the amount that is already processed and several other changes to better showcase what is being done in each different execution type;

• Calls via REST API for reading data of disabled entities.

v.7.0.1: 2023-04-20

NEW FEATURE: API for launching Deletion or Import executions! Now you can launch Deletion or Import configurations through an API call, so you can execute these types of process from another external scheduling manager for example. 

Major internal performance improvements specially in 2 situations:

  1) When entities have a PK that is a FK to the OSSYS_USER entity, and there are hundreds of thousands of users;

  2) In OutSystems Oracle database queries  processes, with a 7x speed increase.

Bug fixes related to the site property "IsToStopExecutionsTimer" and within the new security feature of the REST API (setup of entities as "private" and not available through the API).

v.7.0.0: 2023-04-13

NEW MAJOR FEATURE: Scheduling and Sequencing of Operations! Now you can both schedule operations (Migrations, Exports, Deletes and Scrambles) and sequence them directly on the DMM user interface! You first have to save the configurations you want to schedule/sequence, and then you can select them in the new option.

You can now setup entities as private so as not to be available through the REST API  (in the Settings screen, option "Private Entities" on the page bottom).

Now when Exporting to External Database, you can setup a flag in the configuration so sensitive data from OSSYS_USERS (email, phone and password columns) isn't copied. Additionally, you now have also the option to setup the export ignoring all OSSYS tables (they aren't copied and neither are FKs into them copied).

Fixed several AI Mentor issues related to technical debt.

When using a saved configuration, now DMM gives the option to execute a validation on the configuration versus the current status of the OutSystems database, identifying possible changes and adapting the interface with the changes automatically (deleted entities, deleted fields, added fields, deleted connections, added connections).

Added 2 new mappers to OSSYS_APPLICATION_ICON and to OSSYS_EXTENSION_VERSION, so business tables with FKs into them could be correctly migrated.

Changed Site Properties Migration option so now even if a property already exists it is updated with the source value.

Changed the default of the site property MapperUsersInMemory  to TRUE. This will consume a bit more memory in some situations, but speed up migration speed in all clients!

Visually used italic in the entity selection screens to differentiate Static Entities. DMM does not migrate data from Static Entities, since those should be synched through the platform's standard processes.

When configuring connections to non-OutSystems databases (for use in the Export feature), don't allow those connections to be selected in other types of operations. Now when creating a new connection you have to say if it's an OutSystems database (where there exists all the metadata tables of the platform) or not.

Increased resilience to metamodel issues, as found in some clients:

1. Now if there is a FK to an entity defined in the OutSystems metamodel, but that entity does not exist in the OSSYS_ENTITY table, DMM treats the column as a simple not FK column.

2. If there are 2 exact same entities defined in the metamodel, but one is active and the other is inactive , DMM will now just use the active one.

Internal structure validations in DMM (related to execution processes), with added "verbose level" logs.

Correction for Oracle databases, when using the User Mapped table and the table had other constraints, in some situations an error was being thrown.

Fixed tooltip in the selection of entities for a Migration execution, correctly showing the Module/Application.

Improved error messages, particularly related to OutSystems timers and detection of when they stop running.

Improved the Export process, cleaning up redundant information.

Fixed and issue that was making the frontend not updating after midnight.

V.6.7.1: 2023-03-03

NEW FEATURES: API for Automatic or External scheduling of Migration executions! Now you can launch Migration configurations through an API call, so you can execute this process from another external scheduling manager. 

Now you can configure the Demo App timers to schedule multiple consecutive executions (of Exports or Migrations).

New option, in the Export to Database feature, so you can now automatically set all table names equal to the entity names (instead of the default physical table name in OutSystems, or having to manually set all destination table names).

Added new mappers so now DMM also deals with FKs to the system tables: OSSYS_MODULE, OSSYS_MOBILE_PLATFORM and OSSYS_EXTENSION.

Correction of a bug in the Trial license that didn't show the entity list in the final configuration screen when migrating by application.

Correction of a bug in the Export to Database feature, happening when an entity had a FK to another entity that had a FK to the Users table.

Fix in the Delta/incremental migration, when an entity had a PK that wasn't set to autonumber in OutSystems and using the User Mapped Table.

Fixed an issue in the logs related to tenant/espace not found in migration executions.

Fixed an issue in BPT migration when using the Smart User Migration option.

Fixed an issue 

Fixed an issue in the offline activation of the DMM subscription, where an error message was being presented although the activation went OK.

Improved tooltips in the online/offline DMM activation screens.

Improvement in the DMM logs, particularly related to license not found logs.

v.6.7.0: 2023-01-31

NEW FEATURES: API for automatic scheduling of Export executions! Now you can launch Export to External Database executions through an API call, so you can execute this process from another external scheduling manager.

Increased visibility of Static Entities (UX improvement) in the configuration screens.

New Site Properties available for: cache management, log level, keeping list of Users in memory for better performance of migration of entities with FK to Users, creating FKs constraints when exporting to external databases, execute BLOB migrations in Oracle without the Bulk instruction (performance improvement in some Oracle installations).

In the Export to External Database, if the DELETE_RULE of a FK field is set to Ignore, don't create a constraint in the destination database.

More information in the Activate License screen, logs and messages presented.

Two corner cases related to FK's to OSSYS_USER and bulk timeout exceptions with workaround implemented.

Cleanup of timer's logs.

More History information on the feature of Site Properties migration.

Improvement in the shown entities/apps when in the Migration feature one chooses by "Application" instead of "Module". Now in the final configuration screen, if the choosen application has entities with FKs to entities in another application, the other application is also shown so it's possible to select it's entities and apply the other features (User Mapped Table, Filters, Special Actions).

Removal of the ArdoJSON dependency, to avoid a DLL Hell situation in some OutSystems clients, related to the Newtonsoft.json library dependencies.

v.6.6.3: 2022-12-19

NEW FEATURES: Demo application with timers for automatic scheduling of Migration or Export executions

Correction in a license validation for the Professional version.

v.6.6.2: 2022-12-13

NEW FEATURES: Automatic Timezone offset in Migration executions! Now you can setup an offset during the Migration configuration, so all date fields migrated have that offset applied to the date/hour - very useful when migrating data between infrastructures in different timezones, for example.

Now in the Export to External Database feature it is possible to define the destination table names that DMM will use/create. 

Improvement in log messages related to licensing issues. 

New internal mapper for the platform system table OSSYS_EMAIL_DEFINITION.

v.6.6.1: 2022-11-28

BUG FIX - version 6.6.0 had a bug where FK information wasn't correctly migrated (fields would be left NULL). This version corrects it.

v.6.6.0: 2022-11-18

NEW FEATURE: MIGRATION OF SITE PROPERTIES!

With lots of OutSystems clients going to the PaaS Cloud, moving between datacenters or simply segregating pipelines, we've received several requests to augment DMM to be able to migrate also OutSystems Site Properties - you can do it now just as easily as migrating the data! This new feature is available in the Enterprise subscription.

NEW FREE FEATURE: DATA DELETION is now a free feature, you can now execute BULK DELETE operations without any paid subscription!

We've added the possibility in a Migration to Ignore/Nullify a FK field of an entity.

Optimizations made in the mapping of FKs between entities to speed up the initial part of the migration.

Correction of a bug in Oracle installations during migrations - if you are working in Oracle, please update your DMM version to v6.6.0 or later.

Correction of a bug in the Export to External Database feature, related to the OSSYS_USER system table being exported.

Increased resilience added in the migration process when OutSystems abnormally "kills" threads/timers. DMM will now check if the latest chunk was fully migrated or not (rollbacked), and try to re-start the migration from that chunk (or the next).

v.6.5.1: 2022-10-12

Added the support for Azure SQL connections (just use the standard SQLServer connection parameters in the Settings).

Addition in the new feature Export Directly to non-OutSystems Databases the capabilities of:

• when doing repeated exports, DMM will just update the existing destination data (not creating any duplicate records) with the new origin data (DMM will either update records or create new records in the destination as needed, depending on changes made in the source environment);

• the Special Actions capabilities (to scramble, anonymize, replace or ignore data in the export process);

• the capability of exporting datetime column type (instead of assuming the time type, now the hours data will also be exported);

• when an entity doesn't have active fields, other entities with FK to it will have those fields as simple;

• general improvements on the generated log information for the Export feature, especially when field types change and DMM cannot export (column/table must be dropped in destination by user directly);

Fixed some issues with the new feature Export Directly to non-OutSystems Databases , namely:

• Fixed an issue related to the Create Table command;

• Fixed an issue related to log creation in multiple migrations, and another to binary exports;

New documentation on the online manual about the Site Properties available and the Health Check feature.

New Site Property "IgnoreOSSYSNotFoundEntries", so DMM can ignore PK FK relations to OutSystems platform system entities non-existing in the destination.

Now when a user is exporting/importing to/from files, when deleting the log entry in the Log History section the old files in the filesystem (used by the IIS when doing the import/export) are also deleted.

Improvements on the Delta Migration algorithm, in the checksum validation of text columns and if they were changed since previous migrations.

Correction of some typo's in the DMM logs, and in the numbering of the actions in the logs.

Fixed an issue related to user sync in Oracle, when a user belongs to an inactive tenant.

Fixed an issue when the user pressed the Stop Migration button that DMM seemed to generate several log files (they appeared in the Log History section) but actually only 1 was the valid one and the others weren't available to download.

Fixed a couple issues that happened in cascading User Mapped Tables during migrations between Oracle environment databases.

Fixed a issue in the migration execution result related to counting the number of records migrated (the log information and the dashboard information). Only whole records are now counted. Also corrected the behaviour of the Log History filters.

Changed the name of the final configuration screen - to Final Configuration Screen :)

v.6.5.0: 2022-08-26

NEW FEATURE: EXPORT DIRECTLY TO NON-OUTSYSTEMS DATABASES!

With DMM you can now export your business data from OutSystems directly into any SQL Server/Azure SQL database, DMM will create and maintain the needed database structure and connections. Get your data, even from the OutSystems PaaS, into your analytics solution or data lake directly!

FREE FEATURES: Comparator, Health Check, Bulk Data Import into OutSystems & Data Scramble are now free features, you can use them without any paid subscription!

New site property, SynchronizeUsersData , if set to TRUE makes DMM synchronize the OutSystems Users data (with the default behavior, which is the property being FALSE, DMM just checks if the Username exists in the Destination, and if it does already exist, no user data is changed). All users can be updated, updated columns include MobilePhone, EMail, External_Id, IS_ACTIVE.

Change in the User Mapped Table feature so its behavior by default is now case-insensitive. If needed, behavior can be set back to case-sensitive using the site property "UserMappedTableCaseSensitive".

Change in the User migration so it is case insensitive (match of usernames). Site property available to make it case-sensitive "UsersCaseSensitive ".

Improvement on the DMM internal cleanup timer, it is now able to clean information in chunks. Site property "ClearMapperChunkSize", if it is >0 will delete using chunks of the size of the number of records indicated in the property (default behavior is the same, property set to is 0).

Changes in the DMM interface to be more obvious what features are only available in the Enterprise subscription.

Additional test made in the Health Check feature: validating OSSYS_ENTITY for repeated physical_table_name and ss_key/espace.

Usability improvement in the Export feature, so setting up the User Mapped Table feature doesn't "close" the entity interface automatically.

Removal of site property MaxIteractions (new algorithm doesn't need it).

Added capability / corner case compatibility  in Delta/Incremental migration, when using User Mapped Table and tables without Primary Keys with auto-increment defined, and other edge cases using the User Mapped Table feature.

Correction in migration process, situation with 4+ entities cascading User Mapped tables with FKs in the selected columns. 

Improvement in adding extra logs to the executions, saying the final status of each chunk execution status. Several improvements on the Import feature logs. 

Improvements on the Import feature performance and a bug correction.

Improvements to the BPT/Events migrations including capabilities related to identifying different physical table ID columns. 

Correction in the report screen, which was showing a difference in Elapsed Time and Execution Time; another fix to situations where the log of past executions wasn't being shown, and the Overall counter started counting all executions and not only migrations.

v.6.4.1: 2022-07-14

FREE FEATURE: Comparator feature is now free, you can use it without any paid subscription! This feature offers a comparison report between 2 different OutSystems databases. You can see: the counts of total records per entity; data schema comparisons; relationships/FKs comparisons.

OTHER FREE FEATURES: Health Check, Data Import & Data Scramble are free features, you can use them without any paid subscription!

Updated DMM's terms of service.

Created new site property "MigrateServiceCenterUsers" set to FALSE by default. DMM will no longer synchronize technical users by default, only when this site property is set to TRUE:

• LifeTime synchronizes IT users across environments. In this case, Users from Service Center user provider (typically, tenant_id = 1) don´t need to be migrated.

• If LifeTime is not being used to manage the infrastructure and applications life-cycle, the Service Center manages the IT users in each environment. In this case, if the applications being migrated utilize Service Center as User Provider, you should take these users into consideration to be migrated.

New system mappers for tables OSSYS_BASIC_TYPE and OSSYS_CALLBACK;

Visual improvement on the List of Entities to be migrated, now static entities are now shown in italic to differentiate them from the other business entities.

Multiple performance improvements in Data Migration for bigger (100's GB, millions of users) migrations.

Added more information in the History of executions (if they were OK, and a popup when clicking on the line text with extra information on the Migration execution).

Improvement in the Import feature, to account for the situation of FK constraints on non-mandatory columns.

Two fixes on the Import feature, when the User Mapped table was being used (one of them related to circular dependencies).

Fix in the Export feature, when there were FK references with a depth bigger than 10 connected entities.

Improvement on the Deletion feature, so when a timeout occurs DMM will retry with a lower chunk size.

Show list of relationships/dependencies when selecting Entities for Deletion before launching execution. The dependencies will have their FK fields set to NULL.

Improvement on the Migration feature, allowing the User Mapped Table to be used with a NULL value, and cascade.

Fix on the Filter Tenants feature in the Migration when entities had no records.

Fix on the Migration feature, when setting up the Special Action "Ignore" on a field with NULL value.

Fix on the Migration feature, related to the joint use of Tenant automatic filters and the Smart User Migration options.

Fix on the Migration feature, related to Users that had apostrophes in their names.

Visual fix in the History of executions, where it was showing a duplicate of the execution time/date.

Visual fixes on the Comparation feature.

Added information in the Import operation log if the CSV header information is missing.

Improvements in the Text Connection logs.

Improvements in the standard DMM logs (for example, adding ERROR keyword in some situations where it wasn't clear).

v.6.4.0: 2022-05-17

New DMM Feature: Comparator, now you can use this feature to get a comparison report between 2 different OutSystems databases. You can see the counts of total records per entity; data schema comparisons; relationships/FKs comparisons.

FREE FEATURE: Health Check feature is now free, you can use it without any paid subscription!

In the Migration option, changed the matching of Modules & Entities between the 2 selected environments to just the SSKEY (previously it was matched by Name+SSKEY).

Added the visual indication of Static entities by presenting them in italic in the list of selected entities (Migration, Export, Deletion). In  a future release, this information will also be present in the tooltip of the entity.

Added a new internal flag, SynchronizeOSSYSInactiveEntries. By default, DMM doesn’t migrate any of the table records in an entity if one of the the entity records PK is pointing to a record on an OSSYS table that is inactive, showing an error for the entity (DMM would try to insert null in the PK of that record which would fail, and so by default no records are migrated, even the ones where their PK is a FK to active records in the OSSYS table). This flag SynchronizeOSSYSInactiveEntries will allow the records that are “right” (the FK exists in the OSSYS table) to be migrated. The default flag value is False.

New FK field mappers for OSSYS tables (for example, OSSYS_SEGMENTATIONKIND, OSSYS_SOLUTION), now DMM also maps these record FKs to the corresponding ones in the destination.

Added the feedback in the execution History screen if the execution had any errors.

Log improvements on SQL error messages.

Performance improvement of the Smart User feature, loading the users separately for each chunk.

Improvements in Migration process when DDL connections are available (for example, resetting sequences in empty tables).

Capability to deploy the User Mapped Table feature in tables with over 100 columns (tables where DMM migrates data not only in "horizontal chunks" but also "vertical chunks" to improve performance).

Log improvements on Import errors messages for use-cases of "import from other non-OutSystems sources" (identifying line/column with errors).

Correction in the Delta Migration feature that in a corner situation could duplicate records in the destination.

Correction in the new Health Check feature of a bug in Oracle databases.

Correction in the Import feature, when using the User Mapped Table feature.

Front-end correction in the selection of the modules of an application to Export (if the previous app selected didn't have any modules, in the 2nd app selected DMM would show an error instead of the correct list of modules to select, forcing the user to refresh the page to select modules from the 2nd app).

Correction of an issue in the Smart User Migration feature, for situations where records had FKs to inactive users.

Correction of an issue in the Import feature, when an entity just had 2 fields and 1 of them was a binary field.

Correction of an issue in the configuration of the Export path.

v.6.3.1: 2022-04-04

Added the possibility of using a FK column in the UserMappedTable feature.

UI improvement on the definition of the Special Actions (scramble, anonymize, replace, ignore) to take into account the entity fields' size. 

UX improvement so the entity "block" zone in the UI last configuration screen of the Special Actions doesn't automatically close when setting an action. 

Changed Application mappers between environments, now the same app can have different Names in different OutSystems environments and DMM will still map them correctly between environments.

Change in the execution report, so that when an error happens the non-executed entities are not shown as having ended the execution.

Correction in the execution report, in the BPT values presented.

Improvements on the execution logs to better debug issues.

Added possibility of migrating records of entities whose PK is pointing to an OSSYS entity with inactive records (flag SynchronizeOSSYSInactiveEntries).

Added a link in the Menu so users can return to the initial Dashboard page.

Fixed a 2 bugs in Oracle installations, related to BPT and email migration features (ORA-01795:maximum number of expressions in a list is 1000; ORA-00957: duplicate column name).

v.6.3.0: 2022-03-10

New DMM Feature: HEALTH CHECK, you can now use DMM Enterprise to perform a health check on the metadata of your OutSystems database environment (also validating it against the physical tables structure) with 8 types of different checks!

Visually added (in the configuration) the number of records on the lists, during the selections of apps, modules or entities.

Improvements on the logs of several types of executions (Migration, Export, Import, Delete).

Correction of an error in the User Mapped Table feature, when an entity only had 2 columns (ID+1 other that was mapped).

Allowed in the configuration of mySQL databases the setup of the database schema field.

Correction of an error in the Smart User Migration feature.

v.6.2.2: 2022-02-10

Updated Filesystem module dependency to the latest version to avoid Forge component installation alert message.

Added support (healthchecks)  for cases where active FKs are pointing to inactive entities.

Added validation that the user has a valid session prior to saving a configuration, so as not to overwrite a configuration with null values if the session is no longer valid.

Correction in the entry Dashboard that was producing a "divide by zero" error in some installations.

Correction of a bug in the User Mapped Table feature specific to the Import functionality.  

Change to allow the User Mapped Table feature to be used in tables whose OutSystems defined ID column is not configured as autonumber or the field doesn't have the name "Id".

Added additional mappers to OSSYS tables.

v.6.2.1: 2022-01-14

New DMM Feature: TENANT FILTERING (in DELETION), now you can choose which tenants data you want to delete data from in the Deletion configuration screen, and that filter will be applied to all entities in the deletion configuration automatically, without the need to create individual filters for each entity.

Large improvement in error messages, to make them more comprehensible to the casual user as well as making him able to correct potential issues.

Update on the execution of the timer ClearDataExecutionParallel , so it will only launch if no other execution is running, saving resources.

Correction in the execution counters of the Export feature when used in the context of the Bulk Import Data From External Sources.

New internal mappers to OSSYS tables (like for ex. OSSYS_USER_ROLE).

Improvements in the Deletion feature, when FKs exist that are PKs / IDs in other entities.

Optimization in the configuration of the Scramble/obfuscation special action option, so a smaller subset of data is used in the preview.

v.6.2.0: 2021-12-03

New DMM Feature: TENANT FILTERING , now you can choose which tenants data you want to migrate in the configuration screen, and that filter will be applied to all entities in the migration configuration automatically, without the need to create individual filters for each entity.

Email migration performance optimization.

Added additional protections and process workarounds so DMM can recuperate from OutSystems platform and timers abnormal behaviors, specifically in case of very large migrations (>500GB, in highly complex scenarios) that were having issues with large entities (with dozens of millions of records and with 400+ fields in the same entity).

v.6.1.0: 2021-10-29

New DMM Feature: MULTI-COLUMN USER MAPPED TABLE , now you can use multiple columns in an entity as PK when migrating data (so if that compound unique key exists in the destination, DMM updates the destination record instead of trying to insert a new one).

Changed the "Timer_GetDashboardValues" so that if a migration is running, it will not run and end gracefully (so as not to add any load to a database while a migration is running). Removed the timer "SetDashboardValues" from the module due to redundancy.

Small interface visual changes/corrections, in both DMM and IPA modules.

New logs added in the Export feature, in the Deletion feature and in the Email Migration feature, in situations where due to the processing of very large amounts of data taking too much time the frontend stopped receiving updates and thought the process had stopped when it was actually still running OK on the server.

In some specific situations the migration worker timer was abnormaly terminated after a time by the OutSystems platform, and the DMM process entered a "sleep" cycle in the migration process it didn't recuperate. In this version checks and a recuperation process are added to overcome these platform timer situations and resume the migration work where it was stopped.

Added a new site property, "IsToStopExecutionsTimer", that when set to True stops the DMM migration timer cycle.

v.6.0.0: 2021-09-29

Intelligent Performance Advisor (IPA) openly available to all in a new component module.

New DMM Feature: SMART USER MIGRATION (allows for the migration of just the users/groups that data depends on, instead of forcing migration of all the users/groups as previous versions).

Fixing a cleanup timer queries for Oracle installations.

Correction in REST type connections when accessing the OutSystems SaaS Cloud production environment.

BPT migration process now also allows for tables in other Oracle schemas.

Data Migration feature now detects network errors like "The remote server returned an error: (504) Gateway Timeout" and  "ORA-03146: invalid buffer length for TTC field", and DMM automatically recovers by trying to redo the chunk while splitting it in smaller chunk sizes.

Automatic Timezone conversion in the BPT Migration feature (according to the server defined timezone of Origin/Destination databases).

v.5.3.5: 2021-09-15

Security Update (non-disclosed changes).

On the Historic report, when running a pre-saved configuration (selected from the list of Configurations) show the config name in the report's first column.

In the list of saved configurations, show the last time each config was run.

Changes in Delta Migration option: previously Delta Migration information was cleaned after 48h from the DMM internal tables memory. Now this information is persisted until the user explicitly chooses to erase it in the new menu option "Management > Delta Migration History".

Improvements in labels in the Reporting page.

New "hidden" feature of Simple Migration to allow thousands of entities to be selected in the OutSystems front-end for migration, where previously the front-end was not capable of managing all the objects in the "regular" migration option with all the features.

Detection by DMM of the network error "The remote server returned an error: (504) Gateway Timeout" and retry of the migration chunk with a smaller size in this case.

Corrections in a timer, responsible for cleaning DMM internal tables after 48h, where for Oracle  instances the SQL wasn't correctly built and was showing errors when running.

END OF MAINSTREAM SUPPORT FOR DMM's FRONTEND FORGE VERSIONS ON OUTSYSTEMS 10

New features and bugfixes for DMM's FORGE component will only be released for OutSystems v11 going forward from DMM v5.3.4. (this will be the latest version supporting OS v10).

We recommend our clients to upgrade to OutSystems 11. Please contact Infosistema (dmm.support@infosistema.com) in case you need additional information!

v.5.3.4: 2021-09-02

Group creation: allow the creation of Groups in the Destination when the User that created the Group (in the Source) no longer exists, inserting NULL in the "created by" column of the Group record in the Destination.

Allow for capitalization changes in several OutSystems system tables (databases that are case-sensitive, affecting data & BPT operations).

Allow for a tenant numbered 0 (zero).

Trial license: correction to allow the Trial license to execute more than 5 times (that will be a limitation exclusive to the Free version).

Better management of new records that are created in the Source while DMM is executing a Migration: in previous versions, when parent+child records are added in the Source while the Migration is executing and already migrated the parent Entity, the whole last chunk of child records would fail since some records didn't have the parent migrated. This version will migrate all child records it can, so the last chunk is not all failing in the migration (just the records whose parent weren't migrated). 

Dependencies on OutSystems internal User tables improved: business tables with dependencies or FK to User tables had an issue in specific situations where DMM would try to insert a record (and fail) instead of updating the existing one in the destination.

Database bulk operations timeout: when having a database timeout when executing a bulk operation to the database, DMM will now try to automatically re-execute the operation breaking the chunk in smaller pieces. 

In the Migration feature, when selecting Entities, the SELECT ALL button selecting the whole list and not only the Entities shown in the page. Additional button SELECT PAGE added. Also added pagination on top of the Entities list (instead of just the bottom).

Corrections in the BPT migration feature, in a malformed warning that was presented as an error and in an index.

Refactoring of BPT migration to remove the need of read access to the internal table "USER_TAB_COLUMNS" during BPT migration.

Change max size in EULA confirmation field to allow for bigger usernames.

[Older entries removed; if needed get in touch with us]