Data migration is the main feature of DMM. It is the entire process from setting up a migration configuration to executing it. As the process is separated in those two steps, they can be executed at different times. They can also be executed in different applications that leverage the DMM engine.

We will start by describing the configuration step and it's resulting JSON configuration file (which you can save, inside DMM or outside of it) and afterward, we will detail the execution process.

Migration Configuration

The purpose of performing the migration configuration is to obtain a reusable JSON configuration file that describes a set of migration options. When the DMM engine receives a configuration file, it will analyze it, connect to both databases involved, and execute the necessary commands on the destination database to make the information from the source appear there.

To start your configuration, just click on "Executions > Data Migration" in the top menu of DMM OutSystems Component.

Database Information

In this first step, two dropdowns will appear. The available values for these dropdowns are the configured connection string from the settings screen. You must choose two connections: the one that will represent the source of the information and a second one that will be the destination of the migration. It is always possible to pick "Runtime Connection" from one of the dropdowns to use the environment where Infosistema DMM is installed as a source or destination of the information. Note that Runtime Connection is limited to Data Append as it cannot perform Identity Inserts.

Please note that no validations are made on this step, which means that if you pick a connection for a destination that does not have the required privileges for execution, an error may appear later on. A detailed list of the required privileges can be found on the Requirements page of this manual. Notice that you can have valid configuration files, in the sense that they correctly describe a migration for DMM, which are not executable if the underlying connections are not valid.

You can also upload a previously saved configuration and jump directly to the Final Configuration Screen step.

Finally, you choose if you want to base your configuration at an Application Level where you simply pick the application(s) to migrate or at a Module Level, where you can cherry-pick specific entities to be migrated. 

ATTENTION: If you choose Application Level, you will only see (in the Final Configuration Screen, shown a few steps below) the chosen application and it's entities, but if those entities have connections / FKs to other entities in other non-selected applications, data from those other entities will also be migrated - this is a way for DMM to guarantee data consistency. So to allow you to see the whole list of Entities being migrated, it is best to ONLY CHOOSE APPLICATION LEVEL FOR STAND-ALONE APPLICATIONS OR WHEN SELECTING ALL APPLICATIONS THAT DEPEND ON EACH OTHER, and use Module Level in the remaining cases, since it allows you to see a full list of Entities Selected (even if automatically selected from other applications).

Also, in followup screens you will be able to choose if you want to migrate BPT together with the data.

Depending on which choice you make here (Application Level or Module Level), the BPT migration process execution will be slightly different:

  Option 1) - you choose Application Level. In this case, DMM will migrate all BPTs related to that application, even if they aren't connected to Entities.

  Option 2) - you chose Module Level. In this case, DMM will only migrate BPTs that are connected to the chosen Entities (if migrated data depends on BPTs from other Entities that are not migrated, DMM will show an error in the log for those records).

Module Level

Choose Applications, Modules and Entities

In this step, a list of applications that exist in both databases is displayed to the left. 

When you select one, a list of Modules belonging to those applications is displayed in the center column. 

Picking the desired Module for migration will load the entities that belong to that Module - and clicking on the + (plus sign) next to each one adds it to the Selected list of entities that will be migrated. The Selected list can have entities from any module/application, they are all together in the same list since all will be migrated in this configuration execution. 

Note that the DMM engine will evaluate Entity dependencies when you select each entity (or press the Select All or Select Page buttons) and also add the Entities to whom FKs point to the Selected list automatically. This will assure that the migration will not break any constraint when executed.

Entity configurations

Below the above general migration options, there is a block where all entities that will be subject to migration are listed.  If you chose Application level, you have to expand the Application to see the list of Entities.

Expanding an entity will allow the configuration of how Infosistema DMM will handle it during the migration. There are three possible configurations that can be made on each entity:

Filtering

This allows you to write the SQL syntax that you want to apply. Just write the segment that comes after the WHERE. By default, no filter is applied and the entire table is migrated. Note that the SQL syntax should match your specific database technology for the source database.

Direct JOIN clauses in the filter are not supported, since the filter is appended after WHERE. However, you can filter based on other tables using subqueries with IN or EXISTS. For example:

Using IN:

col1 IN (SELECT id FROM other_table WHERE status = 'active')

Using EXISTS:

EXISTS (SELECT 1 FROM other_table o WHERE o.id = main_table.col1 AND o.status = 'active')

By using subqueries in IN or EXISTS, you can effectively "join" another table in your filter logic.

By default, DMM will *not* cascade the filters. This means that if you apply a filter to an entity, and there are FKs to it in another second entity, in that other entity you should also create a JOIN clause in the filter and add the relevant filter - if you don't, when DMM goes to migrate the second (dependent) entity, and tries to migrate a record that has a FK to a record in the first table that you filtered out, in Data Append it will not find that FK reference and insert NULL in that FK field, otherwise it will use the same FK reference and insert the original value in that FK field, which can lead to a exception.

If you want to cascade the filter, you can use the Propagate Clause button. With this feature, DMM retrieves all entities that have a foreign key referencing the selected entity and applies a clause to ensure the operation only affects records whose foreign key points to the filtered records in the parent entity. 

This operation is recursive, meaning DMM will also apply the same search to those dependent entities, and so on. If DMM detects a circular dependency, a message will appear at the top, and the clause will only be propagated up to the entity where the circular dependency was found.

User Mapped Table

To state that an entity has a unique column that should be used to map between the source and destination. This column will be used as the identifier value for each row and will allow Infosistema DMM to decide if the destination needs to be inserted or updated. The column/field does not need to be setup in OutSystems as unique.

This functionality is most useful when using the Runtime Connection , which doesn't have  identity insert privileges and so by default uses the Data Append mode,  to avoid duplicate data in the destination.

The mapping (validation of the uniqueness of the field) is made on the total amount of data in the table, not on the filtered subset.

Static Entities are a special case and they are themselves mapped internally by DMM, there is no need to configure them - DMM does not migrate data from Static Entities, sice this can and should be done by the OutSystems Service Center functionalities. Even if the IDs change between environments for the static entity records (as is usual), DMM will guarantee consistency through OutSystems metadata information.

  There is no relation between the "Special Actions" (below) and this "User Mapped Table" functionality.

Special Actions

The special actions include one of the following choices you can tell DMM to apply for a specific field/column in an Entity:

●     None - The default value. No special action is made, and data flows normally between the source and destination.

●     Scramble - Randomly generated values are placed in the destination while making sure that the generated values will maintain the semantic structure of the source values. Emails will look like emails, dates will look like dates, and the same is true for ZIP codes, credit card numbers, and so on; Scrambling requires a minimum set of different data in the origin field to be able to detect a pattern and execute the scrambling action (usually 20-50 records is enough).

●     Anonymize - Randomly generated values are placed in the destination, but these values will not have any semantic value. They will just be garbled, meaningless text.

●     Static Value - Define a single value that will be placed in all records written into the destination.

●     Ignore - This will strip the column from the migration, not writing anything in the destination database in this column (the record is copied).

Data transformed by these Special Actions is written on the destination already changed.

When you're satisfied with your configurations, you can choose to save the configuration directly in DMM, download the configuration file and save it for later use, and/or you can continue directly to the execution screen.

Subscription Validation

Depending on your subscription version, a Validation & Subscription Check screen may appear here.

If it does, you can find more information about the process at this link.

Migration Execution

Inside the OutSystems Component, there is a single screen that handles the execution. Whether you've followed the configuration process or you've uploaded a previously saved configuration file, the same screen appears.

On this screen, confirming the operation with the checkbox and clicking on "Start Migration" will start the deletion process, will start the migration process. Under the hood, Infosistema DMM is creating an execution plan and comparing records from each database, performing the required changes on the destination to make it equal to the source database. After the migration is started, Infosistema DMM will jump to the reporting page where you may monitor the execution.