Data export is a feature available for subscribed DMM customers that allows extraction of entity data from an OutSystems database to a format that can be handled by several other platforms or applications. It is the entire process from setting up an extraction configuration to executing it. The structures and steps are very similar to those of the data migration as it is also separated in two steps: configuration and execution, which can be executed at different times. They can also be executed in different applications that leverage the DMM engine.

Exporting can be paired with the Import feature to allow migration in scenarios where the environments cannot have connectivity to each other. It can also be used for backing up information from applications in the context of a disaster recovery plan.

We will start by describing the configuration step and it's resulting JSON configuration file and afterward, we will detail the execution process.

Export Configuration

The purpose of doing the export configuration is to obtain a reusable JSON configuration file that describes a set of export options. When the DMM engine receives a configuration file it will analyze it, connect to the database, download the data, and produce a zip with the result of the extraction.

To start your configuration, just click on "Executions > Data Export" in the top menu of DMM OutSystems Component.

Database Information

In this first step, a single dropdown will appear. The available values for this dropdown are the configured connection strings from the settings screen. You must choose the connection that will represent the database where you want to extract data from. Note that you can select Runtime Connection to export information directly from the environment DMM is installed in.

Please note that no validations are made on this step, which means that if you pick a connection for the destination that does not have the required privileges for execution, an error may appear later on. A detailed list of the required privileges may be found in the Requirements page of this manual. Notice that you can have valid configuration files, in the sense that they correctly describe an export for DMM, that are not executable as the underlying connections are not valid.

You can also upload a previously saved configuration and jump directly to the preview and execution step.

As the destination connection, you can select:

• Export ZIP file through Browser Download: with this option, after the export process finishes, you can download the resulting ZIP file directly from the DMM Execution Status page.

• Direct Export to Non-OutSystems Database: select this option to export the data directly to a database. This database connection must have administration privileges so DMM can create new tables and FKs as needed, as well as change the record IDs. In the Export to Database, since it has these elevated privileges, DMM always maintains the record IDs - so if you repeat an export process, DMM will see if the record IDs already exists in the destination, and if it does, DMM will update the remaining record fields/columns (if changes happened in the source) instead of trying to insert again the record. Using the Export feature, when you select the DB destination option, DMM will then on the export execution step create the database tables as needed (with the names chosen, you can put the OutSystems Physical table names, the entity names or define your own) and insert/update the records (by their ID). To directly export to a database you must have defined previously the database connection, to a non-Outsystems DB, in the DMM settings - currently SQLServer drivers are available for such external connections (if you need other type of connection, please open a ticket with DMM's support). You can repeat the export and DMM will insert/update records as needed - but if a record is deleted in the source it will not be deleted in the destination, so this will be slightly different behaviour to the Delta/incremental option you can find in the Migration feature.

• Export ZIP file to SFTP Path: with this option, the result zip will be sent to a SFTP connection.

• Export ZIP file to Mapped Folder: with this option, the result zip will be copied to the specified path, which can be a mapped folder. Consult this link for more information about user permissions . 

Finally, you choose if you want to base your configuration at an Application Level where you simply pick the application(s) to export or at a Module Level, where you can cherry-pick specific entities to be exported. 

Application Level

Choose Applications

In this step, a list of applications that exist in both databases is displayed to the left. When you select one, it will be added to the list of applications to be exported.

Module Level

Choose Applications, Modules and Entities

In this step, a list of applications that exist in the database is displayed to the left. When you select one, a list of eSpaces belonging to those applications is displayed in the center column. Picking the desired eSpace will display all its entities. Clicking on the plus sign near each entity will add it to the list of selected applications to be exported. Note that the DMM engine will evaluate its dependencies and also add them to the selection. This will assure that the export will also bring in all related information.

Review fields to export and special actions

• In the final step of the configuration phase, you can review all entities and their respective fields to better understand what's being exported.

There is also a configuration called "Only Export Structure." This will cause DMM to not export any data and instead produce a clean ZIP file with the structure of the data. This is useful for importing data from legacy applications. The process would be:

●     Export "Only Structure" from the destination OutSystems application to a zip file containing several Excel files that describe the expected structural format

●     Use an ETL process and fill those Excel files with data from the source application

●     Use the import feature to fill the data into the OutSystems application

There is also the Option of "Export Plain", this will make DMM create the export file (the one who is downloaded) without any OutSystems metada - it can be useful to load the information as-is into a non-OutSystems solution, but this way the file isn't ready to be loaded back into an OutSystems with DMM's Import feature.

When expanding an entity, you will find two fields: a top one where you can define a custom filter for your export and a second one where you can specify special actions.

For the filter, you just need to write the SQL syntax that you want to apply. Just write the segment that comes after the WHERE. By default, no filter is applied and the entire table is exported. Note that the SQL syntax should match your specific database technology for the source database. 

Direct JOIN clauses in the filter are not supported, since the filter is appended after WHERE. However, you can filter based on other tables using subqueries with IN or EXISTS. For example:

Using IN:

col1 IN (SELECT id FROM other_table WHERE status = 'active')

Using EXISTS:

EXISTS (SELECT 1 FROM other_table o WHERE o.id = main_table.col1 AND o.status = 'active')

By using subqueries in IN or EXISTS, you can effectively "join" another table in your filter logic.

By default, DMM will *not* cascade the filters. This means that if you apply a filter to an entity, and there are FKs to it in another second entity, in that other entity you should also create a JOIN clause in the filter and add the relevant filter - if you don't, when DMM goes to export the second (dependent) entity, and tries to export a record that has a FK to a record in the first table that you filtered out, it will use the same FK reference and insert the original value in that FK field, which can lead to a exception.

If you want to cascade the filter, you can use the Propagate Clause button. With this feature, DMM retrieves all entities that have a foreign key referencing the selected entity and applies a clause to ensure the operation only affects records whose foreign key points to the filtered records in the parent entity. 

This operation is recursive, meaning DMM will also apply the same search to those dependent entities, and so on. If DMM detects a circular dependency, a message will appear at the top, and the clause will only be propagated up to the entity where the circular dependency was found.

The special actions include one of the following choices:

●     None - The default value. No special action is made, data flows are placed in the zip as they are in the source database.

●     Scramble - Randomly generated values are placed in the resulting CSV files while making sure that the generated values will maintain the semantic structure of the source values. Emails will look like emails, dates will look like dates, ZIP codes, credit card numbers, and so on.  To detect a scrambling pattern, DMM must find  enough data records (usually ~20-50 records). If not enough records are available in the database, an error message is shown  "Attribute does not have enough data to be scrambled".

●     Anonymize - Randomly generated values are placed in the resulting CSV, but these values will not have any semantic value. They will just be garbled, meaningless text.

●     Static Value - Defines a single value that will fill the entire column.

●     Ignore - This will strip the column from the migration, not writing anything in the destination database

When you're satisfied with your configurations, you can download the configuration file to save it for later use or you can continue directly to the execution screen.

Subscription Validation

Depending on your subscription version, a Validation & Subscription Check screen may appear here.

If it does, you can find more information about the process at this link.

Export Execution

Inside the OutSystems Component, there is a single screen that handles the execution. Whether you've followed the configuration process or you've uploaded a previously saved configuration file, the same screen will appear. 

On this screen, confirming the operation with the checkbox and clicking on "Start Exporting" will start the deletion process. Under the hood DMM, is creating an execution plan and extracting records from the database, then writing them to the produced CSV Files (or to the Destination Database, if that was your configuration).