The Guide‎ > ‎

Configuring and maintaining automatic data exchange

Bringing data into your system will require an import module that translates IATI’s formats into your own and has a set of rules and configurations that control the behaviour of the import.

Mapping Fields

IATI xml files are structured according to a strict set of rules (a schema) which package the data into elements and attributes. Each of these needs mapping against fields in your system’s database so that the import routine knows where to put the data. This is not just a technical matter. It is important that the proper meaning of the data is understood and reflected in the translation.

Mapping Codes

Both your system and IATI store a lot of data as codes, rather than free-text descriptions. Sector codes are a good example of this. It isn’t necessary for both systems to use the same code so long as the import routine can access a cross-reference map that tells it to replace an IATI code with one of your own codes. Before you first import IATI data these mappings needed to be both thought out and worked out.

Activity Identifiers

When an activity is imported the import module needs to know whether it is new or an update to a record that already exists in your system. Your partner’s project number – or what IATI calls the activity identifier – is critical to this process. Without this identifier properly mapped and populated data exchange is impossible.

Double Counting

IATI allows for the same activity to be reported by different participants AND for the money to be reported as it flows down the delivery chain. In other words the same transaction can be reported twice AND two different transactions may relate to the same money handled by different organisations at different stages of implementation. IATI has rules for tracking this and it is not a problem for importing and storing the data, BUT IT IS IMPORTANT to bear this in mind when designing reports.

Retrieving IATI data from the Datastore
  • Source data, the Registry and the Datastore
    • All publishers store there IATI data on their own websites.
    • The data may be stored in a collection of files, often but not always segmented by country or In a public facing database that is accessed through an API.
    • The IATI Registry contains a catalogue of all IATI datasets (files) available from legitimately registered publishers. In other words, the Registry only contains the index (metadata) not the IATI data itself.
    • The Registry trawls its complete catalogue nightly and visits every dataset on every publisher’s website to check for any changes in the past 24 hours. The change of a single character will flag up a change to the dataset on the registry.
    • Information on these changes is accessible to developers via the Registry’s API.
    • In most instances, however, the best approach is to access the data you need through the Datastore.
  • Accessing the Datastore
    • The IATI Datastore is an online activity-based repository maintained by the IATI Technical Team, accessible through an API, that (in addition to its CSV service discussed earlier) can deliver individual activities in json or xml. The benefits of using the Datastore over the Registry in accessing data are:
    • Selecting all activities for a country from the Datastore will return more records than the Registry because:
    • The Registry indexes country information at dataset, not activity level. A publisher may include an activity for Kenya in an East Africa file, or in an unsegmented file of all its activities. A search of the Registry for Kenyan activities will miss this record.
    • IATI has a rule that an activity should only be published once. If an activity covers both Kenya and Tanzania a search of the Registry cannot discover both countries.
    • The Datastore API allows you to retrieve only those activities that you need that have changed
    • The Registry API will tell you that a file has changed. You will have to process the whole file and determine for yourself which activities in the file have changed.
    • Using the Datastore API you can download a stream of data that only contains those activities that you are interested that have changed since you last checked
  • How often?
    • How often you decide to check the Datastore for changes will depend on:
    • How many publishers’ data you are downloading?
    • How often your publishers are refreshing their data?
    • How often are you and your staff capable of checking new data
Checking the data

Your import procedures should provide you and your staff with an opportunity to review all new data that becomes available. There are two ways in which this may be done:
  • Either by all new and changed records being held during import in a temporary workspace from which you have an opportunity to review and manually commit the changes into your live system.
  • Or by importing the data into your live system, flagging it so that you know which records to check and storing the previous version so that you roll-back an update if you so wish.

Handling New Activities

It is generally recommended that all new activities are automatically accepted into your system and flagged for you and your development partner (this depending on your own practices) to:
  • Check that they accurately reflect what you are expecting from that development partner
  • Review sector and other classifications and change them if necessary
  • Add information on fields that are important to your system – AND other parts of your governments public financial management systems - but that are either not available in IATI or omitted by this development partner
  • Review titles and descriptions, modify them and/or add translations where necessary.
The only piece of data that absolutely MUST NEVER be altered is the activity identifier (or project number) as this maintains the link between the data in your system, the IATI data AND the data in your development partner’s own system.

As argued in the section on Who ‘Owns’ the Data? above, the process of taking over the ownership of the data and modifying it to your own needs is important in making the best use of information management. Your development partner’s version of their data remains publicly available via IATI.

Handling Activity Updates

As an activity progresses your development partners will continue to keep their project and financial management records up to date and modified versions of the activity will be published. The way IATI works is for the whole record, not just those parts that have been changed, to be republished each time. In a typical IATI reporting scenario most of the details of the activity are in place when first reported, and updates contain new financial transactions, activity budgets (forward-looking predictions of spend) and, where reported, information on results.

If your import system is configured to alert you to any and every difference between your partner’s version of an activity and your own record, all the changes you made when the activity was new (as suggested in the previous section), will be flagged up as discrepancies to be resolved every time the activity is updated. This will bog your staff down in hours of unnecessary time wasting. For that reason it is suggested that:
  • Financial transactions, budgets and results are automatically accepted during updates.
  • Other fields that you decide (in your configuration) you will never modify may be added to this list.
  • You NEVER make any changes to these fields. (See the next section on fixing data in these fields)
  • All other fields are NOT updated. If your system is clever enough to alert you to when your partner has made a change (from their original) to one of these field – this is a bonus.

Disputing updated data

The previous two sections have effectively split each activity between two owners: you own the fields that you change, and your partner changes the fields under their control. What happens, then if you detect an error in a budget, or dispute a reported disbursement?
  • You should report the matter to your partner: to their country office or, if not available, to their head office (IATI can help with this if necessary).
  • Your partner should fix the problem in their own system.
  • This change will be reflected the next time they refresh their IATI data
  • And the change will reflected in your own system the next time you import the data.

Comments