Embedded Files
A pipeline is way to automate the retrieval of static data files (as opposed to an API feed of dynamic data). To add a data processor pipeline, you must specify a source, search pattern and a frequency of retrieval.
A pipeline is way to automate the retrieval of static data files (as opposed to an API feed of dynamic data). To add a data processor pipeline, you must specify a source, search pattern and a frequency of retrieval.
The first step the user is presented with a form requiring completion.
The first step the user is presented with a form requiring completion.
You must specify a data source for the data. Once a source is selected the application will display a tree view of the folder and file structure in the cloud storage.
You must specify a data source for the data. Once a source is selected the application will display a tree view of the folder and file structure in the cloud storage.
The next step is to select a file from the folder view and specify the file type (the delimiter).
The next step is to select a file from the folder view and specify the file type (the delimiter).
It is important to specify the correct file type otherwise the application will not let your proceed to the next step.
It is important to specify the correct file type otherwise the application will not let your proceed to the next step.
You can also specify a pattern to match if you want this process to retrieve data as it is saved in this location. The pattern needs to be a sequence of text characters that appear in consecutive order in the file name. The pattern must be general enough to work for all future filenames. Using a specific pattern can also be useful to prevent the pipeline from importing hundreds of files that meet a criteria if you want to test the logic.
You can also specify a pattern to match if you want this process to retrieve data as it is saved in this location. The pattern needs to be a sequence of text characters that appear in consecutive order in the file name. The pattern must be general enough to work for all future filenames. Using a specific pattern can also be useful to prevent the pipeline from importing hundreds of files that meet a criteria if you want to test the logic.
There is also an option to delete the static data that is produced. If you are delivering data to a collection you might not want to duplicate the data in the static data file system. Checking this box will ensure that the files produced by this pipeline will not be saved.
There is also an option to delete the static data that is produced. If you are delivering data to a collection you might not want to duplicate the data in the static data file system. Checking this box will ensure that the files produced by this pipeline will not be saved.
The final required input is the frequency to run the pipeline to check for new data. The application can be set to run every minute, hour, day, or week at a specific time. The time is UTC time. Since this can be a bit confusing it is worth having a separate browser open displaying the UTC time (Google what is the time in UTC zone). Once this is done press Next.
The final required input is the frequency to run the pipeline to check for new data. The application can be set to run every minute, hour, day, or week at a specific time. The time is UTC time. Since this can be a bit confusing it is worth having a separate browser open displaying the UTC time (Google what is the time in UTC zone). Once this is done press Next.
The next step is to apply mapping files on the selected (and subsequent) files. This can get complicated if you are relabelling, removing or adding columns. Use the preview button to show the file structure of the final file. If you make a change to the mapping/operational files used you will need to refresh the preview to see those changes in the file.
The next step is to apply mapping files on the selected (and subsequent) files. This can get complicated if you are relabelling, removing or adding columns. Use the preview button to show the file structure of the final file. If you make a change to the mapping/operational files used you will need to refresh the preview to see those changes in the file.
Once the file structure is produced, you can specify where you want the file to be delivered (external cloud storage) and if you want the file to be added to a collection.
Once the file structure is produced, you can specify where you want the file to be delivered (external cloud storage) and if you want the file to be added to a collection.
Pressing the Add Delivery button lets you specify where you want the file to be delivered and if you want compression. The reload button reloads the view (if you need to create a new folder in a separate browser session) so that you can see folders created after this form was displayed.
Pressing the Add Delivery button lets you specify where you want the file to be delivered and if you want compression. The reload button reloads the view (if you need to create a new folder in a separate browser session) so that you can see folders created after this form was displayed.
The Add To Collection button displays the form that lets you send the data into a collection (a database).
The Add To Collection button displays the form that lets you send the data into a collection (a database).
**Friendly warning**: Only **one** pipeline can deliver data into a collection and once a pipeline is set up to deliver data into a collection you should not modify the fields.
**Friendly warning**: Only **one** pipeline can deliver data into a collection and once a pipeline is set up to deliver data into a collection you should not modify the fields.
Collection structures are preset with standard fields and come customizable fields. In this form you have to pair file fields (on the left) with database fields represented by the orange buttons. Once a file field has been assigned to a collection field it is displayed on the right.
Collection structures are preset with standard fields and come customizable fields. In this form you have to pair file fields (on the left) with database fields represented by the orange buttons. Once a file field has been assigned to a collection field it is displayed on the right.
Note that Quantity, Country and Store are mandatory but have default values. So if these fields are not in your data file then the following default values are applied:
Note that Quantity, Country and Store are mandatory but have default values. So if these fields are not in your data file then the following default values are applied:
- QTY: default value = 1
- Country: Country of the account
- Store: The value specified in the Default Store text box.
Users must also specify the date format. Pressing save closes the form and returns you to the Process Pipeline specification page.
Users must also specify the date format. Pressing save closes the form and returns you to the Process Pipeline specification page.
The next step is to add an additional suffix to the static data file that is produced. This is useful if the file is feeding another pipeline as it can be easily distinguished from the input pipeline. Lastly you can select an email to send a notification to once this pipeline has run.
The next step is to add an additional suffix to the static data file that is produced. This is useful if the file is feeding another pipeline as it can be easily distinguished from the input pipeline. Lastly you can select an email to send a notification to once this pipeline has run.
For the pipeline to be saved and ready you must press Save. If you cancel, changes will not be saved and a warning message is displayed: "Pipeline Will Not Run. Do you want to continue editing it?".
For the pipeline to be saved and ready you must press Save. If you cancel, changes will not be saved and a warning message is displayed: "Pipeline Will Not Run. Do you want to continue editing it?".
Click here to show Manage Pipelines page.
Click here to show Manage Pipelines page.
Report abuse