Interface file: a data exchange file sent between different systems or different departments, e.g.:
CRM system - CUSTOMER.DAT - invoicing system
bank card system - DAILYTXN.DAT - accounting system
The partner can be:
other corporation
other department
other mainframe
SAP
Oracle
DB2 export
CSV generator web system
Most common batch processing patterns on z/OS. The pattern extracts data from one or more source datasets, transforms the data into the required format, and loads the result into a target dataset.
FIGURE OF ETL PATTERN:
Use this pattern when:
migrating data
creating interfaces
generating CSV files
converting layouts
integrating multiple systems
creating reporting datasets
Extract (read source data):
REPRO + OPTION COPY or
SORTIN
Transform (modify records):
INREC
OUTREC
JOINKEYS
SUM
Load (create target output):
SORTOUT
OUTFIL
REPRO
ETL Optimization Principles:
filter early (fewer records must be processed)
normalize before joining (INREC before JOINKEYS)
aggregate before reporting (SUM before OUTFIL)
EXAMPLE FOR ETL:
Real world examples:
banking (Customers + transactions = monthly statement)
insurance (policies + claims = risk report)
utilities (customers + meter readings = billing file)
retail (products + sales = sales statistics)
Extending a short technical key with information understood by humans.
FIGURE OF LOOKUP ENRICHMENT PATTERN:
Use this pattern when:
replacing codes with descriptions
enriching transaction files
preparing reports
generating CSV exports
joining reference data
Optimization Principles:
join after filtering
join after ID normalization
EXAMPLE FOR TYPICAL LOOKUP TABLES:
EXAMPLE FOR LOOKUP ENRICHMENT PATTERNS:
Real world examples:
banking (status code - status name)
insurance (risk code - risk description)
utilities (meter type ID - meter type name and parameters)
retail (product category ID - product category name)
The most typical mainframe processing model: only the changes are processed.
FIGURE OF PROCESSING PIPELINE OF INCREMENTAL UPDATE PATTERN:
Use this pattern when:
daily batch processing
account balance updates
inventory updates
customer master updates
utility meter processing
insurance policy maintenance
banking transaction processing
Optimization Principles:
always aggregate before joining
left outer join (master records without changes will remain in new master)
process only new changes
The data update phase can happen entirely in JCL but it is more practical in COBOL: the computational business logic is better maintenable there, and the JCL pipeline becomes independent from the business logic.
EXAMPLE FOR INCREMENTAL UPDATE PATTERN:
Real world examples:
banking (accounts + daily transactions = new account state)
insurance (policies + claims = updates policy data)
utilities (customer + meter readings = billing data)
retail (inventory + sales = updated inventory)
For creation of business reports from existing data.
FIGURE OF BASIC PIPELINE OF REPORTS:
Use this pattern when:
monthly reports
management statistics
financial reports
inventory reports
compliance reports
audit reports
EXAMPLE FOR REPORTING PATTERN:
The classic riport has:
header
detail
trailer (footer)
Header:
report title
generation date
generation time
Detail:
Customer totals
Product totals
Daily transactions
Trailer:
Record count
Grand totals
Control totals
Common optimalization:
report after filtering and aggregating
An export is read by a program.
A report is read by a human.
EXAMPLE FOR REPORTING PATTERNS:
For validation of inputs, and proving that processing was done, and the output is consistent.
Use this pattern when:
validating incoming files
detecting layout errors
generating audit trails
producing control totals
documenting batch execution
verifying data integrity
FIGURE OF TYPICAL PROCESSING CHAIN:
Validation of an interface file sent by a partner.
Main idea: in case of bad data, stop early, instead of corrupted processing.
EXAMPLE FOR INPUT VALIDATION PATTERN:
Main idea: checking whether the number of processed records equals with the number of output records, or the output has the predicted number of records.
Used in DFSORT:
SORT+SUM
OUTFIL TRAILER1 with COUNT
EXAMPLE FOR AUDIT REPORT:
Main principle in financial processes:
amount must not be lost (total input = total output)
Comparing 2 outputs. In the result, only the differences must be presented.
FIGURE OF CHECKING FOR RECONCILIATION, WITH SQL EQUIVALENT:
Showing only the invalid records or their numbers.
DFSORT:
INCLUDE COND=(...)
OMIT COND=(...)
OUTFIL (INVALID ACCOUNT: xxx, INVALID DATE: xxx, ...)
In order to document a process, archivation is needed for:
input
output
control totals
execution log
FIGURE OF TYPICAL AUDIT FLOWS:
Use this pattern when:
preserving original data
regulatory requirements
audit requirements
rollback capability
historical analysis
disaster recovery
FIGURE OF ARCHIVAL AND BACKUP PATTERN:
TYPICAL PATTERNS:
There are 2 strategies:
archive the input only (small storage; reports cannot be reproduced directly)
full audit (input, process log, output, report are archived; full traceability; higher storage requirement)
TYPICAL JCL PATTERN:
The aim is to process valid records, while separating invalid records and making riports about them for preparing repairing them.
Use this pattern when:
interface files
customer imports
transaction imports
data migration
audit processing
exception reporting
When invalid record was found, the whole process stops (the input file is rejected).
FIGURE OF REJECT FILE:
The invalid records are collected into a separate output file (exception file).
FIGURE OF EXCEPTION HANDLING PATTERN:
EXAMPLE FOR EXCEPTION HANDLING PATTERN:
Extension of the exception handling pattern.
Saves not just the invalid records (in the exception file) but the errors as well (in an error file).
FIGURE OF ERROR CODE HANDLING PATTERN:
EXAMPLE FOR ERROR CODE HANDLING PATTERN:
Stage 1 (DFSORT VERIFY):
field lengths
field formats
numeric fields
Stage 2 (COBOL or DB2):
Customer exists?
Product exists?
Account active?
Stage 3: (JOINKEYS or COBOL):
Transaction Customer-ID - Exists in Customer Master?
Continuation of the process depends on the number of found errors.
Often in large systems.
FIGURE OF THRESHOLD BASED HANDLING PATTERN:
Use this pattern when:
large daily processing
overnight processing
banking systems
insurance systems
utility billing
long-running COBOL jobs
expensive batch chains
Basic models:
restart from the beginning (ABEND - delete outputs - run again; simple; high costs in case of larger data)
restart from checkpoint (ABEND - last saved state - continue; fast; difficulter planning)
In mainframe world, starting from beginning is often followed (CPU is cheaper, developer time is pricy).
Temporary Dataset Strategy:
datasets are saved after pre-determined steps
in case a process stops, it is re-startable from a previously saved state
Idempotent Processing:
runnable for the same inputs multiple times with getting the same results
Archive First:
archivation happens before process
in case of error the process startable from the archivation point, not from the potencially dinamically changing inputs
Master File Update:
update/creation of the master file happens after the finished process
when the new result is created, it can step in as the new master with high probability of errorlessness
in case updating/creation of the new master happens in every steps, upon ABEND rolling back would be very problematic and costy
one of the most important strategy in batch
Checkpoint:
in case of large batches, a temporary state is created after certain number of processed records
if ABEND happens, the previous state is available, and until the last successfully processed records it is not needed to re-process those
this is more ofter visible in COBOL than JCL
Typical banking solution:
Transactions - Validation - Archive - Processing - New Master - Report
instead of 1 monolithic-gigantic processing step, the process is separated into more steps, with checkpoints between them
in case ABEND upon report, re-executing the processing phase is not needed
EXAMPLE FOR ...:
EXAMPLE FOR ...: