ODF Implementation, Interoperability and Conformance (IIC)

Charter.

It is the intent of the ODF Implementation, Interoperability and Conformance
(IIC) TC to provide a means for software implementers and service providers to
create applications which adhere to the ODF standard and are able to
inter-operate. As such, the purpose of the IIC TC includes the following:

1. To publish test suites of ODF for applications of ODF
2. To check their conformance with the Standard and to confirm their interoperability;
3. To provide feedback, where necessary, to the ODF TC on ways in which the standard could improve interoperability;
4. To produce a set of implementation guidelines;
5. To define interoperability with related standards by the creation of profiles or technical reports;
6. To coordinate, in conjunction with the ODF Adoption TC, OASIS Inter-operability demonstration related to ODF;
7. The IIC TC may also liaise with other standard bodies whose work is leveraged in present or future ODF standards. These include, but are not limited to, the W3C and ISO/IEC JTC1/SC34.

Scope.

Proposed reduction in initial scope of work for this group: Conformance and validation of ODF implementations.

Any test requirement in response to this document shall restrict its scope to items explicit within the standard.

No item in the standard defined as 'implementation dependent' shall be tested.

Notation to be used in a test requirement document.

Notation used.

(inf). An informative paragraph.

(norm). Normative paragraphs.

Deliverables.

This group shall deliver a meta requirement for document(s) which constitute the interoperability and conformance tests for ODF. Each statement shall be either normative or informative.

Interoperability definition.

Interoperability assessment includes the specification of the following: terminology, basic concepts, requirements and guidance concerning test methods, the appropriate depth of testing, test specification and means of testing, and requirements and guidance concerning the operation of assessment services and the presentation of results. In technical areas where there is a conformity assessment methodology and an interoperability assessment methodology, the relationship between them must be specified.

For consideration. A report "Authoring Portable Documents: How to maximize interoperability with ODF", which would be targeted to end-users.  This could certainly contain a statement about the effect fonts have on portability." Possibly as an advisory?

from RW proposal. 080613.

1) A conformance test of ODF documents and implementations, i.e., test the formal shall's and should's, etc.
2) An Acid-style test of ODF implementations, i.e., feature and rendering-oriented, essentially highlighting ODF features that are not widely implemented (or implemented correctly) but are desired (by whom???) 

(2)(a) Identification of similar or applicable work that is being done in other OASIS TCs or by other organizations, why there is a need for another effort in this area and how this proposed TC will be different, and what level of liaison will be pursued with these other organizations. As part of that we will need to explain the relationship between the ODF TC's Accessibility SC and our work.

3) A comprehensive test suite of atomic (single feature) tests

Suggest 4-6 be lifted to read:

The IICTC shall define an initial range of profiles. At least one of which shall cater for accessibility testing.

It is desirable that a base profile is created for each document class(wp, spreadsheet and presentation), which is minimally conformant to ODF.

these profiles shall include accessibility requirements.

4) A formal profile of ODF for portability and archiving, aka ODF/A
5) A formal profile of ODF for browser-based applications
6) A formal profile of ODF for mobile devices 

7) A report on best practices for authoring portable documents
8) A periodic interoperability report on the state of ODF interoperability, with specific recommendations for implementors.

DP proposed amendments are included in the above.

Note. These deliverables quite possibly span this groups work and the IITC

080613.RW:

The base deliverable is a "conformity  assessment methodology" (I've also heard it called a "test requirements  document"), a document that details the requirements of a conformance  testing tool.  This would mainly be a task of collecting and collating the normative provisions of each ODF version, along with provisions in  referenced standards, and putting them in a logical order, noting  dependencies, assigning each one an ID.  It would also define scoring  and reporting requirements for a conformance test.

Definitions.

The following definitions shall be used throughout this document.

Atomic test: A test of a single item. The item being tested could be a single paragraph (or part of a paragraph) of the standard, which is a requirement which may be tested. The outcome of the test is a binary result, pass or fail. The action on pass or failure is not specified within the test. An atomic test may have pre-conditions which must be met prior to running the test successfully.

Document conformance: A (mainly) automated process whereby a document instance is tested to establish alignment with the ODF standard by means of objective testing.

Application conformance: (tba)

Conformance: For any test suite there shall be clearly defined what constitutes a pass. Anything other than a pass shall be deemed a failure against that test suite.

Optional test: A test may be defined as being optional. Any test run shall clearly state which optional tests have been included and the result of the test.

ODF standard: For the purposes of this document the term shall mean Open Document Format for Office Applications (OpenDocument) v1.2

Item under test: Either an entire implementation of the standard, or some defined part of it.

"Strict Conformance: conformance of an implementation that employs only the requirements and/or functionality defined in the specification and no more (i.e., no extensions to the specification are implemented)."  (http://www.oasis-open.org/committees/download.php/305/conformance_requirements-v1.pdf) 

(DP. Note. Since the spec allows for non-ODF namespaced content - see section 1.5, this definition is not constraining)

Interoperability: the ability of two or more IT systems to exchange information at one or more standardised interfaces and to make mutual use of the
information that has been exchanged." http://www.jtc1sc34.org/repository/0856rev.pdf, pg. 145.

 Interoperability: From IDABC's  "European Interoperability Framework":  Interoperability means the ability of information and communication technology (ICT) systems and of the business processes they support to exchange data and to enable the sharing of information and knowledge.

Interoperability:  definition from ISO/IEC/JTC 1 Directives,  Annex
I, <http://isotc.iso.org/livelink/livelink.exe/fetch/2000/2489/186491/186605/AnnexI.html>:

"For the purpose of this policy statement, interoperability is
understood to be the ability of two or more IT systems to exchange
information at one or more standardised interfaces and to make mutual
use of the information that has been exchanged.  An IT system is a set
of IT resources providing services at one or more interfaces."

Interoperability: Source (RW: ISO/IEC/JTC 1 Directives)

"Standards designed to facilitate interoperability need to specify clearly and unambiguously the conformity requirements that are essential to achieve the interoperability. Complexity and the number of options should be kept to a minimum and the implementability of the standards should be demonstrable."

Application-level interoperability

Interoperability among ICT systems established through means other than by adherence to a data format or communications protocol specification, for example where knowledge about another ICT system's interface for a given interchange of information is not fully and publicly specified and such information must be obtained through collaboration among developers or through non-collaborative techniques such as litigation, legislation, or reverse engineering. Another variant of application-level interoperability requires the use of an application's programming interfaces in lieu of writing directly to a file format or communications protocol. JTC 1 Directives, Annex I, prohibits standards and technical regulations from enabling only application-level interoperability by requiring that international standards in the ICT sector "clearly and unambiguously specify the conformity requirements essential to achieve the interoperability." The Agreement on Government Procurement has the effect of extending that requirement to technical (procurement) specifications. Source http://www.universal-interop-council.org/glossary/term/2

document-level interoperability

As opposed to application-level interoperability, document-level interoperability in the standards context is achieved without knowledge of another implementation's unique data formats or communications protocols because a standardized interface for the interchange of information is fully specified by a standard, technical regulation, or technical specification. Source http://www.universal-interop-council.org/glossary/term/70

Test Authority: The person or organisation responsible for executing a test run of an application.

Tester:  The person executing a test run.

The relationship between conformity and interoperability. (and interoperability assessment) is spelled out in the following paragraph from the same page of JTC 1 Directives, supra:

*"Standards designed to facilitate interoperability need to specify clearly and unambiguously the conformity requirements that are essential to achieve the interoperability.* Complexity and the number of options should be kept to a minimum and the implementability of the standards should be demonstrable. Verification of conformity to those standards should then give a high degree of confidence in the interoperability of IT systems using those standards. *However, the confidence in interoperability given by conformity to one or more standards is not  always sufficient and there may be need to use an interoperability assessment methodology in demonstrating interoperability between two or more IT systems in practice."*

The emphasized first sentence in that paragraph explains the relationship between conformity and interop, mandating that the conformity requirements essential to achieve interop must be "clearly and unambiguously" specified in the standard itself. The emphasized last sentence in the paragraph identifies interoperability assessment methodology as something that may be necessary in addition to conformity assessment procedures for edge cases where conformity assessment procedures alone are insufficient to give high confidence of interoperability "in practice," i.e., that interoperability must be demonstrated, whatever it takes to get to that goal.

An acid test:  is a complex  ODF document. It uses features that are not in common use yet, because of lack of support, and it crams many tests into one page.
The aim has been to make it simple for users to check if an application passes the test. If it does, the layout will match this reference. If something is wrong, the layout will be distorted.

(DP. Needs definitions for simple, passes, match, distorted)

Document classes

Examples of Conformance checks.

The following lists some examples of conformance checks that may be needed to address the requirements of the standard.

1. Verify the document file name extensions and/or MIME content type and verify that it matches the contents of the underlying document. An ODT file containing a spreadsheet should be noted, for example.

2. Verify the correctness of the Zip container. Is it actually following the referenced Zip specification?

3. Verify the referential integrity of the package. Does the manifest reference files that don't exist, for example? Are all the required parts present?

4. Verify the Relax NG validity of each of the contained XML documents, pre-processing as needed.

5. Verify additional referential integrity constraints. For example, the content XML typically refers to named styles in the styles xml. These cross-document references need to be checked.

6. Verify the various micro-formats contained in ODF. There are some things that are not easily expressed as a schema type, even using a regular expression. For example, spreadsheet functions, with its hundreds of functions, some with variable arguments, which could take cell ranges, named ranges, or constants as parameters. These are defined in the standard via EBNF. A full conformance test would take each of these attributes and verify that they match the production rules defined by the EBNF.

7. Other recommendations of the ODF standard, even where not conformance requirements. These should be checked, and warnings (not errors) emitted. For example, we have a number of accessibility best practices that could be statically verifiable. Similarly, we can have portability warnings. For example, a spreadsheet can have as many rows as it wished, but for portability we might recommend no more than 64K rows.

8. End-users expect the structure, the named styles, and the runtime behaviour to be preserved as well.  If you load a spreadsheet and find that all the named styles had been converted into direct styling attributes, that vector diagrams had been converted into raster images, and dynamic fields (like table of contents) had been hard-coded, then this would not be good interoperability.

Note. Validity should be explicitly defined when using a validating parser. If an instance content is removed prior to validation the rationale shall be explained in the test specification.

E.g. If the text of the standard allows any element in any namespace at some point in an instance, this may be removed prior to validation against a tighter schema.

Micro-formats.

At a number of points in the standard (tbc, examples @FIXME) micro-formats are used to express a form of grammar, e.g. EBNF. For any tests on these requirements input output pairs shall be stated, and the implementation output compared. (@FIXME. Weak. )

Rendering.

In order to reduce user 'surprises' when confronted by totally variant renditions of the same content it is desirable that where the standard addresses rendering, it be tested.

As an end-user, the interoperability issue that deeply annoys me between different word processors and presentation software is layout changes in the same document viewed or printed from different applications. (That MS Word notoriously doesn't keep this consistent on the same version of Word on the same OS on different printers is hair-tearingly frustrating.) Lines ending up on the wrong page, etc. Pixel-perfect I can live without, line- and word-placement consistency is another matter. - David G. 

Extensions to the ODF Standard.

Suggestion only. When testing, extensions shall be identified and a test report made. Extensions may be recognised by the following criteria.

• Foreign (presumed to mean not in a namespaced defined in the standard)
• Known namespace, invalid to the ODF schema
• Other. Extension recognition to be reported
  

Document hierarchy.

The following sequence of documents shows the traceability needed.

1. This document, Conformance and Interoperability specification.

2. The test requirement. A specification stating how each requirement of the standard will be tested and what the expected outcome of each test.

3. The test plan. A matrix linking the test requirement to the standard and a test program to the test requirement.

4. A test program. An implementation of the test requirement.

5. Expected Test results. A static document, indicating the expected output of executing an abstract implementation of the test requirement.

6. Actual test results. Generated by executing a test program.

7. Overall test result. A comparison of actual and expected test outcomes. A boolean statement of pass or fail. Additional text may provide more information.

By this means the manner in which a part of the standard is tested may be traced, and any test may be linked back to determine what purpose it serves.

Test Types.

A number of different test types may be utilized.

1. Boolean, automated tests. E.g. validating an xml instance against a grammar.

2. Manual intervention. A tester is prompted to check some aspect of a test against an expected value, then enter the result of the test.

3. Extended tests. If the standard is not explicit yet a test implementer wants to add checks, then these tests should reference the standard and provide a warning if the test fails. E.g. accessibility recommendations.

4. Ranged tests. If a test result is expected within some tolerance and the actual result is very close to, but within, the boundary, then a warning may be issued, although the test passes.

Test results shall fully define the item under test, the test environment, the person carrying out the tests and the date on which the test was carried out.

Test requirements.

Test requirements shall be expressed as single items per paragraph. This enables a pass or fail to relate to one single item rather than a number of items.

The test environment shall be clearly defined for each test, either explicitly or by reference, including hardware and software.

Test requirements shall be feasibly implementable. (How to define @@FIXME)

If any ancillary test setup is required to carry out a test this shall be fully defined.

If any test pre-conditions are required they shall be fully defined.

If any test post-conditions are expected they shall be fully described.

If manual intervention is required, this shall be explicitly stated.

It is desirable that tests be grouped into associated functionality where feasible.

Item under test.

It is expected that the item under test will be an implementation of the standard. If a test requirement determines that the item under test needs to be broken down further, this shall be stated explicitly or by reference.

Action on test failure.

In order to gain as much information as possible from testing, a general aim of continuing to test as long as is possible should be taken.  Where tests are dependent this may not be possible. Test run parameters shall enable testers to choose which option from:

1. Halt testing on first failure
2. Report the failure and continue
   

Issues List.

1. Which version of ODF are we testing? 1.0 or 1.2?
2. Full-featured editors available that are capable of not generating application-specific extensions to the formats?
3. Interoperability of implementations mandatory?
4. Interoperability between different IT systems either demonstrable or demonstrated?
5. Profiles developed and required for interoperability?
6. Methodology specified for interoperability between less and more featureful applications?
7. Specifies conformity requirements essential to achieve interoperability?
8. Interoperability conformity assessment procedure(s) formally established and validated?
9. Document validation procedures valid?
10. Specifies an interoperability framework?
11. Vendor-specific extensions classified as non-conformant?
12. Preservation of metadata necessary to achieve interoperability mandatory?
13. XML namespaces for incorporated standards properly implemented?  (ODF-only failure because Microsoft didn't incorporate any relevant standards.)
14. Optional feature interop breakpoints eliminated?
15. Scripting language fully specified for embedded scripts?
16. Hooks fully specified for use by embedded scripts?
17. Standard is vendor- and application-neutral?
18. Capable of converging desktop, server, Web, and mobile device editors and viewers?
19. How do we address the "interoperability" issue with regards to presentation, if we cannot guarantee the same fonts will be available.  Do we make that a condition of testing - all tested platforms have the testing font available?  But does that become a meaningless test for an end user who encounters this but is not aware of the issue?  After all the document will look different, so the "interoperability" test will be a failure for that user.  (as a tech/developer, I see obvious answers here - i.e. install the font.  But to the general public/end user, this may not be so obvious.)-Shawn.

ODF Feature list.

A first take on how a feature list might look, with comments (no more) on testability.

Just taken (randomly) from 1.2 rev 6, Section 4.1, headings.

Notes:

[Warning] Not an error, issue a warning only.