by Tom Orrell with contributions from Beata Lisowska.
At the Friday Seminar that preceded this year’s UN Statistical Commission, Open Data Watch’s Eric Swanson asked me a challenging yet pertinent question following my presentation to the plenary. He asked: “The definition and principles of ‘open data’ are quite clear and simple but the principles of joined-up data are less clear. Can you enunciate five principles of joined-up data that could serve as a practical guide for others?”
This is a question that we at the Joined-Up Data Standards (JUDS) project have been beginning to answer through our discussion papers, blogs and consultation paper. That said, Eric touched on a real gap in terms of concrete guidance when it comes to a commonly recognised list of principles for interoperability – the ability to access and process data from multiple sources without losing meaning, and integrate them for mapping, visualisation, and other forms of analysis – at a global level.
This blog builds on the answer that I gave to the Friday seminar and sets out five core interoperability principles:
Proposed checklist for new standards from our consultation paper
- Is there a clear need and demand?
- Does it duplicate the efforts or compete directly with standards that already exist?
- Are the design of the architecture and individual elements intellectually, logically and methodologically sound?
- Do components (building blocks) within the standard adopt other existing standards wherever possible?
- Is it designed to ensure comparability and interoperability with other standards?
- Will the data be available through open, sustainable and easily accessible channels?
- Is there political buy-in from the institutions that need to produce the data?
- Are timelines for development, implementation and adoption realistic?
- Does the data that can feed into the standard already exist?
- Is it realistic to expect that new data can be produced to feed the standard?
- Does any historical data exist that can act as a ‘rear-view mirror’ for the standard?
Principle 1: Use and Reuse existing data standards
Perhaps the most basic principle that underpins joined-up data is the notion that new classifications – how data are described – and standards – schema into which data are input – should not be developed unless absolutely necessary. Where possible, those seeking to develop a new standard should spend time considering what is already out there and whether an open data standard already exists that can simply and easily be adapted to their needs. This principle is implicitly recognised within our consultation paper, where we suggest a ‘checklist for new data standards’ as a guide for anyone seeking to produce a new data standard. Moreover, any new standard developed must be compatible with existing standards.
Principle 2: Don’t forget metadata
Metadata standards are arguably the most important prerequisite to joined-up data. Metadata includes information on the source of a piece of data, its author, the version being published and the link to the original dataset. Taken together, this information is crucial to ensuring that both machine and human users can discover, identify and contextualise data. Ensuring that machine-readable metadata formats are standardised and used across data producing institutions and bodies therefore greatly enhances the ability of data to be joined-up.
These attributes make metadata particularly important for the official statistics community as it starts to consider how statistical data can be made open by default. As my colleague Beata Lisowska recently put it in another blog, when it comes to metadata, “in essence, we’re really asking: can we trust this data?”
Principle 3: Use common classifications wherever possible
As more and more data are made open and proactively published by governments, international institutions, private sector actors, open standard initiatives and others, we need to make sure that the language used – or the classifications to which data are published – is the same. Often, similar information is classified using slightly different definitions, which hinders the machine-readability and so interoperability of that data. Within the international development sector, it’s crucial that data standards are fit for purpose and actively used, or at least linked to, by all stakeholders producing data.
Classifications of organisations and time formats are two cases in point where the absence of universally agreed definitions can seriously inhibit broad-scale interoperability. The identify-org.net site succinctly explains why the issue of organisational identifiers is important: “If my dataset tells you I have contacts with ‘IBM Ltd’. ‘International Business Machines’ and ‘I.B.M’ – how many firms am I working with?” Unique identifiers would go a long way to overcoming basic semantic challenges like this.
The United Nations Statistics Division has published a registry of classifications that it maintains at UN Classifications Registry. However the list does not include other international classifications such as UNESCO’s International Classification of Education (ISCED), WHO’s International Classification of Disease, or many other important classifications. A comprehensive inventory of all international and relevant national classification systems would be a boon to interoperability.
Principle 4: Publish data in machine-readable formats
For joined-up data solutions to offer real efficiency gains and value, it’s imperative that a machine is able to do most of the hard work in joining up the data. This is already possible but requires many data publishers to change the way they currently publish their data. Publishing data only in PDF format is not enough; data must also be published in machine-readable formats such as RDF, XML and JSON. Publishing in these formats would enable a computer to access, identify and filter data in an automated way, making it far simpler and less time-consuming for data users to put data to good use.
Principle 5: Ensure standards are user-driven
The explosion in open data publication that has taken place over the last twenty-odd years has happened with the key consideration of ‘openness’ at its heart. Whilst this is great and important, openness does not automatically equate to usability. For data to be usable they must be driven by the needs of users themselves. Take the Humanitarian eXchange Language (HXL) standard for example. Its beauty and functionality emanate from its incredible simplicity and ease of use. The process of ensuring that an interoperable standard is ‘usable’ can be a complex one that requires trial and error. Sticking with the HXL example, a linked-data approach was tried and tested but failed given the complexity of user needs in the humanitarian space. A hashtag approach was later agreed, which put user-needs at the heart of the endeavour[1].
These are some of the core principles of interoperability that we’ve uncovered during our research. They offer a starting point for further discussion and we will continue to explore these issues and others with the various stakeholders involved. One thing that we can be sure of already is that to find solutions to interoperability challenges, political will and policy coordination between governments, international organisations, open standard setters and others is key.
Up until the end of March 2017, the Joined-up Data Standards project is inviting feedback on their consultation paper, and will be publishing an updated paper building on these principles and other aspects of their work in the summer of 2017.
[1] For more information see: http://www.alnap.org/pool/files/alnap-innovation-hxl-case-study.pdf, 17.03.17
