Canadian Centre for Energy Information Web Data Service User Guide

The Canadian Centre for Energy Information (CCEI) is a convenient one-stop virtual shop for independent and trusted information on energy in Canada. It houses resources on all things energy-related, including production, consumption and international trade.

The CCEI is providing a web data service to retrieve data and metadata from datasets included on the CCEI website. Available datasets will be added to the web data service on an ongoing basis.

The CCEI Application Programming Interface (API) is compliant with the Statistical Data and Metadata eXchange (SDMX) RESTful API Specifications and you can find useful documentation and a cheat sheet on the SDMX Technical Working GroupOpens in a new tab's website. This guide refers to terms and technical elements as defined in the standard.

With the SDMX API, users can request various dimensions as a parameter in the Uniform Resource Locator (URL) such as physical flows, sectors or products. The resulting data can be downloaded as a comma-separated value (CSV), JavaScript Object Notation (JSON) and Extensible Markup Language (XML).

The Statistical Data and Metadata eXchange standard

The SDMXOpens in a new tab is an International Organization for Standardization (ISO) standard (ISO 17369:2013) that has been designed to describe statistical data and metadata. Statistics Canada and many other statistical organizations have adopted this standard for statistical data exchange.

It is used to define a common data model and uses standardized data access interfaces, known as APIs.

Data model overview

All elements of a specific domain can be described using concepts. These concepts can be broken down into dimensions, attributes and observation values. Grouped together, they form a concept scheme.

In the SDMX, the data are managed using dataflows, which each contain a set of data with a common structure. These structural metadata are represented using a Data Structure Definition (DSD), which can be reused across dataflows. Imagine the dataflow as a container for data with a common structure (similar topics are usually grouped, with the structure based on a hypercube model).

A set of dimensions are used to define the data structure definition and the dataflow. In CCEI, they may vary from one data structure to another. Common concepts used as dimensions can be:

• frequency
• geography
• product
• industry
• time period.

To maintain a controlled list of codes, the allowed values for each dimension are kept in a codelist. This means that, for example, the codelist connected to the frequency dimension would contain the list of allowed frequencies. Codelists can be used by several structures.

Each datapoint has a unique identifier called a key, which represents the combination of values in each dimension. The order of the dimension in the dimension list is important in building the key. The key consists of the list of dimension values for each dimension separated by a dot (.) and lists the dimension by the order it appears in the DSD dimension list. For example, the key ".CA.142.211" represents filtering the dataset by the Geography=Canada(CA), the Product=Natural gas (142) and the Industry=Oil and gas extraction (211).

An observation can have attributes attached to it such as the unit of measure, comments, and footnotes.

SDMX can be represented using different access formats, such as:

• JSON for data and structures
• XML for data and structures
• CSV for data.

Statistical Data and Metadata eXchange data web service

All the data stored for census profiles can be retrieved using the query string described below:

protocol://wsEntryPoint/resource/flowRef/key?parameters

Syntax definition

protocol

The web service is available over Hypertext Transfer Protocol (HTTP), http and https.

wsEntryPoint

The web service entry point is available at the same location of the SDMX data and metadata entry point, i.e., "energy-information.canada.ca/sdmx/rest/".

resource

The resource for queries is data.

flowRef

A reference to the dataflow describing the data that must be returned.

Syntax refers to the identifier of the agency that is maintaining the dataflow, followed by the dataflow identifier, then the dataflow version, each separated by a comma (,).

For example, in the reference "STC,DF_38100139,1.0", the agency is STC, it maintains the DF_38100139 dataflow and the version of the dataflow is 1.0.

The important component above is the dataflow name (DF_38100139).

If the flowRef value has only one element, it is referred to as the identifier of the dataflow. The identifier for the agency will default to "all", while the value for the dataflow version will default to "latest".

If the flowRef value has two elements, they are referred to as the identifier of the agency and the identifier of the dataflow. The version will default to "latest".

key

As explained in the data model, the combination of dimensions allows statistical data to be uniquely identified. In the SDMX, this combination is known as a series key—it is needed in the key parameter.

Let's say, for example, that the dataflow DF_38100139 values can be uniquely identified by the following:

• frequency at which they are measured (e.g., every year = code A)
• geography being referenced (e.g., Canada = code 1)
• physical flow (e.g., Direct energy use = code 1)
• good and service (product) (e.g., Support services = code 14).

To build the series key, you must take the value for each of the dimensions (in the order in which the dimensions are defined in the DSD) and join them with a dot (.). The series key for the example above therefore becomes A.1.1.14.

Wildcarding is supported by omitting the value for the dimension to be wildcarded. For example, the following series key can be used to retrieve the data for all physical flows for Canada and support services: A.1..14.

Use the plus (+) character as the "OR" operator. For example, the following key can be used to retrieve the data for all physical flows for Canada and the products of support services and construction services: A.1..14+15.

parameters

In this query string, parameters are defined as

startPeriod=value&endPeriod=value&firstNObservations=value&lastNObservations=value&detail=value&format=value

In the case of Canadian Centre for Energy Information, the applicable parameters are detail and format.

detail

To specify the amount of information you want to be returned by the web service, use the detail parameter.

Possible options include:

• full: All data, including series and observations, and attributes will be returned. This is the default.
• dataonly: The returned message will exclude attributes.
• serieskeysonly: The returned message will include only the series. Attributes and observations will be excluded. For example, this option would allow you to list series matching a specific query, but not return the actual data.
• nodata: The returned message will include the series, as well as the attributes, but not the observations.

format

Using the format parameter, you can instruct the web service to return data in different formats (see also Content negotiation).

Possible options are:

• csv
• jsondata

Examples

1. Retrieve all the data (Canada / All physical flows / All products) for the DF_38100139 dataflow.

https://information-energie.canada.ca/sdmx/rest/data/STC,DF_38100139/

2. Retrieve the products data for Support services (Canada / All physical flows) for the DF_38100139 dataflow.

https://information-energie.canada.ca/sdmx/rest/data/STC,DF_38100139/A...14

Statistical Data and Metadata eXchange metadata web service

All metadata can be retrieved using the query string described below.

protocol://wsEntryPoint/resource/agencyID/resourceID/version?detail=value&references=value

Syntax definition

protocol

The web service is available over https.

wsEntryPoint

The web service entry point is available at the same location of the SDMX metadata entry point, i.e., "energy-information.canada.ca/sdmx/rest/".

resource

The following resources are supported by the web service: datastructure, codelist and dataflow. Please note that not all of them are used at the moment.

agencyID

Organizations defining metadata are known as agencies in the SDMX. To retrieve an artefact, you should know the identifier of its maintainer. The agencyID code to represent Statistics Canada and CCEI are respectively STC and CCEI.

resourceID

The identifier of the resource, such as CL_AREA for the geography codelist, are maintained by the agency, which in this case is STC.

version

The version of the artefact to be returned. When the version number is not supplied, the latest version of the resource is returned.

detail

You can specify the desired amount of information to be returned by using the detail parameter. For example, it is possible to instruct the web service to return only basic information about the resource (i.e., its ID, agencyID, version and name). This is also known in the SDMX as a stub.

The allowed values are:

• full: All available information for all artefacts will be returned. This is the default.
• allstubs: All artefacts will be returned as stubs.
• referencestubs: The referenced artefacts will be returned as stubs.

references

You can instruct the web service to return (or not) the artefacts referenced by the artefact to be returned (e.g., the codelists and concepts used by the DSD matching the query) by using the references parameter. You can also retrieve the artefacts that use the matching artefact (e.g., the dataflows that use the DSD matching the query).

Possible values are:

• none: No references will be returned. This is the default.
• parents: The artefacts that use the artefact matching the query (e.g., the dataflows that use the DSD that match the query) will be returned.
• parentsandsiblings: The artefacts that use the artefact matching the query, as well as the artefacts referenced by these artefacts will be returned.
• children: The artefacts referenced by the matching artefact will be returned (e.g., the concept schemes and codelists used in a DSD).
• descendants: References of references, up to any level, will also be returned.
• all: The combination of parentsandsiblings and descendants will be returned.

A concrete type of resource may also be used (e.g., references=codelist).

Content negotiation

You can select the representation to be returned by using the HTTP content negotiation mechanism. You can also instruct the service to compress the data to be returned.

Format selection

The following data formats are supported by the web service:

• Data formats:
  • SDMX-ML (SDMX-Meta Language) 2.1 Generic Data format: application/vnd.sdmx.genericdata+xml;version=2.1. This is the default for data queries.
  • SDMX-ML 2.1 Structure Specific Data format: application/vnd.sdmx.structurespecificdata+xml;version=2.1
  • SDMX-JSON: application/vnd.sdmx.data+json;version=1.0.0-wd
  • CSV: text/csv
• Metadata formats:
  • SDMX-ML Structure format: application/vnd.sdmx.structure+xml;version=2.1
  • SDMX-JSON Structure format: application/vnd.sdmx.structure+json;version=1.0

Generic mime types (application/json, application/xml) are also supported, but they will always point to the most recent version of the SDMX formats supported by these web services. That version will change in the future whenever new versions of the various SDMX formats are made available.

It is therefore highly recommended that implementers use one of the specific mime types above rather than a generic one to avoid issues when new versions of the formats are released.

Output compression

You can also enable data compression using the Accept-Encoding HTTP header field. Compressed messages are typically much smaller than uncompressed messages, which can lead to improvements when transferring large amount data over the network.

Status codes

The web service returns the following HTTP status codes.

Useful tips

The SDMX Technical Working GroupOpens in a new tab publishes a useful list of tips and tricks for web service clients worth reading.

The SDMX Technical Working GroupOpens in a new tab has also published a two-page cheat sheet that summarizes the main points of the SDMX 2.1 RESTful API.

If the documentation does not contain the information you require, or if you have any general comments or feedback regarding our web service, please contact usOpens in a new tab.

All sample queries in this document can also be executed using command-line tools such as curl or wget:

wget -O data.xml \

--header="Accept:application/vnd.sdmx.structurespecificdata+xml;version=2.1" \

https://energy-information.canada.ca/sdmx/rest/data/STC,DF_38100139,1.0/A.1.1.14

curl -k -o data.xml \

--header "Accept:application/vnd.sdmx.structurespecificdata+xml;version=2.1" \

https://energy-information.canada.ca/sdmx/rest/data/STC,DF_38100139,1.0/A.1.1.14