Data Aggregation and Query Modules

Most of what goes on in the Data Aggregation and Query modules happens "behind the scenes". For this reason we cover these modules very briefly. We provide some background information on how Zednet works, as well as to describe some of the terminology used in the Zednet interfaces. Some of the details are often implementation specific for different clients, depending on the specific type of loggers as well as method of data transfer used in a particular installation.

How data is imported and aggregated

Loggers and channels

Typically measurement instruments, what we call "assets" in the Zednet system, are connected to GSM (Global System for Mobile communications) capable loggers in the field. Measurements might include pressure, flow, water quality, water level and so on.
On the Zednet system, the time-series data-stream produced by a measurement instrument is referred to as a "channel". While assets may be replaced, and loggers physically moved from site to site, the data channel representing the state of the water resource at a particular measurement point remains valid.

Loggers typically have polling intervals between 5 and 15 minutes, and send data to the Zednet server using GPRS modems between 1 and 4 times per day. For practical purposes the information is not effectively “live” in other words. This is part of the design methodology that allows the logging equipment to operate for more than 5 years on a single battery supply without any external power. Shorter download times can be accommodated, however, for most practical applications this is not necessary and will simply reduce the battery life of the logging unit without adding any real benefit.

Alternative sources

Zednet presently mainly receives data via GSM. The system is however also able to "pull", or receive "push" data via any of the standard web protocols (FTP, HTTP, SCP).
This means that data can also be manually imported where GSM equipment is not avaliable.

Conversion

In most cases data will be uploaded to the Zednet servers in the proprietary format of a particular logger manufacturer. This then requires that data files are converted to a generic format after they are received.
For example, Zednet is able to integrate with the PMAC software distributed by Technolog to help convert the data uploaded by Technolog's Cello logging equipment.

The system implements a range of data filters, and can therefore receive data in standard CSV or XML formats as well. These formats can be extended to meet your requirements.

Aggregation

Once data is converted to standardised format, a number of data scrubbing operations are optionally made:

  • Calibration offsets can be applied where it is necessary.
  • Data can be converted using analog look-up table. This is useful when converting between water measures, or providing computed/compound data.
  • Meter start values, overruns etc. are accommodated.
  • Gaps in the data set are filled and data is re-requested where needed.
Storage

Scrubbed data is stored in a standardised, central database which acts both as data archive and query cache. This means that data from a range of devices can be queried in a standardised format, and data sets compared or combined regardless of source. The database runs in a dedicated environment, optimised for fast retrieval of records spanning even years of data. Data is also backed up from here in an offsite location.

Query

In most cases, end users engage with the query module via the various front end interfaces. The query module supplies a user with data when they view a chart, edit a channel's properties, or do anything to manipulate data via any of Zednet's front end interfaces. You don't need to know the intricate details of the query module, just that it "wraps" the data storage to to let users get at data in intelligent ways.

If your developers need to know more, contact the Zednet team directly for more information on the API exposed by the query module.