VOLUNTEERS - HOMEOFFICE

FWC Project Overview

Last Update: 2022/08/15

Behind the scenes, a FWC homeoffice overview. Information and updates on "stuff" that keeps our project running smoothly, from equipment hosts, to how we streamline data capture, charting & mapping information display and data sharing.

Our FWC objectives

As well as the FWC project objectives, some of the what we are doing (creating) in the project, and how we are doing it, are outlined below.

Our FWC project - How It All Works

Managing an island wide field project, centralised data capture, data QC and mapping and charting requires a "Cloud" database and interfaces to this cloud data to do stuff...

The SSIFWC project overview & workflow

All the elements of what makes our FWC (and keeps it live, with no re-inventing the wheel) are captured in this project overview.

We are:
Making a catalogue - "a complete, list of items, often with notes giving detail" It could also (ultimately) be an illustrated book... :-)
and
Cataloguing - "to understand what we have, where it is located and what are the stories. Cataloguing helps prioritise work and resources through understanding how important any given item is"

As a spin-off of the freshwater cataloguing project an island "creeks-to-be-named" (and watersheds to be named) campaign is in the works. Why(?) - names bring attachment and life! Our "official" FWC island map will incorporate basic key learnings from project field work: seasonal, year-round stream type(s), "new" creeks, feature names, revised watershed boundaries etc...
It's really not all there currently - measuring helps manage!

The "IT" behind the SSIFWC webmap

Our SSIFWC database is stored in an Epicollect5 cloud datastore, supported and managed by the Wellcome Foundation. Amazon AWS and Google services are used to host our public-facing user interfaces - the SSIFWC webmap, and this website respectively.

Our FWC field data is uploaded to a dedicated FWC cloud database from an Epicollect5 smartphone FWC app. Data can be uploaded in the field (no data coverage required), or a via a web browser.

Our public-facing data displays access our FWC Cloud database, and work using the following steps:

FWC data is pulled from our Epicollect5 cloud database via an Epicollect5 API
An "Epicollect Migrator", a small Python app, "gets" the cloud data (every 24 hours) and places it in a PostGres JSON database
The WebGIS calls JSON data from PostGres via a small Lambda/Python "API-migrator”
Our SSIFWC “webmap" (or webGIS) uses a Javascript React front-end for the end-user (maps, charts etc) display interfaces.

Git is used for cloning/checkouts, commits/push, master/branching development work etc.

Once our data is in the SSIFWC cloud database we have some automated download procedures for "pulling" our cloud data into any GIS environment - contact SSIFWC project lead for further details

The "IT" behind the SSIFWC webmap

Our FWC field data is uploaded to a dedicated FWC cloud database from an Epicollect5 smartphone FWC app. Data can be uploaded in the field (no data coverage required), or a via a web browser.

Our public-facing data displays access our FWC Cloud database, and work using the following steps:

FWC data is pulled from our Epicollect5 cloud database via an Epicollect5 API
An "Epicollect Migrator", a small Python app, "gets" the cloud data (every 24 hours) and places it in a PostGres JSON database
The WebGIS calls JSON data from PostGres via a small Lambda/Python "API-migrator”
Our SSIFWC “webmap" (or webGIS) uses a Javascript React front-end for the end-user (maps, charts etc) display interfaces.

"Raw" SSIFWC data - in the Cloud

An Epicollect5 cloud interface supports the downloads of our "raw" SSIFWC Epicollect5 cloud datasets* for study use. Our FreshWater Catalogue (SSIFWC) database contains some 5,200+ field locations (as at 2022/01), with information being stored on the following Salt Spring Island freshwater data attributes:

- geographic location/freshwater feature type
- presence/absence of flow
- water quantity (l/sec), measurement technique, and associated parameters used for flow calculation
- water chemistry (pH, actual conductivity, specific conductivity and water temperature , with some dissolve oxygen records)

*Note - raw SSIFWC field data is currently stored in a V2 database (2,777 data points) and in a V3 database (2,456+ data points).

The raw SSIFWC Epicollect5 cloud dataset can be visualised via our public-facing SSIFWC webmap and via the basic data charting user-interface accessible via this webmap.

As at 2022/03/16 we now have our SSIFWC datasets, from our volunteer data collection network, more accessible, interrogatable and downloadable - via a live "Pacific DataStream" FWC data link - FWC Datastream. This Data Stream interface gives access to FWC charts, maps & statistics, a great data visualisation toolset!

"Processed" SSIFWC data - from Cloud database to Excel

In addition to the SSIFWC raw data a "FWC processed data workflow", is available for (Excel) charting and data analysis. For the latter our FWC Cloud data is compiled, sorted, and formulae are applied to calculate flow in Excel. This processed dataset i used for "advanced" charting and analysis, principally using the field location names for sorting/charting. The raw data processing steps are as follows:

A SSIFWC data .csv file is exported from our SSIFWC Epicollect5 database
The exported SSIFWC Epicollect5 data is copied to an intermediate “DLV3FWC” Excel file, for running data re-formatting macros
After macros execution and data validation the finalised “DLV3FWC” data is transferred to our Excel “Master Workbook” for charting/interpretation,
the processed data is transferred to discrete watershed worksheets for watershed-level charting, and compiled for across-island watershed comparisons .

Documentation on SSIFWC processed data preparation, data analysis and interpretation guidelines is in-preparation.

The SSIFWC webmaps

The maps available in our SSI FWC webmap are a mix of "static" (fixed) and "dynamic" (updated/live) map layers. Much of the data being displayed changes infrequently (eg SSI LOT lines, geological faults etc) and static maps are adequate for display and interpretation, some of the data being displayed changes regularly (eg our FWC field data) so a dynamic map is required.

As a "browse a few maps" mapping starting point have a look at the Useful Island Mapping Tools.

Due acknowledgments are made to the IT, CRD, FLNRO, the BC Province, and Geological Survey Canada, for the map contents shared on our webpages and in our SSIFWC webmap.

Mapping and processing our FWC cloud data - using QGIS/other GIS packages

Automated FWC cloud database access to (and downloads from) our FWC data, via postGIS and PostGres links, can be set up, for more technical GIS user (QGIS, ArcGIS, +). Please contact the SSIFWC Project Lead for further details on how to do this, or for an introduction to QGIS.

How the FWC started...

During the first part of 2018, and involved in a related freshwater role and studies review role (in the SSIWPA Technical Working Group), it became clear that we don't have any significant data (flowrates, groundwater contributions) on the majority of SSI island's surface water bodies... Around EarthDay, 2018 the WPS kicked off the FreshWater Catalogue, to address this shortage of key data, adopting that oft used mantra (rephrased) - "if you don't measure how can you manage it?".

Page updated

Google Sites

Report abuse