Search

WP10: ACTRIS Data Centre

WP10 ACTRIS Data Centre

Location (town, country): The main structure and installations of the ACTRIS Data Centre is located at
NILU - Norwegian Institute for Air Research, Kjeller, Norway. The other installations are at National Research Council - Institute of Environmental Analysis (CNR), Tito Scalo, Potenza, Italy, at University of Lille, Villeneuve d'Ascq, France, and at Reading University, Reading, UK. There will be a transfer of the installation from Reading University to FMI (Finnish Meteorological Institute) in Helsinki, Finland, during the course of the project.

Web site address

ACTRIS dataportal, EBAS: http://actris.nilu.no/, http://ebas.nilu.no/
EARLINET DB: http://access.earlinet.org/EARLINET/LoginPage.aspx
Cloudnet DB: http://www.cloud-net.org/data
ICARE Data Centre: http://www.icare.univ-lille1.fr

Annual operating costs (excl. investment costs) of the infrastructure (€):
Legal name of organisation operating the infrastructure: NILU - Norwegian Institute for Air Research
Location of organisation (town, country): Kjeller, Norway
Annual operating costs (excl. investment costs) of the infrastructure (€):2388293
(1232493 € for NILU, 71600 € for CNR, 946000 € for CNRS, 138200 € for FMI).

Description of the infrastructure

The overall goal of the ACTRIS Data Centre is to provide scientists and other user groups with free and open access to all ACTRIS infrastructure data, complemented with access to innovative and mature data products and tools for QA, data analysis and research.
ACTRIS-2 will further develop the data centre in the ACTRIS project, taking full advantage of all of the efforts and achievements in ACTRIS. The numerous measurement methodologies applied in ACTRIS result in a considerable diversity of the data collected. In accordance with these requirements, the ACTRIS Data Centre consists of three topic data bases archiving the measurement data, which are all linked in the ACTRIS data portal. Furthermore, ICARE will contribute with production and provision of satellite data linked to ACTRIS ground based data. Hence, ACTRIS Data Centre is founded on 4 virtual infrastructures:

I EARLINET DB (hosted by CNR) includes all aerosol profile data provided by ACTRIS partners, associated partners, and stations not participating in the ACTRIS project (e.g. from EARLINET). The aerosol profiles are quality-assured through standardized protocols developed within EARLINET and ACTRIS. More advanced products developed in NA2 will be implemented during ACTRIS-2, in collaboration with NA2. Access to new types of data will be offered. These products will be designed and optimized within NA2 activities and then made accessible through the EARLINET database and in the ACTRIS data portal.

II Cloudnet DB (hosted by UNIVLEEDS, transfer to FMI) provides vertical profiles of cloud and precipitation properties at high spatial and temporal resolution from ACTRIS sites and partners, associated partners, and stations not participating in the ACTRIS project (e.g. within the Cloudnet network). All products are quality-assured through standardized protocols developed during Cloudnet and ACTRIS. Meteorological profiles extracted from climate and weather forecast models are also provided. New measurements and products arising from NA2 activities will be implemented within ACTRIS-2. Ancillary datasets (e.g. NWP or reanalysis profiles) will be provided for short-term campaigns, as will online tools for objective evaluation of model upgrade testing. The database will be transferred to FMI during first year to ensure long-term technical support.

III EBAS (hosted by NILU) includes all ACTRIS near-surface data, both quality assured data and NRT data, in addition to data from numerous relevant networks such as EMEP, GAW-WDCA, AMAP, and research projects such as InGOS (EU FP7), EUSAAR (EU FP6), CREATE (EU FP5), EUCAARI and many more. EBAS offers specialized products for in-depth analysis of trace gas and aerosol near-surface data. These have been developed in ACTRIS and other projects, with new data submission, QA, and improved plotting tools being provided within the course of ACTRIS-2 after development and implementation in collaboration with NA3.

IV: ICARE (hosted by CNRS) provides services to support in fields related to aerosols, clouds, radiation, water cycle, and their interactions. ICARE's primary focus is the production and distribution of satellite remote sensing data from various space agencies. ICARE develops science algorithms and production codes, building on the expertise from expert partners, and distributes products to the user community. ICARE generates value-added aerosol products pertinent to ACTRIS, combining complementary ground-based measurements such as lidars and sunphotometers. ICARE will also complement the ACTRIS data centre with various data services, including visualization, reprojection, colocation, reformatting, and satellite vs. ground-based observations comparisons, to facilitate the combined use of ACTRIS ground-based measurements with satellite observations and model outputs.

The ACTRIS Data Portal (hosted by NILU) links the 3 topic databases archiving ACTRIS-2 measurement data, and provides access to all data through one joint portal. The portal facilitates the combined analysis of all ACTRIS data, offering advanced tools for plotting and combining ACTRIS data from the topic data bases, and mapping tools for user defined visualization of the variables across atmospheric sites, networks and projects. The ACTRIS data portal will further develop and provide access to mature tools and products developed in ACTRIS-2 NAs and JRAs. ACTRIS SOPs, recommendations, and meta data descriptions will be provided in accordance with the progress and development in NA2, NA3 and JRA3.

Services currently offered by the infrastructure

The ACTRIS Data Centre established during ACTRIS provides a comprehensive data management service, including open access to all ACTRIS data and products. As of April 2014, the data management in ACTRIS handles ACTRIS data from more than 50 European sites, with approximately 120 different atmospheric variables from 28 instruments and methodologies. The variables include 90 different trace gases, 10 aerosol in situ variables, 8 aerosol profile variables and 8 cloud profile variables. These measurements are all QA and, furthermore, around 25 of these variables are available in NRT. Service and support on personal and technical level are offered to the data providers and personnel at the sites to achieve and maintain this huge task. Hence, the ACTRIS data management service comprises development and set up of correct procedures for data transfer from the various instruments and sites, active participation in QA and QC of data, collection of data in accordance with defined ACTRIS procedures set in the NAs. Continuous service and maintenance of the infrastructures ensure open access to all data, day and night, all year round.

The comprehensive ACTRIS data management of ground based data is unique in the European and global perspective, both with respect to number of variables and sites and also proven long term sustainability of data preservation. All ACTRIS data will be collected and archived in long-term sustainable databases. The data centre also has a global link to GAW through NILU and EBAS. EBAS hosts the GAW-WDCA. GAW activities are voluntary based, and ACTRIS also offers a crucial service to GAW, not only hosting GAW-WDCA, but EBAS serves as a node data centre for WDCGG as well improving European data represented in WDCGG. Open access and download of all relevant data in EBAS is in place within the ACTRIS portal. This includes data from long-term monitoring programs EMEP and GAW, as well as AMAP, InGOS (halocarbons), EUSAAR, CREATE, EUCAARI, GUAN SOGE, and other projects. Access to data from the EU-project PEGASUS will also be set up in 2015. Access to all data from EARLINET and Cloudnet is also offered as well through the portal, in addition to a selection of data from other relevant frameworks not available in the core data bases; e.g. all NDACC data.

Both access and download of ACTRIS data has been monitored throughout ACTRIS. This reveals that since the project start (2011), almost 2 million data files with ACTRIS variables have been downloaded through the data centre, and more than 2000 unique users identified from 50 different countries, from all continents. In addition to individual scientists and users, central user communities outside the project are e.g. MACC, MACC2, EMEP, IPCC 5th AR, WGI, (e.g. Chapter 2: Observations: Atmosphere and Surface, in particular).

Modality of access under this proposal

Access to all measurement data will be made available through the ACTRIS data web portal, as well as through the web pages of the core databases, to ensure suitable interfaces covering with the widespread of user needs. Access to the relevant products and tools developed in NAs and JRAs will be provided through the ACTRIS data portal. This access will be freely available and the ambition is to provide access 24h per day, 365 days per year. Free and open access means unrestricted access at no cost for all interested individuals, whether they are within or outside of the project, but an acceptance of the ACTRIS data policy will be required. For ground-based remote sensing data, registration and password may be required to have access, but no restrictions apply. Access to all data and data products will be recorded through web-based monthly user statistics for all virtual access activities.

Support offered under this proposal

The ACTRIS Data Centre will further develop under this proposal, and will ensure a continuation of the existing data centre and all efforts and achievements in ACTRIS. The ACTRIS Data Centre offers extensive and broad support for all users of ACTRIS RI data and tools, and is a continuation of the support currently offered. The support offered is divided into the following 5 main tasks:

Task 10.1: Scientific and technical support with collection, archiving and access to all ACTRIS-2 measurement data (NILU, CNR, FMI)
This is the core activity of the data centre and includes support for the data management of all ACTRIS measurements. Data from more than 50 sites employing ca. 25 different observational methodologies are expected. It is an unambiguous goal of the project that all ACTRIS measurements data will be collected and made available for the project participants, as well as for a broad user community at all times. Thus this support benefits all possible users, internal and external to the project. Support will be given to both data providers and users, and also for new types of data defined within NA2 and NA3. ACTRIS-2 aims to participate in the “Open Research Data Pilot” action in Horizon 2020. As a part of this, ACTRIS-2 will establish a complete “Data Management Plan - DMP” for the full project specifying all data the project will generate, the data repositories and security/back-up systems, and how data and data products will be made accessible for verification and use during project period, and beyond. The DMP will be revised during the project in accordance with the needs.

Task 10.1.1: Managing the ACTRIS data portal (NILU)
This includes maintenance and operation of the ACTRIS data portal during the project, including the implementation and access to new data, functionalities and tools developed in NAs and JRAs. Support for extraction of large data volumes of selected in situ data and profiles will be offered. An interface (e.g. machine-to-machine transfer, ftp, specified formats/procedures etc.) will be set up for ACTRIS NRT data provision to operational models for model evaluation and data assimilation, to facilitate the objectives of JRA3. Active interaction between modelling partners and data providers will be facilitated. This will be done in close collaboration with WP5, task 10.2 and the work on data interoperability. Special efforts will be made in collaboration with WP5 to improve identification and citation and data-usage tracking in ACTRIS-2.

Task 10.1.2: Data management of aerosol profile data in the EARLINET DB (CNR)
This includes maintenance and operation of the EARLINET DB during the ACTRIS-2 project. At present, the network includes about 27 stations with profiles of 10 aerosol variables. During the project the number of stations and observed variables are expected to increase. In particular, the development of a new database structure planned in NA2 will lead to a significant increase of variables and files accessible through the database. Furthermore, a closer link to GALION will be explored in ACTRIS-2 to strengthen global access to data as well.  

Task 10.1.3: Data management of cloud data in the Cloudnet DB (FMI)
The Cloudnet database provides the framework encompassing the data collection, quality control, processing routines, and subsequent product generation of ACTRIS cloud data. This ensures the harmonization of procedures at ca. 10 sites, with provision for NRT data flow. Support for all users in utilizing the processing suite will be further developed in ACTRIS-2. The ACTRIS cloud processing suite provides a framework for instrument data processing and ingest, and for the rapid incorporation of new algorithms. Tools, assistance, development and testing of processing routines for new instrumentation and products will be provided in NA2. Once new algorithms become mature and accepted by the community (through collaboration within NA2), these will also be implemented transparently and automatically across all cloud-profiling sites within ACTRIS.

Task 10.1.4: Data management of in situ data in EBAS (NILU)
This task includes data management of all ACTRIS-2 quality assured annually reported data, and NRT data flow, as well as the operational service on daily basis. Based on what has been achieved in ACTRIS, and the plans in ACTRIS-2, it is expected that the workload will be handling data from about 35 sites, comprising 12 aerosol in situ variables, and ca. 100 different trace gases. User support includes first line assistance in data formatting and submission in accordance with the ACTRIS procedures and deadlines. This is offered through our support system on a daily basis and also through implementation of QA/QC tools and dedicated training courses during the project. The support offered is in close collaboration and agreement with NA3. EBAS is also widely used by other projects, e.g. EMEP, GAW-WDCA, AMAP. ACTRIS-2 will take full advantage of this and complement these activities. We also suggest strengthening our support for GAW-WDCA data providers in ACTRIS-2.

Task 10.2: Access to interactive data submission portal (NILU, CNR,FMI)

Task 10.2.1: Interactive submission tools for ACTRIS near-surface data (NILU)
For ACTRIS in situ data, the submission site http://ebas-submit.nilu.no offered by NILU will be further developed in ACTRIS-2 in accordance with the project requirements and provide access to an interactive website for syntax and consistency checking of submitted aerosol and trace gas in situ data (e.g. meta data included, check for data boundaries and outliers). This will also be accessible from the ACTRIS portal. The tools will be available for all EBAS data providers and contribute to an improved overall data quality for the benefit of the broad scientific community. Hence, this support benefits all possible data submitters, internal and external, to the project as 3rd parties, and also improves data submitted by other frameworks such as GAW and EMEP.

Task 10.2.2: Interactive submission tools for ACTRIS profile data (CNR, FMI)
The interactive access to the Single Calculus Chain (SCC) for aerosol lidar data analysis will be offered by CNR-IMAA through the ACTRIS data portal to both internal and external data submitters. The SCC has been developed during EARLINET-ASOS and ACTRIS projects and will be furthermore implemented during the ACTRIS-2 project with new functionalities. The SCC will be made freely available through the ACTRIS data portal for all users. This will promote the SCC to external users and harmonization of the data within other framework as GAW – GALION. Documentation and tutorials on the use of SCC will be available through the ACTRIS data portal.
Access to the Cloudnet processing suite will be offered by FMI to sites operating the requisite instrumentation. Data submission for Cloudnet is automatic as all sites use the processing suite ensuring that all data ingested conforms to ACTRIS standards through pre-processing. Additional products will be implemented through updates to the processing suite.

Task 10.3: Access to QA/QC tools for ACTRIS aerosol measurement data (CNR, NILU, FMI)
Starting with the experience from the ongoing ACTRIS project, various tools for QA/QC will be implemented and made accessible for both near-surface trace gases and aerosols, and profiling of aerosol and cloud. Building on ACTRIS, QA/QC tools will be developed in NA2 and NA3 in collaboration with the data centre, and implemented in this task for wide user access. This includes improved plotting and access to QA tools. Proposed tools include routines for checking the consistency of the data in terms of ratios between specific trace gases (near-surface data), aerosol extinction–to-backscatter ratio, depolarization ratio and Ångström exponent (lidar data) and consistency between aerosol variables from different instruments through closure tests (using in situ aerosol). Data will also be checked through the comparison with climatological data. The work will be guided by the needs for these tools in the respective network activities (NA2 and NA3) and from the feedbacks of external users (NA5 and JRA3). The implementation of these tools will improve the data quality. Tools will be available to all potential users of ACTRIS data, both internal and external, enlarging the data centre impact.

Task 10.4: Access to value-added products using ACTRIS data in mature algorithms (CNRS, CNR)
ICARE has developed a production framework capable of processing ground-based remote observations to generate value-added products that will benefit ACTRIS-2. ICARE is collaborating with CNRS/LOA to implement several algorithms capable of deriving aerosol properties from lidar and sunphotometer measurements. One is a new algorithm named GARRLiC (Generalized Aerosol Retrieval from Radiometer and Lidar Combined data) which has been developed and tested during ACTRIS, and has reached a mature stage, although further developments will be made during ACTRIS-2 in NA2. ICARE proposes to transfer this algorithm from research mode to operational, generating a consistent suite of integrated products across the whole ACTRIS network. ICARE proposes to retrieve and process quality-assured level-1 lidar data from the EARLINET DB, then provide the derived products to CNR-IMAA to complement the EARLINET DB. ICARE will make these new products available as soon as the operational implementation is validated, potentially by month 24 of the project. ICARE will further consider integrating possible algorithm improvements coming from NA2 during the course of ACTRIS-2, including the implementation of new alternate algorithms when they reach a sufficient level of maturity to consider routine production. In addition, through its infrastructure and expertise, ICARE can offer technical resources and support at a marginal cost for ACTRIS participants who would wish to deploy across the whole database other algorithms of common interest which are not listed here.

Task 10.5: Access to aggregated data and products combining ACTRIS measurements with model or satellite data (NILU, CNR, FMI, CNRS)
This task offers user support and access to innovative scientific products for a wide user community through the ACTRIS data portal. Typical products that will be provided are aggregated data such as, daily, monthly or annual means of selected variables. Extensive support for extraction of selected specific in situ measurements and aerosol profiles in support for the development and work in JRA3 will be offered. The VA10.5 task will be guided and performed in close collaboration with JRA3 and will undergo revisions in accordance with scientific requests during the project. To be able to meet user needs and requests that might emerge during the project, some funding is possible to distribute to the most relevant and qualified partner or SME during the project for successful implementation of new products defined during the project.

Task 10.5.1: Access to the mature products resulting from ACTRIS activities (NILU, CNR, FMI)
Access to the mature products resulting from ACTRIS will be implemented, as will online access to the outcomes and results from the work in JRA3. This includes comparisons of aggregated data and time series of observations with model outputs. Trend products developed in JRA3, task 4, will be made accessible when relevant, and efforts will be made to implement an interface with tools for trend assessments for advanced users.

Task 10.5.2 : ACTRIS NRT data to model community (NILU)
An interface (e.g. machine-to-machine) will be set up for ACTRIS NRT in situ data provision and operational models for model evaluation and data assimilation (see JRA3, task 1).This will be based upon requests and can also be used by 3rd parties and users outside the project, after acceptance of the data policy.
Task 10.5.3: Evaluation of user-submitted model runs employing ACTRIS (FMI)
Cloudnet provides a framework for objective NWP and climate model evaluation. Current functionality for monthly, seasonal, diurnal and inter-annual composites and variability will be developed further in ACTRIS-2 to include climatic / synoptic regime compositing and interactive user selection of time period of interest. Provision of an online portal for evaluation of user-submitted model runs (upgrade testing, parallel suites, reprocessing over specific time periods, microphysics parameterization experiments) and accompanying assistance will be implemented.

Task 10.5.4: Model, satellite and ground based colocation service (CNRS, NILU)
Links to satellite data will be prioritized through ICARE, and a satellite-ground based colocation service will be provided combining ACTRIS data from ground with satellite data and model outputs, including CMIP5. For example, users will be able to use this interface to combine ground-based aerosol observations with coincident remote sensing retrieval from satellites, such as aerosol optical depth, Ångström exponent, etc., or consolidate data analysis with model outputs such as the MACC aerosol analyses. Also, users can use this service to get contextual cloud and aerosol satellite observations in the vicinity of a ground-based station to support data analysis. ICARE will work with NILU to make this colocation service directly available through the ACTRIS portal.
Users often need a work environment with easy access to both data archives and computational resources to develop and test new algorithms, or to perform validation activities. CNRS-ICARE provides this service in the frame of other projects (e.g., Aerosol-CCI) and proposes to provide such a work environment also to ACTRIS partners, with online access to the full ICARE's satellite and ground-based data archive.

Outreach to new users

An important goal is to facilitate the use of ACTRIS data and products for the scientific community, as well as other user communities. In ACTRIS-2, we plan to take further advantage of the specificity of each partner data centre that constitutes the ACTRIS Data Centre to optimally serve specific user communities and expand data dissemination overall. One strategy in order to strengthen and attract new users is involvement ICARE, as a new installation of the data centre. ICARE contributes in particular, with the link between the ACTRIS ground-based community, and the satellite community and opens a new approach to a well-established user community. Furthermore, the addition of ECMWF as a new partner in ACTRIS-2 and the close collaboration defined between JRA3 and VA1 will put emphasis on the interaction with new model users, in particular, the use of observational data in operational model prediction and data assimilation. Through the collaboration with JRA3, more products (e.g. regular products for trend assessment reports) will be offered. This is of particular relevance for users working on trend assessment reports (e.g. IPCC, EMEP), and for authorities, stakeholders and decision makers working on mitigation strategies. These products will be offered through the data centre. Finally, partners of VA1 will not only be elements of the ACTRIS data centre but also have an active role in the outreach activity in NA5.

Review procedure under this proposal

The data centre will be reviewed regularly by an external board of international experts (ACTRIS DC-EB) following the procedures given in section 3.2. The board will assess the development and use of the facility by reviewing the core services, and a set of deliverables (D10.1-3, D10.7-9, andD10.12-13) are defined as valuable input for both the ACTRIS DC-EB, but also important summaries for all involved in VA1 and the whole consortium. These will document the number of new ACTRIS data sets included in the data centre, the access and download of the date from the data centre, assess the implementation of new tools and products, document number of unique users etc. A set of evaluation criteria will be defined by the consortia together with the ACTRIS DC-EB.