Comparison metrics simulation challenge — Stage 2
Status: registration open
The challenge starts on June 1st, 2021 and closes on August 31st, 2021. You can sign up to take part here at any time. This challenge is being run on the WeDoWind data sharing and collaboration platform operated by The Swiss Wind Energy R&D Network and in collaboration with the International Energy Agency IEA-Wind Task 31 “Wakebench”.
Background and motivation
In wind energy, the accuracy of the estimation of the wind resource has an enormous effect on the expected rate of return of a project. Due to the complex nature of the weather and of the wind flow over the earth’s surface, it can be very challenging to measure and model the wind resource correctly. For a given project, the modeller is faced with a difficult choice of a wide range of simulation tools with varying accuracies and costs. If this choice is made incorrectly, either many resources are wasted in needless high accuracy simulations, or the rate of return is incorrect and investors risk losing large amounts of money. As there are currently no guidelines or tools available to the modeller to help with this choice, it is usually left to gut feeling — and this can be catastrophic for investors or acquirers of wind parks.
In order to help modellers make this choice, a new decision tool is being developed as part of a Swiss-funded project, “Comparison metrics simulation challenge”. The decision tool creates a plot of model accuracy, or skill score, against cost score similar to Figure 1. This shows a schematic representation of the skill score against cost score for a range of different tools, which are represented by the individual points. The areas marked in red are the areas deemed unacceptable by the modeller, where the skill score is too low and the cost score is too high. These areas may vary depending on the expectations and requirements of the modeller. The most effective solution is then chosen as the one with the highest skill score for the lowest cost score within the acceptable region, at the flattening-off part of the curve. Modellers need to be able to choose the most appropriate model before carrying out any simulations. Therefore a method needs to be developed for estimating the skill score and cost score of each intended model before carrying out the simulations.
In recent work, a new method for estimating the skill and cost scores of different wind modelling tools was developed and applied to a range of tools for different complex terrain sites. This involved comparing predicted skill and cost scores, which are estimated before carrying out the simulations, to the actual skill and cost scores, established after carrying out the simulations. The method was shown to work well; however, further studies are required with a larger volume of data [1], [2], [3].
In order to achieve this, a new public simulation challenge “Comparison metrics simulation challenge” was designed, which involves collecting a wide range of data for developing project-specific transfer functions between the predicted and actual cost and skill scores. The challenge consists of two stages, described below.
Stage 1 (completed)
Stage 1 was carried out in 2020, where participants simulated the Perdigão site in Portugal with various simulation tools. The participants filled out a survey, which was then used to determine the predicted skill and cost scores. After that each participant simulated the site with the simulation tool of their choice and submitted wind profiles and Annual Energy Production (AEP) values at different positions on the site. Details of the challenge design can be found in [2]. The analysis is ongoing (Figure 2) and the final results will be presented at the WESC2021 (25. — 28. May) and the TORQUE 2022 conference.
The results of Stage 1, combined with results from four other sites as part of a separate project, allowed us to refine the method and to start the development of an automated decision tool for the optimal choice of model for a given project. An overview of the tool is given in Figure 3.
Stage 2 (registration open)
The goal of Stage 2 is to collect simulation results and site complexity descriptions in order to further improve and validate our automated decision tool for the optimal choice of model for a given project. Participants are asked to submit results of already existing simulations and WRAs for any site.
This will be done using a pre-defined template and survey, in which you enter details of the site, your tool set-up, your experience, the estimated set-up and simulation costs as well as the resulting wind speeds and/or energy production at a location for which measurement data is available. Our decision tool will then estimate skill and cost scores of the submitted results using pre-defined weighted parameters, as well as classify the terrain complexity. We will compare the results to the actual accuracy of the simulations, obtained by comparing the predictions to the provided measurement data. This will help us refine the decision algorithm, parameter weightings and complex terrain classification method.
This challenge is being run on the WeDoWind data sharing and collaboration platform operated by The Swiss Wind Energy R&D Network. This will allow you to interact with the other participants, exchange ideas and data. We will show an overview of the results that will be updated regularly during and after the challenge.
Expected Results
The results are dependent on the amount of participation. However, we hope for:
(1) For all sites:
- Complex terrain classification results.
- Summary of submitted results.
- Summary of predicted skill and cost scores compared to actual skill and costs.
(2) For sites for which three or more results are submitted:
- Direct comparison of wind speed and AEP accuracies of different models / workflows.
- Direct comparison of costs of different models / workflows.
- Skill score vs. cost score scatter plots (using wind speed accuracy).
- Skill score vs. cost score scatter plots (using AEP accuracy).
(3) Description and code for the new decision tool on GitLab.
Why participate?
This project provides you with a unique opportunity to:
- Contribute to developing a decision tool for choosing the most effective WRA tool for a given wind energy project.
- Access the exclusive new WeDoWind data sharing and collaboration platform.
- Gain a better understanding of how your choices affect the overall project risks.
- Receive a summary and comparison of different WRA workflow costs and accuracies for many different sites.
- Access the results and code for a new decision tool for choosing the most effective WRA tool for a given wind energy project.
- Get involved in a large-scale international research project as part of IEA Wind Task 31 for a relatively low effort.
- Strengthen the link between research and industry.
- Share and discuss your results with the international research community.
- Get inspired to develop new project ideas with international research and industry partners.
Related information
More information about Stage 1 can be found here.
How to take part
Sign up here and receive the details of the challenge and how to submit your results on the WeDoWind platform. You can submit results until August 31st, 2021.
Questions?
Please contact Sarah Barber on sarah.barber@ost.ch.
Acknowledgements
This challenge is being organised and run by the Eastern Switzerland University of Applied Sciences as part of a project funded by the Swiss Federal Office of Energy (project number SI/501955–01).
References
[1] S Barber, A Schubiger, N Wagenbrenner, N Fatras, and H Nordborg. A new method for the pragmatic choice of wind models for wind resource assessment in complex terrain. Wind Energy Science Conference 2019 presentation, 2019.
