Comparison metrics microscale simulation challenge for wind resource assessment — Stage 1

Status

The deadline for this benchmark has been moved from September 30th, 2020 to December 31st, 2020!

The benchmark was launched on April 1st, 2020, and will be open for participation until December 31st, 2020. Interim results will be presented and discussed at the Wind Energy Science Conference in May, 2021. Final results will be presented at the Torque2022 conference. Register for the challenge here.

The kick-off webinar took place at 16:00–17:00 CET on April 7th, 2020. The slides are available on Zenodo. Please note that some organisational changes have been made since publishing these slides. For submission details, please therefore use the newest information in this blog article, and not the information in the slides.

Background

For a given wind energy project, the modeller carrying out the wind resource assessment is faced with a difficult choice of a wide range of simulation tools with varying accuracies and costs. In order to help modellers make this choice, a new decision process is being developed as part of a Swiss-funded project, “Comparison metrics simulation challenge”. The aim of this decision process is to create a plot of model accuracy, or skill score, against cost score similar to Figure 1(a). This shows a schematic representation of the skill score against cost score for a range of different tools, which are represented by the individual points. The areas marked in red are the areas deemed unacceptable by the modeller, where the skill score is too low and the cost score is too high. These areas may vary depending on the expectations and requirements of the modeller. The most effective solution is then chosen as the one with the highest skill score for the lowest cost score within the acceptable region, at the flattening-off part of the curve. Modellers need to be able to choose the most appropriate model before carrying out any simulations. In recent work, a new method for estimating the skill and cost scores of different wind modelling tools for a given project has been developed and applied to a range of tools for the Bolund Hill experiment by comparing predicted skill and cost scores (estimated before carrying out the simulations) to the actual skill and cost scores (established after carrying out the simulations). The method was shown to work well; however, further studies are required with a larger volume of data and an extension for calculating all wind directions and the Annual Energy Production (AEP).

In order to achieve this, this new public simulation challenge has been designed, which involves collecting a wide range of data for developing project-specific transfer functions between the predicted and actual cost and skill scores. In Stage 1, results will be collected for a pre-defined site and simulation set-up in a blind test. In Stage 2, the site will not be pre-defined, and results will be collected for simulations for any site, as long as multiple tools are compared for the same site. This will allow relationships between the site type and the transfer functions to be established. The challenge is coordinated within the International Energy Agency IEA Wind Task 31 “Wakebench”. Further background information can be found in this blog post.

Figure 1. (a) Schematic skill vs. cost scores, (b) Expected challenge results.

Scope and objectives

The goal of Stage 1 of this challenge is to collect comparison metrics data regarding the skill and cost scores of a range of different simulation tools for the Perdigão site in Portugal, both before and after carrying out the simulations. As the focus is on Wind Resource Assessments, the focus will be on obtaining average wind speed profiles over the measurement period as well as estimating the long-term Annual Energy Production at validation positions.

The results are expected to look something like Figure 1(b) above, where each point represents one tool. The clusters are expected due to different category of tool and solver type. There will be a discrepancy between the metrics predicted beforehand and those determined using the results of the simulations. Transfer functions to better predict the skill scores and costs will be developed based on the results.

Participants are asked to submit their estimations for pre-defined parameters for calculating the skill and cost scores before and after carrying out the simulations, as well as the simulation results. This is a blind test, so validation data will not be accessible until the submission window is closed. The results will be presented at the Wind Energy Science Conference 2021, therefore participants are expected to submit their data by the end of September 2020 at the latest.

Simulation case: Perdigão

In this Stage 1 of the challenge, participants are asked to carry out simulations at the Perdigão site in Portugal, consisting of flow over two parallel ridges with southeast–northwest orientation, which are 4 km long and 500–550 m tall and separated by about 1.5 km (Figure 2). The two main wind directions are NE and WSW, both approximately perpendicular to the ridges. This site has been chosen due to its complex terrain combined with the high number of field tests and open measurement data available for validation. A large measurement campaign was undertaken in 2017 as part of a large EU-US collaborative field experiment [3]. Measurement data from the three 100 m met masts (numbers 20, 25 and 29), the six 60 m met masts (numbers 7, 10, 22, 27, 34 and 37) as well as from the operating Enercon wind turbine will be used for this challenge (Figure 2).

Left: Perdigao site [3]. Right: met masts used in this challenge.

Input data

Participants may use the topography and roughness maps for an area of their choice around the site as provided here: https://perdigao.fe.up.pt/maps

The cleaned ten-minute average and standard deviations of the measurement data from all seven sonic anemometers and air temperature and humidity sensors on met mast number 29 will be provided as input data in a NetCDF file via a B2Drop link following registration. A description of this mast with its coordinates is available here. A time period of 09.03.2017— 17.06.2017 is used in order to completely overlap with the validation data. A guideline for help with NetCDF files can be found at the same B2Drop link.

Information about the wind turbine at the site can be found here. The hub height is 78 m.

Validation data

The cleaned ten-minute average and standard deviations of the measurement data from all the sonic anemometers on mast numbers 7, 10, 20, 22, 25, 27, 34 and 37 as well as temperature and humidity data will be used for validation. Descriptions of these masts with their coordinates are available here. A time period of 09.03.2017–17.06.2017 is used in order to completely overlap with the input data.

Additionally, we are in the process of trying to get access to SCADA data from the operating wind turbine over the same time period for validating the calculated energy production.

As this is a blind test, the validation data will only be provided after the challenge window has been closed. The comparison of the simulation results with the validation data will be used to estimate the actual skill scores of the models.

Data submission

Participants can carry out simulations with as many different models and set-ups as desired and are free to use whichever filtering, simulation and AEP calculation method they deem most suitable. Multiple set-ups can be submitted. As this project is focused on microscale effects, no mesoscale forcing or nesting should be used. The focus is on comparing entire process chains of the Wind Resource Assessment (excluding wake effects for now) and not just different wind models.

The following submissions are required in order to contribute to this project:

• Initial registration: sign up and define your privacy settings here (anonymous participation is allowed). You will then receive a unique nameID (e.g. HSR or org01) and a separate modelID for each tool you intend to apply (e.g. Fluent or model01) to use as described below.
• Model description: each participant should submit a single model set-up description here, in which each simulation set-up submitted to the challenge has to be described. Different simulation set-ups may include different tools or codes (e.g. ANSYS Fluent), solving approaches (e.g. RANS CFD), grid types (e.g. unstructured hexahedral), grid sizes (e.g. 10 m cells), simulation periods (e.g. 10 minutes), etc. A number of different wind directions and wind speeds may be carried out for each simulation set-up. Each set-up should be assigned a different simID, composed of the nameID, the modelID and simulation number simXX, e.g. HSR_Fluent_sim01 or org01_model01_sim01. Additionally, you will be asked to enter some details regarding the site complexity.
• Parameter scores before: before carrying out each simulation, your estimated parameter scores for calculation of the skill and cost scores must be entered here for each simulation set-up. Further instructions are provided at the link address. Each simulation set-up should be named nameID_modelID_simXX, e.g. HSR_Fluent_sim01 or org01_model01_sim01.
• Parameter scores after: after carrying out each simulation, the parameter scores for calculation of the cost scores must be entered here for each simulation set-up. The skill scores will be calculated by us based on your simulation results. Use the same simulation run name as above.
• Simulation results: for each simulation run, provide the following data via a B2Drop link in a single zip file named nameID_modelID_simXX.zip (note: you will receive the link for doing this only after submitting your model description and parameter scores):

1. One Excel file titled nameID_modelID_simXX_z.xlsx containing: 3D wind vector components of average vertical wind speed profiles extending over the entire domain height at each validation mast and the wind turbine location for each 30 degree wind speed sector averaged over the measurement period, as well as the estimated long-term gross Annual Energy Production of the wind turbine (without any losses).
2. OPTIONAL: One Excel file titled nameID_modelID_simXX_xy.xslx containing: for each wind sector, horizontal planes of 3D wind speed vectors at 100 m and 40 m a.g.l. with horizontal resolution of 100 m, covering the simulation area that you defined in the model description, averaged over the measurement period;
3. OPTIONAL: One Excel file titled nameID_modelID_simXX_xyz.xlsx containing: for each wind sector, vertical planes of 3D wind speed vectors through all the validation masts in the NE direction extending 1 km in the vertical, averaged over the measurement period.

Three Excel templates are provided at the B2Drop link provided following registration.

Note: Your definition of stability must be provided in the model description. If you want to submit more than one stability condition, do this by creating various simIDs.

Expected results

The following results are expected:

• Skill score vs. cost score for each validation mast for all submissions (using wind speed accuracy).
• Average skill score vs. cost score for all validation masts for all submissions (using wind speed accuracy).
• Skill score vs. cost score for all submissions (using AEP accuracy).
• RMSE wind speed (average and per sector) compared to measurements for each validation mast for each submission.
• RMSE AEP (total and per sector) compared to measurements for each submission (if available, otherwise just comparisons with each other).
• Detailed analysis and comparison of the horizontal and vertical planes submitted.
• Reduction of model uncertainty by fusing the output from multiple stochastic simulators (in collaboration with Prof. Chatzi at ETH Zurich, paper to be published at TORQUE2020).
• A Python analysis code will be made available on Github.

Schedule

Stage 1:

• April 1st, 2020: submission window opens.
• April 7th, 2020: kick-off webinar.
• September 30th, 2020: submission window closes.
• Autumn 2020: abstract submissions to Wind Energy Science Conference 2021.
• May 2021: discuss results during Wind Energy Science Conference 2021.

Stage 2:

• June 2020: kick-off webinar Stage 2.

Why participate?

This project provides you with a unique opportunity to:

• Compare and benchmark simulation tools for wind modelling in complex terrain.
• Contribute to development of a new method for quantifying and comparing skill and cost scores of modelling tools for wind energy applications.
• Contribute to the further understanding of the flow phenomena at the Perdigão site.
• Get involved in a large-scale international research project as part of IEA Wind Task 31 for a relatively low effort.
• Strengthen the link between research and industry.
• Share and discuss your results with the international research community.
• Get inspired to develop new project ideas with international research and industry partners.

How to participate?

Just start with the initial registration here and you will receive all the information you need.

Further communication will follow via a Microsoft Teams channel, which you will be given access to after registering.

Questions?

Please contact Sarah Barber on sarah.barber@hsr.ch or +41 55 222 42 62.

Acknowledgements

This challenge is being organised and run by the University of Applied Sciences Rapperswil as part of a project funded by the Swiss Federal Office of Energy (project number SI/501955–01).

References

[1] S Barber, A Schubiger, N Wagenbrenner, N Fatras, and H Nordborg. A new method for the pragmatic choice of wind models for wind resource assessment in complex terrain. Wind Energy Science, (Discussion paper), 2020.

[2] J. Berg, J. Mann, A. Bechmann, M. S. Courtney, and H. E. Jorgensen. The bolund experiment, part i: Flow over a steep, three-dimensional hill. Boundary-Layer Meteorology, 141(2):219-243,2011.

[3] H. J. S. Fernando et al., The Perdigao: Peering into microscale details of mountain winds. Bulletin of the American Meteorological Society, 100(5):799-819, 2019.