RRTMGP error with mpas and L93 mid-top configuration

[adamrher]

What happened?

Running the mpas dycore with FMTHIST causes an error in RRTMGP:

dec2193.hsn.de.hpc.ucar.edu 654:  radiation_tend: ERROR: kdist_sw%gas_optics: gas_optics(): array play has values
dec2193.hsn.de.hpc.ucar.edu 654:   outside range

This occurs with analytic_ic and real world inic, although at different time-steps, and sometimes in the kdist_lw%gas_topics call. The issue is related to there being more than 1 vertical level with pressures < 1 Pa in some grid columns of the mpas L93 configuration. The L93 RRTMGP fix we previously implemented (#1063) only considers the case of 1 level < 1 Pa, and it will not work for 2 or more. I've confirmed that in a year long mpasa120 run, there are routinely 2 levels < 1 Pa in the run, and in a few instances there were 3 levels < 1 Pa. There may be even more in mpasa3.75 km runs (?)

In terms of a solution, should we generalize the 1 level fix #1063 for some arbitrary number of levels < 1 Pa? I think a cleaner way to do this would be to just allow levels < 1 Pa into the RRTMGP pressure arrays in all (some?) instances, and then when doing the calculations in kdist_sw%gas_optics & kdist_lw%gas_optics, force it to use the lowest pressure in the lookup table when < 1 Pa, and produce a warning instead of erroring out. Although we would probably want to error out if pressures were << 1 Pa.

A problem with the #1063 approach, at least for MPAS, is it defines the size nlay of these pressure arrays at init time based on number of levels < 1 Pa in pref_edge in ref_pres.F90. That is, these are not the time varying pressures and so there could be times and grid columns where there is more than 1 level < 1 Pa and that would get snuck into the RRTMGP calculation and trigger the out of bounds error. That's what happened here -- the pref_edge for L93 mpas has 1 level < 1 Pa, and so when 2 or more occur during run time it errors out. So in addition to there being more than 1 level < 1 Pa in mpas runs, pref_edge is not a reliable surrogate for the number levels < 1 Pa at run time.

@briandobbins @brianpm @PeterHjortLauritzen @JulioTBacmeister @fvitt

What are the steps to reproduce the bug?

run a recent tag with FMTHIST and mpas

What CAM tag were you using?

cam6_4_055

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

Intel

Path to a case directory, if applicable

No response

Will you be addressing this bug yourself?

No

Extra info

No response