Test Runs & Statistical Comparisons

Comparisons of equivalent FAMOUS runs on different platforms.

xcggb

Equivalent runs on HPCx and quest cluster compared using tarograd.pro.

Statistics:

t-statistic for global average:
                     temperature at 1.5m  0.468
        total precipitation rate kg/m2/s  0.014
              pressure at mean sea level -0.545
     potential temperature (ocean) deg.c -0.024
          salinity (ocean) (psu-35)/1000  0.637
                aice : ice concentration -0.774

Proportion of area with |t|>2
                     temperature at 1.5m  0.055
        total precipitation rate kg/m2/s  0.062
              pressure at mean sea level  0.068
     potential temperature (ocean) deg.c  0.055
          salinity (ocean) (psu-35)/1000  0.084
                aice : ice concentration  0.112

Plots:

download

xbyvs

We compared equivalent 100 year runs on HPCx and ormen at Bristol, starting from identical restarts. The results were compared using the IDL program tarograd.pro (available on request).

Statistics:

Ormen (xciib) v HPCx (xbdho) 

t-statistic for global average:
                    temperature at 1.5m  1.023
       total precipitation rate kg/m2/s  0.724
             pressure at mean sea level -1.256
    potential temperature (ocean) deg.c  1.421
         salinity (ocean) (psu-35)/1000  1.202
               aice : ice concentration -0.882

Proportion of area with |t|>2
                    temperature at 1.5m  0.076
       total precipitation rate kg/m2/s  0.068
             pressure at mean sea level  0.043
    potential temperature (ocean) deg.c  0.078
         salinity (ocean) (psu-35)/1000  0.056
               aice : ice concentration  0.084

Plots:

download

adtan

Test Runs

There are results for five 100-year runs on HPCx.

xbyjc - This is the “base” FAMOUS job on HPCx. It was set up to be as close as possible to Simon’s “base” Ruby job xbxxy, which is in turn based on the Met Office’s adtan.
xbyje - As the base job but with reconfiguration of the atmosphere and ocean start files.
xbyjh - As the base job but with the optimisation level reduced from O3 to O2.
xbyji - This job starts from atmosphere and ocean restart dumps produced from a 1 year run. The date of the restart file is put back one year to reconcile it with the model start date. This is done using the reconfiguration, which also alters some of the data fields.
xbyjn - As above, the job starts from 1 year restart dump files, however in this case the dates are altered using the change_dump_data utility which changes the date only.

Ensemble Results

Statistics produced by Jonathan Gregory’s program compare_climates.pro, which does a statistical test using the variability within the ensemble to test whether the ensemble mean differs from the reference run (adtan). This is better than using temporal variability as a substitute for intraensemble variability because it avoids have to make assumptions about lack of correlation.

Data for years 2441-2541
Proportion of time-intervals with t-probability <0.1 or >0.9
                     temperature at 1.5m  0.400
        total precipitation rate kg/m2/s  0.100
              pressure at mean sea level  0.400
     potential temperature (ocean) deg.c  0.300
          salinity (ocean) (psu-35)/1000  0.200
                aice : ice concentration  0.400
    net down surface sw flux: sw ts only  0.600
              outgoing sw rad flux (toa)  0.500
      total cloud amount in lw radiation  0.300
              outgoing lw rad flux (toa)  0.400

Proportion of area with t-probability <0.1 or >0.9
                     temperature at 1.5m  0.281
        total precipitation rate kg/m2/s  0.278
              pressure at mean sea level  0.297
     potential temperature (ocean) deg.c  0.242
          salinity (ocean) (psu-35)/1000  0.300
                aice : ice concentration  0.335
    net down surface sw flux: sw ts only  0.296
              outgoing sw rad flux (toa)  0.310
      total cloud amount in lw radiation  0.247
              outgoing lw rad flux (toa)  0.279

(There should be a chance of 0.2 of the t-prob lying <0.1 or >0.9 by construction. Hence stats exceeding 0.2 indicate a significant difference.)

To download the plots:

compare_climates.ps

Individual Comparisons

The decadal means for each test run were compared against adtan results using tarograd.pro - “compares global average timeseries and fields of some basic quantities from corresponding meaning periods of two sources”.

Results:

xbyyg (Ruby)

t-statistic for global average:
                     temperature at 1.5m -0.190
        total precipitation rate kg/m2/s  0.226
              pressure at mean sea level -0.144
     potential temperature (ocean) deg.c -0.060
          salinity (ocean) (psu-35)/1000 -0.685
                aice : ice concentration  1.838

Proportion of area with |t|>2
                     temperature at 1.5m  0.045
        total precipitation rate kg/m2/s  0.053
              pressure at mean sea level  0.035
     potential temperature (ocean) deg.c  0.044
          salinity (ocean) (psu-35)/1000  0.060
                aice : ice concentration  0.078



xbyjc (HPCx - standard)

t-statistic for global average:
                     temperature at 1.5m -1.086
        total precipitation rate kg/m2/s  0.334
              pressure at mean sea level  0.761
     potential temperature (ocean) deg.c -0.463
          salinity (ocean) (psu-35)/1000 -0.235
                aice : ice concentration  2.570

Proportion of area with |t|>2
                     temperature at 1.5m  0.069
        total precipitation rate kg/m2/s  0.065
              pressure at mean sea level  0.094
     potential temperature (ocean) deg.c  0.086
          salinity (ocean) (psu-35)/1000  0.096
                aice : ice concentration  0.093



xbyje (HPCx with reconfiguration)

t-statistic for global average:
                     temperature at 1.5m -2.932
        total precipitation rate kg/m2/s -1.680
              pressure at mean sea level  1.678
     potential temperature (ocean) deg.c -2.800
          salinity (ocean) (psu-35)/1000 -0.430
                aice : ice concentration  3.591

Proportion of area with |t|>2
                     temperature at 1.5m  0.155
        total precipitation rate kg/m2/s  0.067
              pressure at mean sea level  0.086
     potential temperature (ocean) deg.c  0.139
          salinity (ocean) (psu-35)/1000  0.100
                aice : ice concentration  0.077



xbyjh (HPCx with level 2 optimisation)

t-statistic for global average:
                     temperature at 1.5m -2.906
        total precipitation rate kg/m2/s -0.967
              pressure at mean sea level  1.802
     potential temperature (ocean) deg.c -1.299
          salinity (ocean) (psu-35)/1000 -0.434
                aice : ice concentration  6.630

Proportion of area with |t|>2
                     temperature at 1.5m  0.156
        total precipitation rate kg/m2/s  0.079
              pressure at mean sea level  0.094
     potential temperature (ocean) deg.c  0.109
          salinity (ocean) (psu-35)/1000  0.094
                aice : ice concentration  0.212



xbyji (HPCx with started file reconfiged 1 year dump)

t-statistic for global average:
                     temperature at 1.5m -2.073
        total precipitation rate kg/m2/s -0.693
              pressure at mean sea level  1.297
     potential temperature (ocean) deg.c -1.492
          salinity (ocean) (psu-35)/1000  0.353
                aice : ice concentration  2.912

Proportion of area with |t|>2
                     temperature at 1.5m  0.105
        total precipitation rate kg/m2/s  0.052
              pressure at mean sea level  0.046
     potential temperature (ocean) deg.c  0.078
          salinity (ocean) (psu-35)/1000  0.066
                aice : ice concentration  0.110



xbyjn (HPCx - restarted from 1 year dump)

t-statistic for global average:
                     temperature at 1.5m -1.316
        total precipitation rate kg/m2/s  0.145
              pressure at mean sea level  1.013
     potential temperature (ocean) deg.c -0.638
          salinity (ocean) (psu-35)/1000 -0.119
                aice : ice concentration  2.725

Proportion of area with |t|>2
                     temperature at 1.5m  0.072
        total precipitation rate kg/m2/s  0.067
              pressure at mean sea level  0.093
     potential temperature (ocean) deg.c  0.090
          salinity (ocean) (psu-35)/1000  0.094
                aice : ice concentration  0.106

Download plots:

Page last modified on October 29, 2007, at 10:43 AM by annette