The following text

Answers to Frequently Asked Questions about SPEC Benchmarks

is updated regulary by active SPEC members. Last update: 09/11/1995 (Changes to previous version: SPEC member list, Order Form)

SPEC95 Announcement


  1. What is SPEC
  2. How to Contact SPEC
  3. SPEC's Products and Services
  4. Current SPEC Benchmarks
    1. CPU Benchmarks
    2. SDM Benchmarks
    3. SFS Benchmarks
  5. Outdated SPEC Benchmarks
  6. Forthcoming SPEC Benchmarks
    1. Open Systems Group Benchmarks
    2. New CPU Benchmark Suites
    3. High Performance Computing Group Benchmarks
    4. New Benchmarks in General
  7. Membership in SPEC
  8. Order Form
  9. Acknowledgments

  1. What is SPEC
  2. SPEC, the Standard Performance Evaluation Corporation, is a non-profit corporation formed to "establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers" (from SPEC's bylaws). The founders of this organization believe that the user community will benefit greatly from an objective series of applications-oriented tests, which can serve as common reference points and be considered during the evaluation process. While no one benchmark can fully characterize overall system performance, the results of a variety of realistic benchmarks can give valuable insight into expected real performance.

    Current members of SPEC include:

    1. Open Systems Group (OSG):

      Amdahl, AT&T, Auspex, Bull, Chevron USA, Compaq, Cray Research, Dansk Data Elektronik, Data General, Digital Equipment, Electronic Data Systems, FirePower Systems, Fujitsu, HaL Computer, Hewlett-Packard, Hitachi, IBM, Intel, Intergraph, International Computer (ICL), Locus Computing, Motorola, Network Appliance, Nikkei Datapro, Novell, Olivetti, Pyramid Technology, Siemens Nixdorf, Informationssysteme, Silicon Graphics, Solbourne, Sun Microsystems, Tricord Systems, Unisys, ZIFF Davis Publishing.

    2. High Performance Group (HPG):

      Convex Computers, Digital Equipment Electronic Data Systems, Fujitsu America, Hewlett-Packard, IBM, Kuck & Assoc., NEC c/o HNSX Supercomputer, Supercomputer Systems Division of Intel, Silicon Graphics, Sun Microsystems.

    3. HPG Associates:

      University of Illinois, University of Michigan, University of Minnesota, University of Tennessee.

    Legally, SPEC is a non-profit corporation registered in California.

    SPEC basically performs two functions:

  3. How to Contact SPEC
  4.     Standard Performance Evaluation Corporation (SPEC)
        2722 Merrilee Drive, Suite 200
        Fairfax, VA 22031
        Phone:  +1-703-698-9604 
        FAX:    +1-703-560-2752
    Dianne Rice (ext. 325) is the Director of Operations for SPEC and Rodney Sanford (ext. 318) is the SPEC administrator. They are responsible for the adminstration of SPEC's products and for assisting customers with questions and orders. Technical questions regarding the SPEC benchmarks (e.g., problems with execution of the benchmarks), are usually referred to SPEC's technical support person..

  5. SPEC Products and Services
  6. SPEC's main product is benchmark suites. The code in the suites are developed by SPEC from code donated by various sources. SPEC works on portability and creates tools and meaningful workloads for the codes chosen as benchmarks. Therefore, the SPEC benchmarks are not the same as public domain programs that may exist under a similar name or the same name, and their execution times will, in general, be different.

    The SPEC benchmark sources are generally available, but not free. SPEC is charging separately for its benchmark suites. The income from the benchmark source tapes is intended to support the administrative costs of the corporation. benchmarks, and so on. Buyers of the benchmark tapes receive a shrink-wrapped site license with their first order of any of SPEC's tape products along with the manual explaining the rules for result publications. All benchmark suites come on QIC 24 tapes, written in UNIX tar format.

    Current prices are:

      CINT92          $  425.00  (CPU intensive integer benchmarks)
      CFP92           $  575.00  (CPU intensive floating point benchmarks)
      CINT92&CFP92    $  900.00
      SDM             $ 1450.00  (UNIX Software Development Workloads)
      SFS             $ 1200.00  (System level file server (NFS) workload)
    * Accredited universities receive a 50 percent discount on SPEC tape products.

    The SPEC Newsletter is published quarterly and contains result publications for a variety of machines, about 50-70 pages of result pages per issue, with articles dealing with SPEC and benchmarking.

    SPEC Newsletter Prices (1 year subscription, 4 issues)

                      $  550.00 for USA Orders
    		  $  575.00 for International Orders 

  7. Current SPEC Benchmarks
    1. CPU Benchmarks
    2. There are currently two suites of compute-intensive SPEC benchmarks, measuring the performance of CPU, memory system, and compiler code generation. They normally use UNIX as the portability vehicle, but they have been ported to other operating systems as well. The percentage of time spent in operating system and I/O functions is generally negligible.

      CINT92, current release: Rel. 1.1

      This suite contains six benchmarks performing integer computations, all of them are written in C. The individual programs are:

      Number and name  Application                Approx. size
                                                  gross   net
      008.espresso     Logic Design               14800   11000           Interpreter                 7700    5000
      023.eqntott      Logic Design                3600    2600
      026.compress     Data Compression            1500    1000           Spreadsheet                 8500    7100
      085.gcc          Compiler                   87800   58800
                                                 ------   -----
                                                 123900   85500
      The approximate static size is given in numbers of source code lines, including declarations (header files). "Gross" numbers include comments and blank lines, "net" numbers exclude them.

      A somewhat more detailed, though still short description of the benchmarks (from an article by Jeff Reilly, in SPEC Newsletter vol. 4, no. 4):

      008.espresso    Generates and optimizes Programmable Logic Arrays.          Uses a LISP interpreter to solve the nine queens
                      problem, using a recursive backtracking algorithm.
      023.eqntott     Translates a logical representation of a Boolean
                      equation to a truth table.
      026.compress    Reduces the size of input files by using Lempel-Ziv
                      coding.          Calculates budgets, SPEC metrics and amortization
                      schedules in a spreadsheet based on the UNIX cursor-
                      controlled package "curses".
      085.gcc         Translates preprocessed C source files into optimized
                      Sun-3 assembly language output.

      CFP92, current release: Rel. 1.1

      This suite contains 14 benchmarks performing floating-point computations. 12 of them are written in Fortran, 2 in C. The individual programs are:

      Number and name  Application                Lang.   Approx. size
                                                          gross   net
      013.spice2g6     Circuit Design             F      18900   15000
      015.doduc        Simulation                 F       5300    5300
      034.mdljdp2      Quantum Chemistry          F       4500    3600
      039.wave5        Electromagnetism           F       7600    6400
      047.tomcatv      Geometric Translation      F        200     100
      048 ora          Optics                     F        500     300
      052.alvinn       Robotics                   C        300     200
      056.ear          Medical Simulation         C       5200    3300
      077.mdljsp2      Quantum Chemistry          F       3900    3100
      078.swm256       Simulation                 F        500     300
      089.su2cor       Quantum Physics            F       2500    1700
      090.hydro2d      Astrophysics               F       4500    1700
      093.nasa7        NASA Kernels               F       1300     800
      094.fpppp        Quantum Chemistry          F       2700    2100
                                                         -----   -----
                                                         57900   43900
      Short description of the benchmarks:
      013.spice2g6    Simulates analog circuits (double precision).
      015.doduc       Performs Monte-Carlo simulation of the time evolution
                      of a thermo-hydraulic model for a nuclear reactor's
                      component (double precision).
      034.mdljdp2     Solves motion equations for a model of 500 atoms
                      interacting through the idealized Lennard-Jones
                      potential (double precision).
      039.wave5       Solves particle and Maxwell's equations on a
                      Cartesian mesh (single precision).
      047.tomcatv     Generates two-dimensional, boundary-fitted coordinate
                      systems around general geometric domains
                      (vectorizable, double precision).
      048 ora         Traces rays through an optical surface containing
                      spherical and planar surfaces (double precision).
      052.alvinn      Trains a neural network using back propagation
                      (single precision).
      056.ear         Simulates the human ear by converting a sound file to
                      a cochleogram using Fast Fourier Transforms and other
                      math library functions (single precision).
      077.mdljsp2     Similar to 034.mdljdp2, solves motion equations for a
                      model of 500 atoms (single precision).
      078.swm256      Solves the system of shallow water equations using
                      finite difference approximations (single precision).
      089.su2cor      Calculates masses of elementary particles in the
                      framework of the Quark Gluon theory (vectorizable,
                      double precision).
      090.hydro2d     Uses hydrodynamical Navier Stokes equations to
                      calculate galactical jets (vectorizable, double
      093.nasa7       Executes seven program kernels of operations used
                      frequently in NASA applications, such as Fourier
                      transforms and matrix manipulations (double
      094.fpppp       Calculates multi-electron integral derivatives
                      (double precision).
      More information about the individual benchmarks is contained in description files in each benchmark's subdirectory on the SPEC benchmark tape.

      The CPU benchmarks can be used for measurement in two ways:

      Speed Measurement

      The results ("SPEC Ratio" for each individual benchmark) are expressed as the ratio of the wall clock time to execute one single copy of the benchmark, compared to a fixed "SPEC reference time" (which was chosen early-on as the execution time on a VAX 11/780).

      As is apparent from results publications, the different SPEC ratios for a given machine can vary widely. SPEC encourages the public to look at the individual results for each benchmarks. Users should compare the characteristics of their workload with that of the individual SPEC benchmarks and consider those benchmarks that best approximate their jobs. However, SPEC also recognizes the demand for aggregate result numbers and has defined the integer and floating-point averages

        SPECint92 = geometric mean of the six SPEC ratios from CINT92
        SPECfp92  = geometric mean of the 14 SPEC ratios from CFP92
      For "baseline" measurements (see below), the averages are
        SPECbase_int92 = geometric mean of the six SPEC ratios from
      		   CINT92, under the baseline measurement rules
        SPECbase_fp92 = geometric mean of the 14 SPEC ratios from CFP92,
      		  under the baseline measurement rules

      Throughput Measurement

      With this measurement method, called the "homogenuous capacity method", several copies of a given benchmark are executed; this method is particularly suitable for multiprocessor systems. The results, called SPEC rate, express how many jobs of a particular type (characterized by the individual benchmark) can be executed in a given time (The SPEC reference time happens to be a week, the execution times are normalized with respect to a VAX 11/780). The SPEC rates therefore characterize the capacity of a system for compute-intensive jobs of similar characteristics.

      Similar as with the speed metric, SPEC has defined averages

        SPECrate_int92 = geometric mean of the 6 SPEC rates from CINT92
        SPECrate_fp92  = geometric mean of the 14 SPEC rates from CFP92
      For "baseline" measurements (see below), the averages are
        SPECrate_base_int92 = geometric mean of the 6 SPEC rates from
      			CINT92, under the baseline measurement rules
        SPECrate_base_fp92 =  geometric mean of the 14 SPEC rates from
      			CFP92, under the baseline measurement rules
      Because of the different units, the values SPECint92/SPECfp92 and SPECrate_int92/SPECrate_fp92 cannot be compared directly.

      Baseline measurements

      At the January 1994 meeting, the SPEC Steering Committee decided to introduce "baseline results", starting in June 1994. These results (for both speed and throughput measurements) have to be measured with more restrictive run rules, regulating the use of compiler/linker optimization options ("flags"):

      The detailed baseline rules, together with an explanation, are listed in an article "Reviewing the New Baseline Rules" (R. Weicker, J. Reilly) in SPEC Newsletter vol. 6 no. 2 (June 1994).

      The intention is that these results represent the performance a not so sophisticated user would achieve, whereas the traditional rules allow a selection of optimization flags that is more typical for sophisticated users. As a general guideline, a system vendor is expected to endorse the general use of these baseline options by customers who seek to achieve good application performance.

      Effective June 1994, when SPEC's CPU benchmark results are reported, the reports have to include baseline results. Baseline-only reporting is allowed, but since the non-baseline level is equivalent to the previous reporting level, it can be expected that most companies will report numbers for both optimization levels.

      Comprehensive result lists

      Comprehensive lists of all CINT92/CFP92 result average values (speed measurements as well as throughput measurements) published during the year have been printed in the year-end issues of the SPEC Newsletter in December 1992, 1993, and 1994. They contain the system name, the respective average value, and a pointer to the newsletter issue where the full result report can be found. Readers are warned that "SPEC does not recommend that readers use any one value for making comparisons. There is a wealth of information in the Reporting Pages which cannot be easily reduced to summary values".

      No more SPECmark computation

      While the old average "SPECmark[89]" has been popular with the industry and the press (see section 5: Outdated SPEC Benchmarks), SPEC has intentionally *not* defined an average "SPECmark92" over all CPU benchmarks of the 1992 suites, for the following reasons:

    3. SDM Benchmarks
    4. SDM stands for "Systems Development Multiuser"; the benchmarks in this suite (current release: 1.1) characterize the capacity of a system in a multiuser UNIX environment. Contrary to the CPU benchmarks, the SDM benchmarks contain UNIX shell scripts (consisting of commands like "cd", "mkdir", "find", "cc", "nroff", etc.) that exercise the operating system as well as the CPU and I/O components of the system. The suite contains two benchmarks:
      057.sdet    Represents a large commercial UNIX/C based software
                  development environment. This characterization is based
                  on AT&T analysis and models developed by Steve Gaede,
                  formerly with AT&T Bell Laboratories.
      061.kenbus1 Represents UNIX/C usage in a Research and Development
                  environment. This characterization is based on data
                  collection and analysis at Monash University by Ken
      For each benchmark, throughput numbers (scripts, i.e. simulated user loads per hour) are given for several values of concurrent workloads. The reader can determine the peak throughput as well as the ability of a system to sustain throughput over a range of concurrent workloads. Since the workloads for the two benchmarks are different, their throughput values are also different and cannot be compared directly.

    5. SFS Benchmark Suite
    6. SFS stands for "system-level file server"; SFS Release 1 is designed to provide a fair, consistant and complete method for measuring and reporting NFS file server performance.

      SFS Release 1.1 contains one benchmark, 097.LADDIS. This benchmark measures NFS file server performance in terms of NFS response time and throughput. It does this by generating a synthetic NFS workload based on a workload abstraction of an NFS operation mix and an NFS operation request rate.

      Running 097.LADDIS requires a file server (the entity being measured) and two or more "load generators" connected to the file server via a network medium. The load generators are each loaded with 097.LADDIS and perform the 097.LADDIS workload on file systems exported by the file server.

      SFS Release 1.0 results include full server configuration information (hardware and software) and a graph of server response time versus NFS load for the 097.LADDIS operation mix. Compared with the predecessor version SFS 1.0, Release 1.1 contains bug fixes and improved documentation. The workload did not change, so results are comparable.

  8. Outdated SPEC Benchmarks
  9. SPEC has published the first CPU benchmark suite in 1989, the last release was 1.2b. It contained 10 compute-intensive programs, four integer (written in C) and six floating-point (written in Fortran). The following average values had been defined:
      SPECint89  = geometric mean of the SPEC ratios of the 4 integer
                     programs in rel. 1.2b (CPU-Suite of 1989).
      SPECfp89   = geometric mean of the SPEC ratios of the six
                     floating-point programs in rel. 1.2b.
      SPECmark89 = geometric mean of all 10 SPEC ratios of the
                     programs in rel. 1.2b.
    In addition, there was the possibility of throughput measurements, with two copies of a benchmark running per CPU, called "Thruput Method A" (There was never a "Method B"). The following average values had been defined:
      SPECintThruput89 = geometric mean of the Thruput Ratios of the
                          four integer programs
      SPECfpThruput89 = geometric mean of the Thruput Ratios of the
                          six floating-point programs
      SPECThruput89 ("aggregate thruput") = geometric mean of the
                          Thruput Ratios of all 10 programs
    SPEC now discourages use of the 1989 benchmark suite and recommends use of the CINT92 and CFP92 suites, for the following reasons:

    SPEC discontinued selling SPEC Release 1.2b tapes after December 1992; labeled any the 1.2b results published in the SPEC Newsletter with "Benchmark Obsolete". SPEC discontinued result publications for this suite in June 1993. A similar obsoletion process can be expected for the 1992 CPU benchmarks once the 1995 CPU benchmarks are in place (see below; section 6.2).

  10. Forthcoming SPEC Benchmarks
    1. Open Systems Group Benchmarks
    2. A number of areas have been considered or are being considered by SPEC (Open System group) for future benchmark efforts:

      SPEC (SFSSC) is also working on a new version of the SFS benchmark suite.

    3. New CPU Benchmark Suites
    4. The Open Systems Group is in the last phase of the development of new CPU benchmark suites, tentatively called CINT95/CFP95. These suites are intended to replace the current suites CINT92/CFP92, for the following reasons:

      In May 1995, a final vote of all SPEC OSG member companies finalized the benchmark selection process, all 8 integer and 10 floating-point benchmarks proposed by the Open Systems Steering Committee were accepted. First results obtained with the new suites are expected to accompany the announcement (probably August 1995), first newsletter results are anticipated for publication in the September 1995 SPEC Newsletter.

      The run and reporting rules for the new benchmarks suites are currently being finalized. It can be expected that again there will be a peak and a baseline metric. However, details will be different from the current rules.

      The new SPEC CPU benchmarks will also be ported to the Windows NT operating system. Since very few of the CPU benchmarks make operating system calls (they intentionally measure CPU / Memory / Compiler performance only), the two versions cannot be used to compare the efficiency of different operating systems. This port will, however, add a new portability vehicle to the UNIX systems that are used most currently.

    5. High Performance Computing Group Benchmarks
    6. In January 1994, several companies and academic institutions (see list in section 1) formed a new activity within SPEC, the High Performance Group. This group combines members of the "Perfect Club", the organization formerly active in the area of high-performance benchmarking, with members of SPEC. The group will work on benchmarks measuring the performance of systems that push the limits of computational technology.

      The group has elected its own officers and will develop its own benchmarks, together will appropriate run and reporting rules. Since the effort has just started in 1994, no selection of benchmarks has been made yet. Several benchmark candidate codes are being evaluated by the new group.

      To distinguish the two activities that now exist under the SPEC umbrella, the "traditional" SPEC group has now been renamed the Open Systems Group (with the Open Systems Steering Committee as the body for technical decisions), the new group has the name High Performance Computing Group.

    7. New Benchmarks in General
    8. SPEC is always open to suggestions from the computing community for future benchmarking directions. Of course, SPEC is even more receptive to proposals for actual programs that can be used as benchmarks. In particular, the SPEC Open Systems group is currently looking for programs that have similar characteristics as system code (integer code, large, low code locality, structured according to somewhat modern software engineering principles). Also, benchmark candidate programs in languages other than C and Fortran are welcome (C++, Ada, ...). The High Performance group is looking for large application programs that stress modern high-performance computers to their limits.

  11. Membership in SPEC
  12. The costs for SPEC membership are
      Annual Dues   $ 5000.00       Initiation Fee  $ 1000.00
    The dues for membership in OSG and HPG are separate.

    There is also the category of a "SPEC Associate", intended for accredited educational institutions or non-profit organizations:

      Annual Dues   $ 1000.00       Initiation Fee  $  500.00
    Among the membership benefits are Newsletters and benchmark tapes as they are available, with company-wide licenses within their countries. Most important is early access to benchmarks that are being developed, and the possibility to participate in technical work on the benchmarks.

    The benefits for associates are the same as for members except that OSG associates have no voint rights. The intention for associates is that they can act in an advisory capacity to SPEC, getting first-hand experience in an area that is widely neglected in academia but nonetheless very important in the "real world", and providing technical input to SPEC's task.

    SPEC meetings (Open Systems Group and SFS subgroup) are held about every two months, for technical work and decisions about the benchmarks. The SPEC High Performance Computing group meets four times in the year.

    Every member or associate can participate in these meetings and make proposals. In the Open Systems Group, decisions are made by the Open Systems Steering Committee (nine members) elected by the general membership at the Annual Meeting. In the High Performance Computing Group, the Steering Committee consists of all members present at a meeting. All SPEC members of OSG or HPG vote before a benchmark is finally accepted.

  13. SPEC Order Form

  14. SPEC Product Order Form

    Line 2:
    Line 3:
    Country/Postal Code:

    1. Shipping and Handling Charges
      (Orders shipped within the U.S. are sent UPS 2nd Day Service)
      • International orders, please add an additional $35
      • Canada & Caribbean Islands, please add an additional $20
      • Three or more tapes $70

    2. Expedited shipments, please provide your overnight service account number or we can also bill directly to a credit card.

      #________________________ Federal Express

      #________________________ United Parcel Service

      #________________________ DHL

      #________________________ Airborne

    3. Multiple Tapes Copies Discount

      [   ]   2-4 = 10%

      [   ]   5-9 = 20%

      [   ]   10+ = 40%

    4. Sale Tax of 4.5% applicable to all orders within the state of Virginia

            Item No.      Products       Unit Cost             Educational**     
            Non-Profit Cost      
        SPEC Newsletter
      (To order back issues, contact the SPEC office)
      N1 [   ]       1-year subscription $   550 550
      N2 [   ]       1-year subscription (International)       $   575 575
        SPEC Benchmark Suites    
      T3   [   ]       SPEC SFS Release 1.1* $1200 600
      T4   [   ]       SPEC CINT92 Release 1.1* $   425 218
      T5   [   ]       SPEC CINT92 Release 1.1* $   575 288
      T7   [   ]       SPEC SDM Release 1.1 $1450 725

                           Subtotal   _______    _______
      1.  Shipping/Handling Charges       
          (see above)                 _______    _______
      2.  Multi-orders discount
          (see above)                 _______    _______
      3.  Expedited Orders
          (see above)                 _______    _______
      4.  Sales Tax -- VIRGINIA ONLY
          (see above)                 _______    _______
                             TOTAL    _______    _______

      Method of Payment

      [     ] Check made payable to SPEC in U.S. Dollars

      [     ] Wire Transfer (contact SPEC office for bank information)

      Credit Card

      [     ] Visa

      [     ] Mastercard

      [     ] American Express

      Name on credit card (cardholder)   ___________________________________

      Credit Card #   ___________________________________________________

      Expiration Date   _________________________________________________

      Signature   ______________________________________________________

      Date   __________________________________________________________

      Please Mail or Fax your order to:

      2722 Merrilee Drive
      Suite 200
      Fairfax, Virginia 22031-4499

      Phone (703)698-9604, ext. 318 - Rodney Sanford
      Fax (703)560-2752

  15. Acknowledgments
  16. This summary of SPEC's activities has been written by Reinhold Weicker (Siemens Nixdora, Paderborn/Germany), originally for the "Usenet" bulletin board under the title "Answers to Frequently Asked Questions about SPEC". Portions of the text have been carried over from an earlier Usenet article written by Eugene Miya (NASA Ames Research Center). Over the months, updates have been made by Jeff Reilly (Intel) and Reinhold Weicker.

    Managerial and technical inquiries about SPEC should be directed to NCGA at

    E-Mail questions that do not interfere too much with our real work :-) can also be mailed to:  from North America         from Europe or elsewhere