A. MEASUREMENTS

A.1. The bottom line

Although examples often are most illustrative, the cruel world out there is usually more interested in everyday performance figures. To satisfy those people too, we will present a series of measurements on our code expander taken from (close to) real life situations. These include measurements of compile and run times of different programs, compiled with different compilers.

A.2. Compile time measurements

Figure A.2.1 shows compile-time measurements for typical C code: the dhrystone benchmark† dhrystone † To be certain that we only tested the compiler and not the quality of the code in the library, we have added our own version of strcmp and strcpy and have not used the ones present in the library. The numbers represent the duration of each separate pass of the compiler. The numbers at the end of each bar represent the total duration of the compilation process. As with all measurements in this chapter, the quoted time or duration is the sum of user and system time in seconds.

cem:
    C to EM frontend

opt:
    EM peep-hole optimizer

be: EM to assembler backend

cpp:
    Sun's C preprocessor

ccom:
    Sun's C compiler

iropt:
    Sun's optimizer

cg: Sun's code generator

as: Sun's assembler

ld: Sun's linker
    Figure A.2.1: compile-time measurements.

A close examination of the first two bars in fig A.2.1 shows that the maximum achievable compile-time gain compared to cc is about 50% for medium-sized programs.‡ ‡ (cpp+ccom+as+ld)/(cem+as+ld) = 1.53 For small programs the gain will be less, due to the almost constant start-up time of each pass in the compilation process. Only a built-in assembler may increase this number up to 180% in the ideal case that the optimizer, backend and assembler would run in zero time. Speed-ups of 5 to 10 times as mentioned in fast portable compilers are therefore not possible on the Sun-4 family. This is also due to Sun's implementation of saving and restoring register windows. With the current implementation in which only a single window is saved or restored on a register-window overflow, it is very time consuming when programs have highly dynamic stack use due to procedure calls (as is often the case with compilers).

Although we are currently a little slower than cc, it is hard to blame this on our backend. Optimizing the backend so that it would run twice as fast would only reduce the total compilation process by a mere 14%.

Finally it is nice to see that our push/pop-optimization, initially designed to generate faster code, has also increased the compilation speed. (see also figures A.4.1 and A.4.2.)

A.3. Run time performance

Figure A.3.1 shows the run-time performance of different compilers. All results are normalized, where the best available compiler (Sun's compiler with full optimization) is represented by 1.0 on our scale.
Figure A.3.1: run time performance.

The fact that our compiler behaves rather poorly compared to Sun's compiler is due to the fact that the dhrystone benchmark uses relatively many subroutine calls; all of which have to be 'emulated' by our backend.

A.4. Overall performance

In the next two figures we will show the combined run and compile time performance of 'our' compiler (the ACK C frontend and our backend) compared to Sun's C compiler. Figure A.4.1 shows the results from measurements on the dhrystone benchmark.
Figure A.4.1: overall performance on dhrystones.

Fortunately for us, dhrystones are not all there is. The following figure shows the same measurements as the previous one, except this time we took a benchmark that uses no subroutines: an implementation of Eratosthenes' sieve:
Figure A.4.2: overall performance on Eratosthenes' sieve.

Although the above figures speak for themselves, a small comment may be in place. At first it is clear that our compiler is neither faster than cc, nor produces faster code than cc -O4. It should also be noted however, that we do produce better code than cc at only a very small additional cost. It is also worth noticing that push-pop optimization increases run-time speed as well as compile speed. The first seems rather obvious, since optimized code is faster code, but the increase in compile speed may come as a surprise. The main reason is that the as+ld time depends largely on the amount of generated code, which in general depends on the efficiency of the code. Push-pop optimization removes a lot of useless instructions which would otherwise have found their way through to the assembler and the loader. Useless instructions inserted in an early stage in the compilation process will slow down every following stage, so elimination of useless instructions in an early stage, even when it requires a little computational overhead, can often be beneficial to the overall compilation speed.