C Library Work:
===============

Mandate: organize, cleanup, validate
====================================

Distilled Wisdom:
=================
 -  multiple extensions (c,2c,nc,...) are troublesome for make and clarity
 -  multiple source directories (c,c386,a,a386) are confusing
 -  try to put all source for a given function in one place
 -  excessive conditional compilation obscures things
 -  current directory structure is inconsistent
 -  current directory structure is too coarsely grained
 -  current make system is overly complicated
 -  want to be able to invoke one make command to build a top level library

Major To Do Items:
==================
 -  restructure directories
 -  simplify and clean up makefile system
 -  reduce storage used by library hieararchy, (time stamp files)
 -  move to using wasm in place of masm/optasm/386asm etc (phase out asm code)
 -  map out the sequence of events that occur in the runtime support system
    during program execution startup and shut down

Specific Proposed Changes:
==========================
1.  Move the math library out of the clib hierarchy.
    Make a new project for the math library.
    The math library has different requirements than the clib and should
    be separate.  This will help to simplify the structure of the clib.

2.  The new clib hierarchy will consist of two tiers.  The upper tier
    will contain no source code.  It will consist of a target hierarchy
    containing merged libraries and second tier directories containing
    component libraries.
    The target hierarchy will be rooted in clib\library and will contain
    a target directory structure.

3.  The new clib hierarchy will live within the following rules:
    a. all C source files go into a 'c' directory, .c extension
    b. all ASM source files go into a 'a' directory, .asm extension
    c. all header files go into a 'h' directory, .h extension
    d. all object files go into a target directory, .obj extension

4.  In place of the former extension/directory differentiation, file names
    will have suffixes as follows:
        generic         - name
        Intel 16 bit    - name086
        Intel 32 bit    - name386
        os2             - nameos2
        os2 1.x         - nameo16
        os2 2.x         - nameo32
        NT              - namewnt
        Win386          - namew32
        QNX             - nameqnx
        QNX 16          - nameq16
        QNX 32          - nameq32
        NetWare         - namenov
    This may lead to some slightly cramped names.
    The goal is to support having all source for a given function in one
    directory.  The intention is that the finely grained directory structure
    should allow easy selection of the right directory, and then wgrep can be
    used to locate all the files that contain the function.

5.  Existing groupings of files into library components (ANSI,DOS,_DOS,HEAP)
    will be subdivided as follows:
        char            - character manipulation functions
        memory          - memory manipulation functions
        string          - string manipulation functions
        mbyte           - multibyte manipulation functions
        convert         - conversion manipulation functions
        heap            - heap manipulation functions
        search          - searching and sorting functions
        time            - time functions
        streamio        - stream i/o functions
        process         - process primitive functions
        environ         - process environment functions
        direct          - directory management functions
        handleio        - handle level i/o functions
        file            - file manipulation functions
        conio           - console i/o functions
        bios            - BIOS functions
        _dos            - MS-DOS functions
        intel           - 80x86 functions
        misc            - miscellaneous functions
        startup         - run time startup functions
        cgsupp          - compiler support routines
        kanji           - kanji support routines
        mthread         - multi-threading support functions
        defwin          - default windowing system
        qnx             - qnx specific functions
        ...             - there are, no doubt, more
    This is based on the Overview section of the C Library Reference.
    Files that are specific to one platform, and have no analog on any
    other platform will be located the platform specific area.
    (i.e. qnx,...)

6.  The following first tier target libraries will be built:
        Dos16           - tiny,ms,mm,mc,ml,mh,overlay mm,overlay ml
        Dos32           - register ms,stack ms
        OS/2 1.x        - ms,mm,mc,ml,mh,multi-thread ml,DLL ml
        OS/2 2.x        - multi-thread DLL register ms,
                          multi-thread DLL stack ms
        QNX 16          - ms,mm,mc,ml,mh,share
        QNX 32          - register ms
        Win 3.1         - ms,mm,mc,ml
        Win386          - multi-thread DLL register mf,
                          multi-thread DLL stack mf
        Win NT          - multi-thread DLL register ms,
                          multi-thread DLL stack ms
        NetWare         - mulit-thread stack ms
        Kanji 16        - ms,mm,mc,ml,mh
        Kanji 32        - register ms, stack ms
        nec 32          - register ms, stack ms
    Some of these target systems are really /mf but we are building /ms.
    This is to avoid problems with calling into the /ms math library.
    It means that all calls out to user code (including startup) must
    be bracketed with a setup of the segment registers for FLAT and an
    assumption that the user code trashes the segment registers on return.
    This is required for the /r switch that is used to build the library.
    The remaining /mf libraries (Win386??) either don't have
    the math issue or have other issues.

7.  The following naming convention will be used for the first and
    second tier target hierarchies:

       <system>.<processor>\<model>_
                            [<convention>]              - optional
                            [<overlays>]                - optional
                            [<threading>]               - optional
                            [<dll>]                     - optional
                            [<share>]                   - optional
                            [<debug>]                   - optional

   where:
    <system> is one of:
        msdos|qnx|os2|windows|winnt|netware|kanji|necdos
    <processor> is one of
        086|286|386|hob|alf|mip|rs6|...
    <model> is one of:
        tiny            - mt
        small           - ms
        medium          - mm
        compact         - mc
        large           - ml
        huge            - mh
        flat            - mf
    <convention> is only used on 32 bit targets and is one of:
        register        - r
        stack           - s
    <overlays> is only used to designate 16 bit overlaid libs and
        is indicated by - ov
    <threading> is only used to designate 16 bit multi-thread libs and
        is indicated by - mt
    <dll> is only used to designate 16 bit DLL libs and is indicated
        by              - dll
    <share> is only used to designate shareable libs and is indicated
        by              - sh
    <debug> is only used to designate debugging libs and is indicated
        by              - db

    This scheme, coupled with the intended targets, produces the following
    first tier target hierarchy:
        msdos.086\
            mt ms mm mc ml mh mm_ov ml_ov
        msdos.286\
            ms mm mc ml mh ml_mt ml_dll
        msdos.386\
            ms_r ms_s
        os2.286\
            ms mm mc ml mh ml_mt ml_dll
        os2.386\
            ms_r ms_s
        qnx.286\
            ms mm mc ml mh ml_sh
        qnx.386\
            ms_r
        windows.086\
            ms mm mc ml
        windows.386\
            mf_r mf_s
        winnt.386\
            ms_r ms_s
        netware.386\
            ms_s
        kanji.086\
            ms mm mc ml mh
        kanji.386\
            ms_r ms_s
        necdos.386\
            ms_r ms_s

8.  The making of the libraries is still under study.  It has been
    suggested that VIPER (vmake.exe) could be used to maintain the structure
    of the clib and to build its components.  Alternatively, a modification
    of the existing makefile system could be used.
    The following two points describe how the clib could be maintained
    under each system.

9.  The makefile systems will be organized around a tree structure.

    a.  There are three basic make operations:
          i) transform source into object (assemble,compile)
             (no pre-process, as wasm.exe obviates this step)
         ii) combine object into component library
        iii) combine component libraries into complete libraries
        The first 2 operations occur at the second tier of the directory
        structure, the last operation occurs at the top tier of the directory
        structure.

    b.  Each component library will have a two .mif files in the second
        tier.
        One will be optional and list any important dependancies.
        The other will list all the objects required for each of the
        required targets. It will be implemented with make macros
        (potentially nested) to ensure the following:
          i) no object file will be listed more than once
         ii) the addition of an object file will require exactly one change

    c.  Each component target directory will have a makefile that sets several
        options and then includes the .mif file which lists the source/object
        files and then includes one or more .mif files out of the root to
        drive the build.  No model.inc file will be required, thanks to
        the capabilities of wasm.exe.
        Finally, if there are dependancies, the dependancy file will be
        included.

    d.  Each complete target directory will have a makefile that lists the
        required component libraries and then includes a .mif file out of the
        root to drive the merging of the component libraries.

    e.  All other .mif files will reside in the root of the library.
        These .mif files will be organized around the concepts of building
        libraries and merging libraries.
        The building .mif files will attempt to factor out as much
        redundancy as possible, without obscuring the intent of the build.
        The intention is to allow modifications to the library build to be
        effected with a change to one .mif file.
        The merging .mif files will be driven by the list of component
        libraries required for the complete library.

    f.  The top level .mif files will also support the specification of
        extra_cflags, extra_aflags, extra_cppflags, extra_lflags, in the
        target makefiles to allow for special cases.

    g.  Support for controlled makes will be driven by keywords in the
        makefiles. Each makefile will contain several comments at the
        begining that can be used to list the targets that will require
        this library. The categories are as follows:
            all                                         - all targets
            tiny,small,compact,medium,large,huge,flat,
            nomodel                                     - memory model
            16bit,32bit                                 - code target
            stack,register                              - calling convention
            inline,calls                                - floating point
            heap,_dos,nec,string,streamio,handleio,
            filesys,startup,process,cgsupp, kanji,mbyte,
            mthread,defwin,qnx                          - component of library
            overlay                                     - overlay libraries
            threading                                   - multi-thread library
            dll                                         - dll support
            share                                       - sharable library
            debug                                       - debugging library
            dos,os2,qnx,windows,win386,nt,netware,
            kanji,nec                                   - target system

        As an example, the o1lmt target directory will contain the following
            #pmake:all
            #pmake:large
            #pmake:threading
            #pmake:16bit
            #pmake:os2
        A pmake.exe (modified from sweep.exe for depth-first traversal and
        text pattern matching) will visit each target directory and run
        wmake.exe if the makefile contains all the specified targets.

        i.e.
            pmake.exe large threading os2 16bit
                Would sweep the clib and execute the makefile for os2 1.x
                large multi-thread libraries.
            pmake.exe all
                Would sweep the clib and execute the makefile for all targets
                since all makefiles will contain #target=all
            pmake.exe small dos 16bit
                Would sweep the clib and execute the makefile for all small
                dos 16 bit libraries.  This would effectively build the
                ``\lang\lib286\dos\clibs.lib'' file.


10. Viper (vmake.exe) is driven by configuration (viper.cfg), project (.prj)
    target (.tgt), and zombie target (.tgt) files.

    a.  A target (.tgt) file will be created for each .lib file to be
        built.
          - viper allows nesting of zombie target (.tgt) files
          - base zombie files will be defined that simply list object files
          - nested zombie files will be build up that contain other zombie
              files to generate a complete list of objects needed for a
              specific component library
          - a target (.tgt) file will be defined for each .lib file to be
              created, it will contain a zombie that lists all the object
              files required

    b.  A project (.vpr) file for each clib subcomponent (heap,cgsupp...)
        will be defined.
          - when viper is invoked, it takes a project (.vpr) file
          - the project (.vpr) file will point to the target (.tgt) files
              which define each .lib that is created as part of this
              subcomponent
          - thus the clib will consist of approx 20 projects, each containing
              approx 50 target libraries

    c.  The configuration file (viper.cfg) defines all rules used to convert
        input files to output files.
          - when a .tgt file is created it specifies what form of target
              will be created
          - this form can be used to select options out of the conversion
              rules
          - by defining a large number of target forms, one for each target
              library (approx 50), the selection of switches can be driven
              automatically

    d.  Bviper.exe is a batch mode viper that supports generation of a
        makefile to do the work defined by a set of targets. Build machine
        construction of the library would consist of invocations of
        bviper.exe followed by wmake.exe in each project (.vpr) directory.

11. The following targets will be validated:
        Dos16
        Dos16 Pharlap286
        Dos16 16M
        Dos32 Pharlap386
        Dos32 Dos4g
        Dos32 Ergo
        Dos32 CodeBuilder
        Dos32 FlashTek
        OS/2 1.x
        OS/2 2.x
        OS/2 2.x default windowing
        QNX 16
        QNX 32
        Windows 3.1
        Windows 3.1 default windowing
        Win386
        Win386 default windowing
        Windows NT
        Windows NT default windowing
        NetWare 3.11
        NetWare 4.0
        nec - what needs to be tested here?

    Validation will consist of the set of examples found in the C Lib Reference.
    Expansion of the validation test suite should also be performed.

12. The validation process will be automated as much as possible.
    The test environment can be encapsulated into one machine.  The major
    requirement is 2 physical hard disks to allow 8 partition table entries.
    The system needs to be able to boot:
        DOS,NT,QNX,OS/2 2.x,NetWare 3.11,NetWare 4.0
    The systems needs to have installed:
        Windows,WIN-OS2,various DOS extenders


Items Not Yet Addressed:
========================
1.  Implementation of file time stamp method.
2.  Mapping of startup/shutdown events.

Appendix A:
===========

This is a summary of the switches to be used in build the CLIB:

Notes:
======
- make QNX source files conditionally include "shlib.h" when
  "defined(__QNX__) && defined(__SHARE__)"
- make files that are affected by \clib\dos\2h\dll.h conditionally include it
  when "defined(__OS2__) && defined(__DLL__)"
- assume wasm gets a -bt=<id> switch
- wasm.exe needs /fpi87 or equivalent to /e in masm
- define wasm.inc as interim to convert cmd switches to macros
  a. change -3r,-3s to STACK=0, STACK=1
  b. change -ms,-mm,-mc,-ml,-mh to _MODEL=...
  c. change 16 bit BIG_DATA -bt=WINDOWS to _DS_PEGGED
  d. change 32 bit -bt=OS2 to _OS220
  e. change 32 bit -bt=NT to _NT
  f. change 32 bit -bt=NETWARE to __NET386__
- modify net386 C source so that
  a. -bt=NETWARE works for __NET386__

Common All:
===========
wlib:                   /q /n /b (/s /t)-> build machine
C flags:                /r /fpr /w3 /zq /zc /zl /zld /fpc
wasm include:           \clib
wasm flags:             /q

==============================================================================
MS-DOS Settings:
================
wasm flags:             /ms
                        /mm
                        /mc
                        /ml
                 (huge) /ml
                        /3r
                        /3s
                        /fpi87 (on some floating point files)
                        /dERGO_DPMI (for cstart.asm)
                        /dADS (adsstart.obj[adsstart.obj])
                        /dPADI (adsstart.asm[adifstrt.obj])
                        /dPADI /dEADI (adsstart.asm[adiestrt.obj])
C include:              ..\h \clib\h \hdr\dos
C flags:                /ms     /nt=_TEXT
                        /mm
                        /mc /zu /nt=_TEXT
              (overlay) /mm     /of
                        /ml /zu
              (overlay) /ml /zu /of
                        /mh /zu
                        /ms /5r
                        /ms /5s
special case C flags:   /oasl /s (for certain files)
                        /oax (for all other files)
                        /of (for certain 16 bit files)

Current Directories:    cgsupp\(scsd,scbd,bcsd,bcbd,bchd)
                        ansi\(scsd,scbd,bcsd,bcbd,bchd)
                        dos\(scsd,scbd,bcsd,bcsdf,bcbd,bcbdf,bchd)
                        _dos(scsd,scbd,bcsd,bcbd,bchd)
                        heap\(scsd,scbd,bcsd,bcbd,bchd)
                        (tiny,scsd,scbd,bcsd,bcbd,bchd)
                        cgsupp\(ms386r,ms386s)
                        ansi\(ms386r,ms386s)
                        dos\(ms386r,ms386s)
                        _dos\(ms386r,ms386s)
                        (ms386r,ms386s)

==============================================================================
QNX Settings:
=============
wasm flags:             /bt=QNX
                        /ms
                        /mm
                        /mc
                        /ml
                        /mh
                        /ms /3r
C include:              ..\h \clib\h \clib\ansi\h \hdr\qnx
C flags:                /bt=QNX
                        /ms     /nt=_TEXT
                        /mm
                        /mc /zu /nt=_TEXT
                        /ml /zu
                        /mh /zu
                (share) /ml /zu /d__SHARE__
                        /ms /5r
special case C flags:   /oasl /s (for certain files)
                        /oax (for all other files)
                        /of (for certain 16 bit files)

Current Directories:    qnx\stub\(scsd,scbd,bcsd,bcbd,bchd)
                        qnx\(qscsd,qscbd,qbcsd,qbcbd,qbchd,qshare)
                        ansi\(qscsd,qscbd,qbcsd,qbcbd,qbchd,qshare)
                        dos\(qscsd,qscbd,qbcsd,qbcbd,qbchd,qshare)
                        heap\(qscsd,qscbd,qbcsd,qbcbd,qbchd,qshare)
                        (qscsd,qscbd,qbcsd,qbcbd,qbchd,qshare)
                        qnx\(q3r)
                        ansi\(q3r)
                        dos\(q3r)
                        heap\(q3r)
                        (q3r)

==============================================================================
OS/2 Settings:
==============
wasm flags:             /bt=OS2
                        /ms
                        /mm
                        /mc
                        /ml
                        /mh
                   (mt) /ml /d__MT__
                  (dll) /ml /d__MT__
                        /ms /3r /d__MT__? (some 32bit asm files)
                        /ms /3s /d__MT__? (some 32bit asm files)
16bit C include:        ..\h \clib\dos\h \clib\h \hdr\dos \watcom\h
32bit C include:        ..\h \clib\dos\h \clib\h \hdr\dos \watcom\os220_h
                        \watcom\h
C flags:                /bt=OS2
                        /ms     /nt=_TEXT
                        /mm
                        /mc /zu /nt=_TEXT
                        /ml /zu
                        /mh /zu
                   (mt) /ml /zu /d__MT__
                  (dll) /ml /zu /d__MT__ /d__DLL__
                        /ms /5r /d__MT__ /d__OS220__
                        /ms /5s /d__MT__ /d__OS220__
special case C flags:   /osl /s (for certain files)
                        /ox (for all other files)
                        /of (for certain 16 bit files)

Current Directories:    ansi\(os2mt,os2dll)
                        dos\(2scsd,2scbd,2bcsd,2bcbd,2bchd,os2mt,os2dll)
                        heap\(2scsd,2scbd,2bcsd,2bcbd,2bchd,os2mt,os2dll)
                        (2scsd,2scbd,2bcsd,2bcbd,2bchd,os2mt,os2dll)
                        ansi\(23r,23s)
                        dos\(23r,23s)
                        heap\(23r,23s)
                        (23r,23s)

==============================================================================
Windows Settings:
=================
wasm flags:             /bt=WINDOWS
                        /ms
                        /mm
                        /mc
                        /ml
                        /mf? /3r
                        /mf? /3s
C include:              ..\h \clib\h \hdr\dos
C flags:                /bt=WINDOWS
                        /ms     /nt=_TEXT
                        /mm
                        /mc /zu /nt=_TEXT
                        /ml /zu
                        /mf? /5r
                        /mf? /5s
special case C flags:   /oasl /s (for certain files)
                        /oax (for other files)
                        /of (for certain files)

Current Directories:    ansi\(wscsd,wscbd,wbcsd,wbcbd)
                        dos\(wscsd,wscbd,wbcsd,wbcbd)
                        _dos(wscsd,wscbd,wbcsd,wbcbd)
                        heap\(wscsd,wscbd,wbcsd,wbcbd)
                        (wscsd,wscbd,wbcsd,wbcbd)
                        ansi\(w3r,w3s)
                        dos\(w3r,w3s)
                        _dos\(w3r,w3s)
                        heap\(w3r,w3s)
                        (w3r,w3s)

==============================================================================
NT Settings:
============
wasm flags:             /bt=NT /ms /d__MT__
                        /3r
                        /3s
C include:              ..\h \clib\h \clib\dos\nth \hdr\dos \watcom\nt_h
                        \watcom\h
C flags:                /bt=NT /ms /d__MT__
                        /5r
                        /5s
special case flags:     /oasl /s (for certain files)
                        /oax (for rest of files)

Current Directories:    ansi\(ntr,nts)
                        dos\(ntr,nts)
                        heap\(ntr,nts)
                        (ntr,nts)

==============================================================================
NetWare Settings:
=================
wasm flags:             /bt=NETWARE /ms /3s
C include:              \clib\h ..\h \hdr\dos
common C flags:         /bt=NETWARE /ms /5s /ox /d__far= /d__MT__

Current Directories:    ansi\(net386)
                        dos\(net386)
                        (net386)

==============================================================================
NEC PC Settings:
================
C include:              ..\h \clib\h \hdr\dos
C flags:                /ms /ox
                        5r
                        5s

Current Directories:    _nec(ms386r,ms386s)

==============================================================================
Windows Supervisor 32 bit: wind386\(w3r,w3s)
==========================
wasm flags:             /bt=WINDOWS /ms
                        /3r
                        /3s
C include:              ..\h \clib\h \hdr\dos
common C flags:         /bt=WINDOWS /ms /ox
                        5r
                        5s

