From: b...@asiago.cs.wisc.edu (Bart Miller)
Subject: Random testing tech report
Date: 1995/04/14
Message-ID: <3mmae0$iqs@spool.cs.wisc.edu>#1/1
X-Deja-AN: 100441562
organization: U of Wisconsin CS Dept
newsgroups: comp.sources.testers,comp.software.testing

A new tech report is available on the testing and reliability of UNIX utilities
and servers.  The short summary is that we can crash lots of programs on lots
of vendors' machines, using very simple techniques.  We report on the testing
and describe why the programs crash.  You can fetch the paper at:

	ftp://grilled.cs.wisc.edu/technical_papers/fuzz-revisited.ps.Z

The abstract for the report follows:

-------------------------------------------------------------------------------
    "Fuzz Revisited: A Re-examination of the Reliability of
    UNIX Utilities and Services"

    Barton P. Miller, David Koski, Cjin Pheow Lee, Vivekananda Maganty,
    Ravi Murthy, Ajitkumar Natarajan, and Jeff Steidl


    ABSTRACT

    We have tested the reliability of a large collection of basic UNIX utility
    programs, X-Window applications and servers, and network services. We used
    a simple testing method of subjecting these programs to a random input
    stream.  Our testing methods and tools are largely automatic and simple to
    use. We tested programs on nine versions of the UNIX operating system,
    including seven commercial systems and the freely-available GNU utilities
    and Linux.  We report which programs failed on which systems, and identify
    and categorize the causes of these failures.

    The result of our testing is that we can crash (with core dump) or hang
    (infinite loop) over 40% (in the worst case) of the basic programs and
    over 40% of the X-Window applications. We were not able to crash any of
    the network services that we tested nor any of X-Window servers. This
    study parallels our 1990 study (that tested only the basic UNIX utilities);
    all systems that we compared between 1990 and 1995 noticeably improved in
    reliability, but still had significant rates of failure. The reliability of
    the basic utilities from GNU and Linux were noticeably better than those of
    the commercial systems.

    We also tested how utility programs checked their return codes from the
    memory allocation library routines by simulating the unavailability of
    virtual memory. We could crash almost half of the programs that we tested
    in this way.
------------------------------------------------------------------------------