Report on the Canadian Governments

Microcomputer Procurement and PC Benchmarking

This report was prepared by John Clark of the Strategic IT Analysis Division, Office of Information Technology, Office of Governmentwide Policy, General Services Administration. For further information, contact John Clark at (202) 501-4362 or David Middledorf at (202) 501-1551.

BACKGROUND:

The Public Works and Government Services, Canada (PWGSC) introduced a new approach to the acquisition of microcomputer systems (PCs) which includes PC Benchmarking. Prior to this new approach, PWGSC acquired PCs strictly on lowest price based on detailed technical and performance specifications. That process was time consuming (12 to 18 months from solicitation to contract award) and often resulted in the acquisition of poor quality and obsolete PCs.

INTRODUCTION:

The Canadian Informatics Systems Procurement Directorate established the National Master Standing Offerors (NMSO) program for the following commodities:

• Desktop PCs
• Notebook PCs
• Servers
• RISC Systems
• Local Area Networks (LANs)
• PC Peripherals

NMSO is the primary procurement vehicle for microcomputer systems. Pricing and evaluation results are published electronically via the World Wide Web (WWW) and a Faxline Service. Clients (agencies) issue directed orders to NMSO vendors up to $40K ($60K for high-end servers). PWGSC conducts requests for volume discounts (RVDs) for agency PC requirements between $40,000 to $2,000,000. Custom procurements are conducted by PWGSC for agency requirements over $2,000,000. PWGSC still buys PCs based on lowest price but the NMSO program includes a PC Benchmarking process conducted in conjunction with the National Software Testing Laboratory (NSTL) of Philadelphia, PA, to ensure that only quality microcomputer systems are bought.

DISCUSSION:

The NMSO procedures include the following:

• Defined requirements; category specifications and vendor qualifications,
• Issue competitive request for Standing Offer (RFSO) that includes question and answer process during bid period,
• Evaluate proposals, as detailed in the RFSO:
  • mandatory requirements
  • preliminary technical
  • financial
  • benchmark
• Award NMSO's
  • maximum number of systems/vendor per category
  • systems - up to mean price plus one standard deviation
  • vendors - maximum 15 per category

Currently, NMSO has four categories of microcomputer systems as follows:

Systems Category:

• 1.0 Value 75 MHz Standalone Systems
• 1.1 Advanced 75 MHz Standalone Systems
• 2.0 Value 100 MHz Standalone Systems
• 2.1 Advanced 100 MHz Standalone Systems

Notebook Category:

• 1.0 DX4/75 MHz Passive Matrix Notebooks
• 1.1 DX4/75 MHz Active Matrix Notebook Systems
• 2.0 Pentium 75 MHz Passive Matrix Notebooks
• 2.1 Pentium 75 MHz Active Matrix Notebooks

Server Category: 3.0 Value 100 MHz Servers

SQL Server Category: 3.1 Advanced Dual 100 MHz Servers

As mentioned earlier, clients (agencies) may issue directed orders to NMSO vendors up to $40K ($60K for high-end servers). PWGSC conducts requests for volume discounts (RVDs) for agency PC requirements between $40,000 to $2,000,000.

Characteristics of RVDs include the following:

• vendors are requested to provide their best for a volume purchase
• Issued to all vendors listed in the selected category
• Must bid systems from the defined category
• Specifications cannot restrict competition
• Financial evaluation incorporates the overall benchmark score
• Results in approximately 25% savings versus NMSO prices

In effort to keep with the dynamics of the microcomputer market, product substitutions are allowed with the following limitations/results:

Substitutions require a technical evaluation and most require a benchmark if performance may be affected.

Product life cycle is approximately 6 months with an average of 250 product substitutions per year.

PWGSC plans to issue new RFSOs for advanced categories of PCs as technology and client demand dictate.

During the first 6 months of calendar year 1996, the NMSO program had $108.6 million of business. Agency direct orders accounted for approximately $83.3 million of NMSO business volume with the PWGSC RVDs accounting for the remaining $25.3 million.

Custom procurements are conducted agency requirements over $2,000,000. PWGSC encourages agencies to include the PC Benchmarking process in their custom requests for proposals (RFPs).

PWGSC objectives for the NMSO program are as follows:

• Ensure that clients receive the best desktop, portable and server computers at the best possible price.
• Provide the up-to-date information necessary for clients to make informed procurement decisions.
• Provide an easy to use service which allows all of clients to receive the best possible price when acquiring high volume microcomputer hardware.

PC Benchmarking

PWGSC goals for PC Benchmarking are as follows:

• Reduce life cycle costs related to poor quality, incompatible products and poor vendor support.
  • Audit against specifications
  • Compatibility and stress tests
• Quantify microcomputer's relative value based on:
  • Performance
  • Features
  • Usability
  • Problems

Benchmarking Development Approach:

• Originally departments conducted their own benchmarks for all procurements.
  • Lengthy process, inefficient, led to vendor disputes.
• PWGSC developed Benchmarking process.
  • Program funded through benchmark fees collected from vendors.
  • RFSO and testing conducted on 18 month cycle with one mid term refresh, results published on hard copy every 9 months.
• PWGSC instituted a substitution process.
  • Results published on CD-ROM every 6 months
• PWGSC instituted continuous RFSO's and testing
  • Results published daily
  • New categories are added as technology and demand dictate.

PWGSC competitively selected the National Software Testing Laboratories (NSTL) to develop the benchmark methodology.

Advantages of using NSTL:

• Project management, publication facilities
• Impartiality provides vendor respect/acceptance
• Custom development of test suites to meet our requirements
• Process enhancements: WWW publication, WWW Acquisition Guide, configuration auditing tools

PWGSC Benchmarking responsibilities:

• Prepare and maintain specifications and benchmark rules.
• Provide the facilities.
• Identify Operating System and applications to be used for testing.
• Identify desirable features.
• Accept systems for benchmarking.
• Verify specifications compliance.
• Administer resolution of compatibility issues that arise.
• Disqualify products that do not comply with procedures re technical and administration faults.
• Specify content and structure of benchmark report.
• Provide pricing database.

NSTL Benchmarking responsibilities:

• Provide technical support for technical and administrative faults.
• Recommend desirable features for specifications.
• Control the register of products during testing.
• Provide and manage the testing team.
• Conduct the testing.
• Manage Technical elements of faults and compatibility issues.
• Confirm resolution of faults and compatibility issues.
• Prepare benchmark report.

PWGSC provides clients with the up-to-date information necessary to make informed procurement decisions and simplify ordering.

NMSO Benchmark results are On-line:

• Provide microcomputer benchmark and price data via the INTERNET at http://nstl.srv.gc.ca

Electronic Microcomputer Acquisition Guide:

• Provides price and configuration data for Microcomputer hardware and software via the TRANSACT BBS (819) 956-7179.
• Produces call-up form for client (agency).

Faxline:

• Provides complete Standing Offer text and pricing for microcomputer hardware, software, printers and RISC workstations.
• 956-7139.

Automated Configuration Auditing:

PWGSC has instituted an follow-up audit capability to their PC Benchmarking process. PWGSC and NSTL have developed the "Magic Diskette" which will verify that the following components in a client's system correspond to those in the benchmarked system:

• CPU and clock rate
• Hard drive
• Drive controller
• Video card
• Video memory
• Motherboard
• BIOS
• Modem
• UART's
• Parallel Ports
• RAM size
• Cache and type
• Drivers in use

PWGSC also established a Software Brokerage Service under the NMSO program. The Software Brokerage Service provides an easy to use which allows all of PWGSC's clients to receive the best possible price when acquiring high volume microcomputer software.

NMSO program approach follows:

• Identify target products
• Negotiate best possible deal based on potential orders from entire Federal government
• Distribute "right to copy licenses"
• Provide usage reports to vendors quarterly
• Vendors invoice PWGSC based on PWGSC's usage reports
• Program is self funding with nominal markup

Technical Issues

(lesson learned):

• Review every specification prior to release of solicitation document.
• Test systems for quality and "overall value".
• Develop a configuration control system to minimize "bait and switch" and "motherboard of the month syndrome".
• Audit deliveries.
• Publish system configuration, test results and pricing data.
• Publication of Benchmark report resulted in Quality leapfrogging by Vendors

NMSO versus Custom RFP's

Pro NMSO:

• Procurement Time
  • Direct Orders (48 hours delivery from order)
  • RVD up to $2 million (approximately 1 to 2 weeks for delivery from RVD release)
• Pro Custom
  • No artificial dollar limit like the NMSO's $2 million per RVD
  • "Unique" requirements satisfied
  • Same supplier(s) for duration of contract

Procurement Issues:

• NMSO is an efficient method of supply for small to medium value procurements of general purpose microcomputer systems.
• NMSO encourages competition for small orders as pricing is public domain.
• RVD's increase competition for medium value orders.
• Using overall technical benchmark results in financial evaluations yields better value for money for PWGSC customers.

Conclusions:

PWGSC's NMSO program with PC Benchmarking has been very successful in providing an easy to use contractual vehicle to acquire quality microcomputer systems in a timely manner and at reasonable prices. However, some users suggested that thresholds are too low, $40,000 for direct orders and $2,000,000 for RVDs. Additionally, some of the NMSO vendors do not participate in the RVD acquisition. In this regard, approximately 76% of NMSO business comes from agencies' directed orders. RVD acquisitions result in an approximately 25% discount off vendor's NMSO pricing. There is also limited number of vendor slots in the NMSO program. Benchmarking requirements for product substitutes delay introduction of new features. Additionally, each Benchmark costs $2300 with $1800 going to NSTL and $500 to PWGSC to finance the NMSO's PC Benchmarking program.

Canada's PC Benchmarking process does not currently include performance standards for computer accommodations for users with disabilities. However, the PC benchmarking process does ensure robust input/output ports and additional expansion slots. These features are very important for users with disabilities. In this regard, when a computer system is equipped with hardware or software to make it useful for people with disabilities, it always involves the addition of either input or output devices that must operate concurrently with all other computer functions, yet be transparent to those functions. For this reason, adaptive technologies operate most efficiently on systems that "tolerate" a wide variety of add-ons.

GSA's Center for Information Technology Accommodation states that a system's hardware needs to be judged for its accessibility on the following features:

Availability of expansion slots - Systems that utilize most of their own expansion slots for boards necessary to perform routine computing tasks seldom leave room for expansion cards required by many adaptive systems.

Integrated components - If a computer has several of its components especially the video output control chips installed directly on the system board, it may preclude the use of alternative devices. Often alternative devices, such as video boards, can be added to such a system but generally requires a person with technical knowledge of the specific brand of computer being supplied. Systems that do not fully integrate their video and keyboard functions are more accommodating of alternative devices.

Ports and connectors - A system purchased with the expectation that it will now or in the future be used by persons with disabilities should have the maximum number of serial ports and at least two parallel ports. These will allow for the most flexibility when connecting external devices to the system. Proprietary connectors such as those used on some systems preclude the use of alternative devices and should be avoided. The same is true for built-in tracking or pointing devices. Systems that do not allow for the connection of external trackballs, etc. would not be considered particularly adaptable.

Due to the fact that when testing for accessibility, one is actually testing for expandability or flexibility, it is not necessary to actually test such devices as speech braille or large print systems with each computer being benchmarked. Rather, a comprehensive report assessing the ease of adding software and hardware components to system will serve as an accurate gauge to accessibility.

Canada's PC Benchmarking process does a good if not complete job in this respect to ensuring systems that can be adapted for users with disabilities.

In closing, a sample of the NMSO Benchmark report is provided for your information.

As an example, under Advanced 100MHz Standalone Systems category, 10 vendors are identified with their respective performance, features, and usability ratings and price/performance comparison (See attachment).

Complete information about Canada's benchmarking program is available from

INTERNET: http://nstl.srv.gc.ca/sipss96/toc.htm, including the overall scoring methodology.

Feedback:

GSA would like to know what the agencies think about PC Benchmarking. Based on the Canadian Model for PC Benchmarking, would a similar program be appropriate for use by the GSA Multiple Award Schedules (MAS) or Federal agencies' microcomputer system acquisitions? Is there a concern or problem regarding the price and quality of PCs acquired by Federal agencies under current procurement practices? Your feedback is appreciated greatly. Please send your comments to: GSA (MKS), 18th and F Streets, NW, Room 2014, Washington DC 20405 or john.clark@gsa.gov.