Operations of the Santa Cruz County

Information Services Department

 

Background

 

The County Information Services Department (ISD) provides centralized automation, telephone, printing and other support services to County departments to improve productivity and service. The department is responsible for managing the County’s wide-area network and operates and supports a broad range of data processing applications for County departments including public safety applications, which require twenty-four hours a day, seven days per week reliability.

 

Scope

 

This report examines the use of personnel and financial resources to provide county wide “information services/data processing”, which is a major sub-function of the ISD. This report will discuss the appropriateness of the current hardware and software architecture to support modern county operations.

 

Findings

 

  1. The ISD uses mainframe computer hardware first introduced in 1990, which is an upgrade from earlier versions.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

The County agrees with this finding, with the following clarification. The current IBM mainframe is a model 9672, which was first introduced in September 1996.  This basic mainframe architecture has been around for many years, but later models use increasingly reliable and efficient components. ISD also supports new technology server platforms to host many new applications.

 

  1. Current mainframe software includes applications for the Tax Collector, Assessor, Auditor, Payroll, Personnel, District Attorney, Welfare and Planning departments.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

In addition, the Sheriff’s Jail Management and Records systems and the Court’s Civil system should be added to this list.

 

  1. The Planning Department is in the process of migrating away from the mainframe.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

  1. The amount allocated for Information Services/Data Processing in the 2002-2003 county budget is in excess of $9 million.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

  1. Using the financial data provided by the ISD, the Grand Jury estimates that $1.1 million is spent annually on outside hardware, software and other services to support mainframe architectures. The exact costs for Data Processing mainframe operations versus other server operations are not delineated in the published budget.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

The County agrees with this finding with the following clarification. Supporting details for the 2002-2003 budget year anticipated that costs for services, supplies and depreciation for mainframe hardware, software support and other services (excluding staff) would be $1,019,221.  These costs for non-mainframe support (excluding staff) were anticipated to be $1,518,253.

 

  1. Mainframe architectures are expensive to sustain and difficult to evolve to current industry best practices such as web access. To support these mainframe operations, the ISD employs 32 highly skilled programmers and technicians. Given the financial and organizational data provided by the ISD, the Grand Jury estimates this cost to be $3 million annually. The exact Data Processing personnel costs for mainframe operations versus other server operations are not delineated in the published budget.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

The County agrees with this finding with the following clarifications. Support for the mainframe requires some percentage of time from 39 separate individuals.  True costs for operators, systems software and network staff were developed by estimating a percentage of total time each individual spends on mainframe related work. This totaled $388,861.  Mainframe programming support costs were developed based on project time actually recorded by programming staff. This totaled $1,096,930.  Total mainframe staff support costs totaled $1,485,561 for FY 2002-03.

 

  1. The County continues to use a mainframe internal billing structure to allocate ISD expenses across county operations. This requires administrative personnel involved in determining and allocating costs of mainframe usage for the purpose of interdepartmental billing. The Data Processing Department is charged $709,000 annually by ISD for “Management Services.” Some undetermined portion of this is for figuring the internal cross department mainframe charges and administrative overhead.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

The County agrees with this finding with the following clarification. Supporting details for the 2002-2003 budget year anticipated that administrative staff overhead costs would be $117,700.

 

  1. The published Data Processing budget does not delineate monies spent on mainframe architectures versus other server based services.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

The County agrees with this finding, with the clarification that the detail upon which the budget is based does identify anticipated mainframe expenditures.

 

  1. The ISD understands the ultimate need to eliminate the mainframe computer operations by migrating to less expensive current technology, but does not have a formal plan or time line for accomplishing this.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

The County agrees with this finding, with the clarification that the department does have plans and timelines for converting several mainframe applications. New systems are underway in the Planning Department and the District Attorney’s office. Plans and timelines for the migration of the Courts and Human Resources Agency systems are underway, however, these efforts are largely impacted by statewide issues. The Sheriff’s Department also is in the process of implementing its technology plan that provides for mobile data computing (largely completed), a new message switch (underway), and a new records management system for patrol and detention (future).

 

  1. The mainframe software applications are a combination of “in house” and commercial software adapted “in house”. These applications have evolved over a span of four decades.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

  1. Independent investigations by the Grand Jury determined that “Commercial off the shelf” (COTS) software is available for county governmental operations from many vendors for a variety of computing platforms.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

The County agrees with this finding with the following clarification.

            Even though COTS solutions for local government applications exist, many would require extensive customization to meet Santa Cruz County requirements.

 

  1. However, the ISD has a policy to use Windows NT based servers for new server based applications which the limits the available options.

 

Response:  Santa Cruz County Board of Supervisors

 

In the past, the County has selected Windows NT based servers to address business needs. However, it is not ISD’s policy to limit selection of solutions to one product type. Recently the County has deployed five newer servers using the Linux operating system.

 

  1. The ISD staff has reported having difficulty finding COTS software applications to replace the current mainframe applications.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

  1. Santa Cruz County is in a severe budgetary crisis.

 

Response:  Santa Cruz County Board of Supervisors AGREES

 

Conclusions

 

  1. The County is operating a midsized mainframe software organization.

 

  1. As operations are moved away from the mainframe, such as the Planning Department, the ISD mainframe costs for the remaining departments will become more prohibitive. There will be fewer departments sharing the fixed overhead.

 

  1. It is difficult to manage the costs of using mainframe architectures when the exact costs are not delineated.

 

  1. The monies spent on maintaining the use of the mainframe computer hardware, software and personnel could be spent more effectively on contemporary industry standard server-based hardware with COTS software solutions.

 

  1. The ISD policy to only use Windows NT for new server applications artificially restricts candidate COTS software solutions which could replace the mainframe software.

 

  1. The ISD must find ways to reduce Data Processing costs.

 

  1. The ISD needs outside help to facilitate a migration from mainframe architectures to current vendor supplied hardware and software technology.

 

  1. The ISD department does not have the expertise to simultaneously maintain and operate mainframe architectures and at the same time migrate applications to new platforms.

 

Recommendations

 

1.      The ISD should consult with many solution vendors and hire a consulting firm to facilitate the selection of and migration to current hardware and software technology.

 

 

 

Response:  Santa Cruz County Board of Supervisors

 

This recommendation is being implemented. A preliminary assessment of mainframe costs has been initiated with a planned completion date of December 2003. Once the assessment of the remaining applications is complete, a plan for application migration will be developed. The plan will tell us if outside expertise is required. Until the plan is developed, the need for consulting with many solution vendors is not advisable.

 

2.      The ISD should discontinue any new development for mainframe applications and only maintain those which are required for current operations.

 

Response:  Santa Cruz County Board of Supervisors

 

This recommendation has been implemented. ISD has not developed a new mainframe system in 10+ years. However, ISD does implement requested maintenance changes and cost effective enhancements to support mission critical systems still on the mainframe.

 

3.      The ISD should delineate the costs for supporting mainframe architectures in the published Data Processing budget.

 

Response:  Santa Cruz County Board of Supervisors

 

This recommendation will not be implemented. ISD uses itemized mainframe support costs when developing the details that go into the published budget. There is no budgetary reason to provide these costs in the printed proposed budget, which are available for review if necessary.

 

4.      The ISD should use application requirements to drive the selection of new platform architectures.

 

Response:  Santa Cruz County Board of Supervisors

 

This recommendation has already been implemented. The current ISD policy is that no software solution, vendor, or development will proceed without a thorough application requirements definition process. The requirements and associated alternatives cost analysis will determine the appropriate solution recommendation. Both COTS and development alternatives will be considered.  

 

5.      The ISD should develop formal 18 month and 5 year plans that detail the elimination of the current mainframe hardware and software and the complete migration to current vendor supplied solutions that do not require in house software development.

 

Response:  Santa Cruz County Board of Supervisors

 

This recommendation has been implemented. A preliminary assessment of mainframe costs has been initiated with a planned completion date of December 2003.  Once the assessment of the remaining hardware and software is complete, the appropriate plan for application migration will be developed. The plan will tell ISD whether vendor software or development is the best solution.

 

Responses Required

 

Entity

Findings

Recommendations

Respond Within

Santa Cruz County Information Services Department

1-14

1-5

90 Days

(Sept. 30, 2003)

Board of Supervisors of the County of Santa Cruz

1-14

1-5

60 Days

(Sept. 2, 2003)

 

 

Note: Board of Supervisors of the County of Santa Cruz responded for Santa Cruz County Information Services Department.