Case Study: Standardization of automation practices improves operations at water utility
Industrial automation installations come in many shapes and sizes. Applications vary from standalone machines up to large processes, and the operating organization may have just one maintenance person or field an entire engineering group. But a common theme is the need for standardization to streamline design efforts, simplify support and reduce costs.
As these control projects are designed or upgraded, the standardization challenge spans many disciplines. Process and equipment control schemes, design and engineering practices, hardware and software product selections, and programming and configuration methods all play interrelated roles — and standardization efforts must thus integrate all of these and other areas.
There is no single best method for executing these tasks because of the number of variables for each area. Furthermore, many automation projects are implemented and upgraded over years or decades, adding to the difficulty.
Because end users may not always have access to large engineering departments able to churn out optimized standards and procedures, they must look at other options. This article shows how a water utility with a moderately sized staff was able to recognize the increasing difficulty and expense of supporting numerous operating sites of varying vintages and to then internally take action to implement standardization. Just a few practical steps resulted in many benefits for new and retrofit projects, as well as ongoing operations.
A water district controls its future
Helix Water District is a public water district in east San Diego County, California. Helix’s operations include a 106-million gallons-per-day water treatment plant, along with 25 pump stations supplying about 270,000 customers. Typical for these operations, the pump stations are geographically distributed over a wide area (see Figure 1).
A supervisor and four technicians support the existing supervisory control and data acquisition (SCADA) system. The technicians use their control system and electrical expertise to work on motors, instruments, controls and the networked SCADA system.
With some installed assets over 40 years old and showing their age, the Helix engineering department created a 10-year capital improvement plan that included pump station rebuilds and the replacement of several of the oldest motor control centers (MCCs). These projects were typically performed by engaging outside design and contracting entities to execute the detailed work.
Previous experiences had informed the SCADA group that even for straightforward MCC replacements, each design consultant might choose different equipment and ways of doing the work, even as they complied with the general specifications. This problem extended to the associated controls and theory of operation. These variations from site to site made it difficult for the SCADA group to maintain and troubleshoot the systems.
For these reasons, the SCADA group proactively teamed up with their internal engineering department and an external electrical engineering firm to ensure standardization of their upcoming projects.
Template for success
The goal of the teaming effort was to create an MCC design template that could be retrofitted at all current pump stations and implemented for any new stations. This template would include sufficient detail to ensure a highly standardized result for the design, hardware and software, but with enough flexibility to adapt to field variations.
Existing sites generally used MCCs with automation components installed within them for a compact installation minimizing field wiring. This concept was desirable moving forward for new installations. However, for retrofit locations, the new automation elements also had to be arranged so they could be installed into existing MCCs. Because pump stations come in various sizes, the MCC and automation components also had to accommodate different size motors, varying pump quantities and sometimes optional I/O signals.
After a detailed review, the team specified the most crucial automation items without allowing substitution. This included the core AutomationDirect programmable logic controllers (PLCs), input/output (I/O) modules, human-machine interfaces (HMIs) and associated components. While AutomationDirect offers other commodity-oriented components, substitutions were allowed for non-core parts.
The resulting standardized layout was a compact arrangement of terminal blocks, power distribution and PLC components that could fit in any typical MCC while still providing plenty of working space (see Figure 2).
Consistent components and designs enable Helix technicians to be instantly familiar with the automation at any pump station, making it easier to troubleshoot and maintain the control systems. They can stock fewer parts at the maintenance office and in their trucks, yet still be prepared to deal with any issues in the field.
Codifying code
Standardizing the hardware served to simplify physical design efforts for subsequent projects, both new and retrofit. However, the team wisely applied significant effort to software design as well. Specifying the AutomationDirect PLC and HMI families was just the first step. How those platforms were programmed would play a crucial role for implementing, deploying and maintaining these projects.
Software programming and configurations are intangible compared to physical designs, and if the code is not arranged logically and documented thoroughly, it can be downright inscrutable to all but the original programmer. Helix’s first task was to define the PLC logic to reliably handle the basic functional needs of a general pump station.
The team then went to great lengths architecting the logic so that a typical program could be used for different pump quantities, signal ranges and even network technologies simply by making minor adjustments. Memory locations and I/O address assignments were also standardized to promote consistency from site to site.
HMI graphics development received the same level of attention. Standardizing the graphical objects made the configuration effort more efficient and consistent. In turn, the resulting screens are easier for operations personnel to use at each of the pump stations.
With all of the software design pieces coming together, the next step was testing to validate the entire concept.
The big picture
For a water utility like Helix, there is one main operational site and many distributed remote sites. Significant wired and wireless networking infrastructure is in place to bring all of this together into an integrated whole under a single SCADA umbrella. Each of the remote locations must be capable of operating locally as an “island” of operation in case of communication failure, yet under normal circumstances, they must integrate seamlessly.
Local control at any given site was straightforward. The first level of complication was ensuring that PLCs could communicate peer-to-peer, which was needed if a pumping PLC required information about a downstream tank level handled by another PLC. Because this communication affected pump starting and stopping, it required the utmost reliability.
Another required attribute for the automation architecture was a centralized data collector PLC at the main plant for monitoring system values, handling start/stop setpoints, logging other events like communication failures, and alarming on faults. This data collector would also support more advanced systemwide functionality such as identifying time-of-use periods for shedding load during peak hours to reduce energy costs.
Knowing the necessary characteristics, Helix performed preliminary testing. They were able to individually prove out all basic communication, control and visualization functionality, with the assistance of available documentation, online videos and free support from AutomationDirect. This gave the team confidence that the pieces could be integrated into a whole as required.
This multitiered approach of local control, peer communications and centralized data collection was not deployed to the field in a single effort. Instead, careful benchtop testing was performed in advance to ensure the concept would perform as expected (see Figure 3).
By assembling multiple PLCs configured as local stations and networking them with a data collector PLC and each other, the team was able to simulate and pretest all levels of control functionality and communications to their satisfaction. HMI operation was also verified during the test phase. Extensive benchtop testing was key to ensuring success for initial and subsequent field installations.
Proof in deployment
Helix’s standardization efforts, using AutomationDirect hardware and software in particular, have been successful for the entire team for both new construction and retrofit situations. They have effectively developed a hardware and software template adaptable to their varied sites, and the team has now completed multiple deployments.
A completely new pump station was built by a subcontractor following the template, and three existing pump stations were retrofitted with new MCCs and PLC controls by subcontractors using the same approach. Helix also self-performed the retrofit of an existing pump station with new PLC controls. This proves not only the technical suitability of the standardization work, but also how the work can be executed in multiple ways from a commercial standpoint.
The new installations have worked reliably with no significant functional or component faults, but the team knows that eventually there will be some sort of problem requiring troubleshooting.
An added benefit of the new designs are the many options for remote monitoring and notification. The team has built in provisions so local or system failures can be alerted through the SCADA system. During the course of normal operation, or when trouble is indicated, team members can remotely connect to the PLCs and HMIs to inspect the situation, troubleshoot as necessary and initiate a course of action. This provides substantial time and cost savings compared to older pump stations where a site visit by a technician was required.
Standard automation practices are attainable
Most industrial automation end users understand the benefits of incorporating standardization, but they may be concerned as to whether they can implement it based on their staffing, organization and the types of automated processes involved. The Helix Water District team efforts to proactively upgrade, standardize and improve their automation systems are a case study in how to approach the task methodically from the bottom up, starting small and adding value at every level.
There are many options along the path to standardization. End users can do as much of the work as they like or can retain the assistance of contractors. Because each company's end user knows their operations and practices the best, it is recommended for them to play a strong role in specifying the most crucial control platforms, components and architecture aspects.
Another key is pretesting basic automation elements and then performing integrated testing on the bench to ensure functionality and buy-in of all stakeholders. A staged deployment can then be undertaken with confidence. Even once the system goes to the field, there is still an opportunity to learn and improve the defined standards.
All end users will benefit from incorporating standardization, following practical and common-sense steps to define and achieve their goals.
David Reagan, along with Henry Palechek, Bobby Fortuno, Gary Richardson, Kyle James and Jeremy Boone, make up the SCADA team at Helix Water District, a group of operations, maintenance and engineering personnel. Each were involved in the standardization efforts and continue to operate and support the system on a daily basis.
