MAINFRAME > Case Studies > Banking/Finance

DTCC Implements Tools to Ensure its Networking Infrastructure is Ready for Anything

Given the high number of stock transactions DTCC handles, it needs a robust IT infrastructure. Pictured here, from left, are engineers Sigfrido Perdomo, Richard Warren and Anthony Smith at the DTCC-Dallas location. Photography by Aggie Brooks

Entry Points

Despite these being rare occurrences, DTCC decided to be more proactive about managing its networking infrastructure, as well as remediating some of the issues it had already experienced. To that end, it sought a solution provider with all of the tools it needed to control its ever-growing network and found Inside Products.

Inside the Stack (ITS), Inside Products’ flagship product, was essential to helping DTCC tackle some of its more pressing issues. For instance, DTCC has used the ITS business analytics panel to determine which of its clients were having networking problems. Notably, this isn’t done using just IP addresses, but also the company name and departments within those companies associated with that address.

Worldwide, DTCC handled 3.7 million securities issues in 2011 to the tune of $39.5 trillion.

“We have clients coming into our network from a variety of different sources,” remarks Anthony Smith, a DTCC lead network engineer. “This can create some confusion, but if we can associate a company with a provider, we can more easily determine where, for example, a bottleneck might be. We can even dive down into specific company locations, such as location A here, and location B there. This makes it much easier to pinpoint exactly where the entry point into our infrastructure begins and how we might be able to address any issues related to that entry point.”

If customers are experiencing slow response times, ITS enhances DTCC’s capability to diagnose and trace data packets more quickly, as they travel across the network. In one such case—which previously might have taken several hours to debug—using ITS, only took 1 1/2 hours. According to Warren, this is now fairly typical, depending on how large the trace is.

“We used ITS to determine what was causing the delays,” he says. “As it turned out in this example, it was on the mainframe side, with a trace that was running that shouldn’t have been. Once the trace was disabled, the client’s response time normalized to agreed service levels.”

Much of this type of diagnostic work is based on historical networking data stored in Inside Products’ DB2* tables. By combining ITS with that data, DTCC can look at customer networking patterns and narrow down a slowdown’s root cause. Referring back to the customer experiencing networking delays, Smith says, “We were able to grab the representational flow of data to see where it was bogging down—how it normally looked before a transmission, how it looked before the problem occurred and how it looked afterward. Based on that, we were able to figure out which protocol was being adversely affected.”


Additional Tools

DTCC also uses a specific internal code, known as the Connectivity Tester, developed by Perdomo and Warren. It’s a TCP protocol tester and application simulator that confirms connectivity from mainframe to the application listener at the server/client end. When new clients come on board, DTCC uses it to determine if their remote applications are up to snuff.

“We can check their status before we go live,” Perdomo says. “Are things good or bad? If they’re bad, where’s the problem? Based on that, we know right away if a client’s having network issues and can take care of it.” To accommodate DTCC’s requirements, Inside Products built the Application Checker. It allows users to periodically push socket calls to external applications, such as DB2, to proactively identify potential problems and remedy them before they become critical. Similar to Inside Products’ Availability Checker, it confirms the availability of routes and devices from the mainframe by periodically pinging these resources and producing status reports.

DTCC settled nearly $1.7 quadrillion in securities transactions in 2011 alone.

To further promote proactivity, Inside Products has an Early Warning System. Once users, such as DTCC, understand the pattern of a client’s network behavior, they can set thresholds relative to those patterns. If something is off, they’ll be able to address it before the client even notices a problem.

DTCC’s latest effort involves having its network operations staff look at networking problems from a business perspective. One aspect of the solution is the association of the network devices and infrastructure to specific business partners. “This is one of those things that is easy to say but a nontrivial task to perform,” Warren says. “The needed information was in many places, such as descriptive verbiage in routers, switches, firewalls and systems documentation.”

Warren and Perdomo developed the code and procedures to facilitate and maintain what they call “the business entity table” (BET). As part of this, DTCC and Inside Products collaborated on a business entity panel that allows network operations to see which clients have a network problem. The screen shows IP addresses and the business entity, and provides trace-route and application-checking capabilities. Using this, the organization can test the network and application problems for a business partner from one screen.

Senior management is using BET in monthly utilization reports, which are created by DTCC’s Network Software and Planning Team using Inside Products’ DB2 tables. It’s also being reviewed by other internal disciplines, such as DTCC’s Capacity Planning Group and Enterprise Management Group, to incorporate it into their management reports.


Proactive approach

With data moving at nearly the speed of electrons, it’s increasingly important for organizations to ensure both the network and associated applications are available and running as expected. As DTCC has discovered, it’s best to do so proactively.

“We choose the best products to meet DTCC’s network needs regarding functionality, maintenance, cost and scalability,” remarks Neil Wasserman, vice president of Core Network Services at DTCC. “After evaluating several alternatives, we decided Inside Products should be one of the principal elements of our mainframe network-monitoring environment.”

Jim Utsler, IBM Systems Magazine senior writer, has been covering the technology field for more than a decade. Jim can be reached at

comments powered by Disqus



2019 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.

A Sound Investment

Citigroup tames its backup environment with dedicated mainframes


Bankwest Modernizes Its Mainframe Integration Environment

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
Mainframe News Sign Up Today! Past News Letters