Integrating Operational Data With Splunk and IBM Common Data Provider
This article explores an applied use of IBM Common Data Provider by a large European bank to use IBM Z operational data via Splunk.
Image by Tatiana Plakova
By Domenico D'Alterio09/01/2018
Today’s systems generate a massive amount of operational data that needs to be distributed and analyzed by a variety of technicians and stakeholders. To make this data available to a variety of platforms, it’s important to deliver it in a non-proprietary format to enable its consumption by a variety of people and tools. IBM Common Data Provider for Z is a means of accessing common operational data and conveying it to a variety of destinations.
This article explores an applied use of IBM Common Data Provider by a large European bank to use IBM Z* operational data via Splunk. Making this data available to a variety of stakeholders improves communication, outcomes and service for customers.
IT Operations Analytics
IT Operations Analytics (ITOA) can reduce root cause analysis elapsed time, improve the efficiency of overall IT and prevent outages. Over time, more and more IT teams have recognized these benefits. With the expansion of hybrid cloud implementations, the quantity of data generated by systems supporting business-critical applications as well as the variety of data coming from different platforms is growing.
Initial exploitations of ITOA were limited to the analysis of data coming from distributed platforms, with the possible addition of network logs and performance data. However, many applications (i.e., the most business-critical ones), behind a user-friendly web interface or mobile app rely on mainframe backend applications and systems like CICS*, IMS* and Db2* for z/OS*.
Standardizing the Data Gathering Approach
The ability to collect necessary data to prevent IT problems that could directly affect an organization is crucial. More and more frequently, data gathered from different systems or a subset of this data is requested from different teams within an enterprise for various reasons (e.g., problem determination, monitoring, security control, etc.). However, it’s wise to avoid having multiple tools for collecting different data from different sources; this can reduce operational costs while standardizing the data gathering approach.
That’s where IBM Common Data Provider comes in. IBM Common Data Provider provides a single collection point for accessing near real-time operational data like SMF records, RMF data and a wide variety of log data on Z. It can collect Z IT operational data from more than 140 data sources, including over 100 SMF record types (e.g., ISV SMF record for CA Top Secret and ACF2) as well as application logs (e.g., WebSphere*, CICS and IMS) or generic files. It also allows access of analytics data within minutes, collecting the requested data while providing multiple end users and analytics platforms (e.g., Splunk, Elastic Stack, IBM solutions, etc.). Moreover, the end user can easily add definitions of new data types to extend product capabilities. This unique capability allows users to quickly add data types based on current needs.
The web UI provided with the product as a z/OS Management Facility plugin also allows users to easily define how collected data should be configured, which targets should be fed or how a transformation should be applied.
Controlling Business Application Health
One large European bank chose Splunk as its strategic analytic platform. Until recently, z/OS had no means of feeding Z logging data into Splunk. By using IBM Common Data Provider, the Z IT team can now feed Splunk with Z logging data. In addition, real-time CICS transaction logging data is presented with Splunk dashboards and Z job performance data is easily accessible.
As a result, the bank was able to create dashboards to keep business application health under control while providing information on business health and metrics with varying levels of detail. A business-level overview with details on business application health and metrics was at the top of the dashboard. The lower part of the dashboard provided health information and metrics about specific supporting applications, with the possibility to drill-down in the specific application metrics. This allowed the IT team to intercept and resolve problems at a supporting application level before these problems affected business applications. For a visual representation of this dashboard, see Figure 1.
Solving IT Challenges With Data Filtering
An IT team’s ultimate goal is to prevent or solve IT problems before they affect the organization, but the team will face multiple challenges along the way. Some of the team’s main challenges might be:
- Collecting all necessary data across platforms. This should happen in real time, as soon as the data is available.
- Having all data available in an analytics platform that allows for easy data analysis. In most cases, this platform has already been selected at an enterprise level.
- Feeding necessary data to the analytics platform or the data consumer without overloading either with unnecessary data. This limits associated costs and controls who can access the data.
- Selecting what data is strictly needed for the identified use case. The demand for Z data comes from different teams within an enterprise, most of the time without a real understanding of what data is actually needed. Focusing on the organization’s objective of these requests is a must for any IT team.
- Cost control. Sending more data than necessary means additional costs in terms of network bandwidth, resource utilization and ingestion. Some analytics platforms are priced based on the quantity of the data ingested.
- Data access control. Some operational data might contain sensitive information that isn’t needed for the analysis being performed by the consumer of the data. The ability to select which fields of the collected data should be sent is very important for security reasons as well.
IBM Common Data Provider offers the capability to filter the SMF record based on the content of the record. The user can define rules to specify which records should be streamed based on the value of specific fields within the record itself. To reduce the amount of data ingested in the analytics platform, the user can even define which fields out of a collected SMF record should be sent to the analytic platform.
Another filtering option offered by IBM Common Data Provider is based on a regular expression defined by the user, who can decide if the log records that match the regular expression defined should be sent or discarded. For a visual of these filtering capabilities, see Figure 2.
These filtering capabilities were also leveraged by the aforementioned large European bank to substantially reduce the volume of data related to CICS transaction SMF records ingested in Splunk. More specifically, only a subset of the SMF record’s 110 fields were ingested in Splunk. That is, only the ones bringing value were selected, creating significant cost savings.
Creating Business Benefits
Ultimately, IBM Common Data Provider for Z works toward solving IT challenges while creating significant business benefits.
The IT team leader of the European bank reiterated this as well. “IBM Common Data Provider for Z allows us to stream (in real time) IBM Z data to Splunk. By bringing IBM Z data in Splunk along with data from the distributed environments, we can now ingest, correlate and visualize data from across the enterprise to gain valuable insights into infrastructural, application, and end-to-end business functions. Through this we are able to immediately act on these analytic insights, which helps us guarantee continuity for our customers,” he says.
Learn more about IBM Common Data Provider for Z online: ibm.co/2yrnh8F.
Domenico D’Alterio is the IBM offering manager focused on IBM Z Analytics products.
Sponsored ContentAchieve Compliance Without Impacting Productivity
Post a Comment
Note: Comments are moderated and will not appear until approvedcomments powered by Disqus