Skip to main content

Avoiding the Data Deluge

Collaboration, communication and cooperation are crucial for developing a successful data management strategy.

Three "C's" against three gradients of purple.

Data management is one of the foundational elements of the entire network ecosystem. It enables organizations to utilize data as a powerful real-time analysis tool to track performance and reliability, glean insights for informed decision making, recognize usage patterns and predict future trends. 

End users and consultants concur that businesses need a cogent, comprehensive data management plan that combines best practices and data management solutions and tools to make the best use of their data, ensure performance and reliability, and adhere to governance, security and privacy regulations. 

“Every data management plan starts with having a current mapping of existing data inventory,” says Sergio Puga, a vice president of Global Information Security at Bank of America in Dallas.

175 ZB of Data by 2025

This is imperative because corporations are adding data at a record pace. An overwhelming amount of it is unstructured (e.g., emails, text messages, videos and audio files, which are larger than traditional data sets). This contributes to the glut of information that businesses generate, process, transmit and store on a daily basis. 

The amount of data created worldwide will increase from 33 ZB generated in 2018 to a projected 175 ZB by 2025, according to research firm International Data Corp.’s (IDC) 2018 study “Data Age 2025: The Digitization of the World from Edge to Core” (

To put this in perspective, 175 ZB is the equivalent of 3 million times the total collection of books, photographs and other media in the U.S. Library of Congress would occupy if they were fully digitized1.

Managing Myriad Types of Data

Organizations also retain and store data for longer periods of time. The explosion of data exerts pressure on businesses and IT departments to proactively manage all of the myriad types of data they generate and store. 

“Organizations have two choices: take charge of managing their data or have the data manage them,” says Andrew Baker, CEO and founder of Brainwave Consulting, an IT and security services organization in Gassaway, West Virginia. 

“The sheer explosion of data generated and stored daily, monthly and annually is staggering,” Baker observes. “Corporate IT departments need to utilize advanced data management solutions as ‘forensic tools’ and work closely with their vendors, cloud providers and service providers.” 

Bank of America’s Puga agrees, noting that every good data management plan should allow businesses to identify “hot data” and “dirty data.” 

Hot data is accessed daily because it’s critical to corporations’ mission critical business operations. Companies use hot data for fast and accurate decision-making purposes such as making a sales proposal or a when a retailer puts together a new sales price list. 

“Hot data should reside on the fastest storage—usually flash in hybrid or tiered storage environments—for efficient, accelerated retrieval,” Baker says.

Dirty data may be old, outdated or incomplete. It may also contain grammar and spelling errors. In a worst-case scenario, the data may have been changed or manipulated as the result of a targeted hack or data breach. Dirty data heightens the risk of both regulatory noncompliance and litigation. Corporations should use the appropriate data cleansing tool. 

Data Management Strategies 101

The overall success of an organization’s data management’s strategy depends upon strict adherence to these seven best practices:

• Data backup and recovery: This is the most basic data management rule of all. Organizations should regularly back up data on local and remote external servers. This will enable firms to quickly and efficiently restore access in the event of an outage or security breach. All data management stakeholders (e.g., IT administrators and department heads) should construct a contingency plan to restore lost, stolen, damaged, destroyed or changed data. Export/upload data to a trusted cloud service and provider. The company should collaborate closely with external cloud service providers to determine the specific estimated response/restoration time for their SLAs in the event the cloud provider experiences an outage. Insist on an up-to-date list of contacts at the cloud provider. Also ensure that the appropriate security permissions are in place in case you were to get locked out by a targeted breach or ransomware attack. 

• Interoperability and data storage: Given the exponential growth in data retention, it’s crucial for companies to store data in open standard rather than proprietary formats. This will ensure that data is easily retrieved and not lost due to incompatible file formats or interoperability issues. When archiving a file, for example, using a .zip format is advisable; for long-term storage needs, use the cloud. Legacy tape and CD-ROMs may be used as a backup source. 

• Data access/sharing: This should be obvious, but once again, it cannot be overstated. Data access should be granted and shared on a need-to-know basis only. 

• Privacy/ethical concerns: As more data traverses the internet via emails, text messages and via the Internet of Things ecosystem, the issue of ownership is no longer black and white. Organizations and the internal legal teams should stay up to date on the latest intellectual property rights involving all corporate data. To safeguard the company, its IP and employee privacy, encrypt and/or redact any sensitive information prior to sharing it on public forums, archives or repositories.

• Data documentation: Use metadata and a robust metadata tool. Metadata is data that provides a detailed description of the types of data the firm creates and stores. Keep detailed records. The use of metadata is crucial for managing and leveraging information in a digital format because the information is only discoverable through the use of intermediary hardware and software tools.

• Data retention: C-suite executives, IT and security administrators, and the appropriate department managers should construct a cogent, comprehensive data retention policy and create a governance place for backup and restore procedures

• Track ongoing data usage levels: Tracking data usage levels, content and types of data the firm generates and retains annually is crucial for budgeting, planning and upgrading hardware and storage solutions. Whenever possible, organizations should anticipate data generation and retention needs for two to three years in advance. 

A Hybrid Data Management Strategy

Corporations’ data management strategy and solution should align with the technology and business goals. Organizations increasingly generate myriad types of data on a daily basis. The data may be stored on-premises and in public, private or hybrid cloud data centers. Businesses also use open-source, standards-based and proprietary hardware, software and a variety of connectivity solutions. 

There’s no one-size-fits-all approach to data management. Businesses should adopt a flexible and hybrid approach to address the company’s diverse needs.

“Perform due diligence and deploy solutions that incorporate advanced, embedded AI features that can translate and perform queries across multiple vendors, languages, locations and structures,” Baker says. 

AI solutions take data management to the next level. “They go beyond baseline analytics and enable businesses to perform predictive and prescriptive data analysis to be more competitive, cut costs and drive revenue,” Baker adds.

Advanced AI data management solutions also provide IT and security administrators with deeper insights into server and application performance and troubleshooting alerts to avoid costly outages and service interruptions. 

Security and Privacy Are Crucial

Security and data privacy concerns are two issues that impact all businesses. Enterprises are grappling with a sharp increase in regulatory, privacy and legal

requirements, such as the European Union’s (EU) General Data Protection Regulation, which took effect in May 2018 and controls the privacy and processing of personal data for all EU citizens and the transfer of their data outside of the EU. 

Businesses that don’t regularly track privacy and security leave the company vulnerable to attack and also place firms at higher risk of legal action and penalties in the event that the privacy and intellectual property of its employees, customers, business partners and suppliers is exposed.

Cross collaboration, cooperation and communication among various internal IT departments and data management teams and their counterparts at the company’s cloud providers are also necessary elements in managing and securing data. Overcoming the natural inclination to cling to “one’s turf” is challenging but necessary to leverage data to the best advantage and identify weaknesses, potential security threats, governance issues and potential technical flaws in advance, notes Puga. 

“If you want to plan for privacy, you first need to identify and prioritize what data is the most important and focus your efforts on this data,” he adds.

The most important aspect for maintaining data privacy is limiting access and permissions. “Oftentimes, employees have too much access based on the assumption they need it to perform their job,” Puga says. “However, it’s the company’s responsibility that only the right people have the right access to the right data.”

Governance should also be a component of every enterprise’s data management strategy because it can prevent uncontrollable data sprawl. “It doesn’t get any worse than when an organization has terabytes of data that can’t be utilized because there was no consistent way to capture it,” Puga notes. “Disorganized data as a result of poor governance causes many lost opportunities. It’s even worse if no controls are in place to guarantee the data is trustworthy.”

As with security planning, data governance must start with a current inventory of what’s currently being captured, where it’s stored and how it’s being used. Companies can then establish a governance plan to control who should have access to this data, who should be able to change it and how it needs to be audited. 

These efforts shouldn’t be relegated to the IT department. “It must involve all stakeholders—the business and technology teams to make sure data is properly captured, stored and protected,” Puga says. “The onus is on the business representatives to establish the importance of structured and unstructured data to their business operations. They must be vocal advocates on what data is most important to them and their business.” 

“If you want to plan for privacy, you first need to identify and prioritize what data is the most important and focus your efforts on this data.”
–Sergio Puga, vice president, Global Information Security, Bank of America

Overcoming ‘Human Data Silos’

The most effective way to manage and secure all corporate data assets—whether on-premises, in the cloud or at the network edge is to trust but verify.

“If the data is on premises, you can always leverage your internal information security teams to establish controls for data protection. This can be reinforced by conducting periodic reviews and audits of controls,” Puga says. 

When it comes to cloud-based networks, corporations must rely on negotiating contracts that will allow audits and reviews of cloud providers on a periodic basis. In these instances, strong documentation of what company data will reside on the cloud and how it will be protected must be backed by strong contracts and service-level agreements that hold responsible parties accountable. 

Finally, Puga says, the three Cs—collaboration, communication and cooperation—are the key to a successful data management strategy. Businesses must make a conscious effort to overcome the “human data silos” within the organization. 

1. The Measure of Things. Bluebulb Projects’ The Measure of Things. URL:

IBM Systems Webinar Icon

View upcoming and on-demand (IBM Z, IBM i, AIX, Power Systems) webinars.
Register now →