Why Flash is the Fastest Growing Form of Data Storage
Flash memory is used every day in thumb drives and hand-held smart devices. But it’s no longer relegated to consumer electronics. According to a June 2016 survey from IT consultancy 451 Research, almost 90 percent of organizations have some form of flash-based storage installed in their data centers. All-flash storage systems are also becoming increasingly standard for transactional applications.
The findings don’t surprise Andy Walls, CTO and chief architect, Flash Systems, IBM. “In more than 35 years at IBM, I’ve seen storage change in amazing ways,” he says. When Walls joined IBM in 1981, HDDs were the gold standard of data storage. Since then, he notes, they’ve had hyperaccelerated growth. “In many ways, HDDs are one of the most important inventions in all of human history. The improvement—the bits per inch, the scaling—has been absolutely mindboggling.”
While HDD technology is durable—in the past 30 years, scientists and engineers seemed to circumvent physics to continue scaling HDDs—Walls says its history of remarkable innovation is slowing.
“A disk drive is a mechanical device,” he explains. “Spin media and magnetic heads can read the bits off of that media. Physical restraints limit how fast it can spin, so you really can’t get an average response time of less than a few milliseconds, even with all of the tricks we’ve learned.”
Today, more organizations find that flash storage systems provide better performance and efficiency in data centers than HDDs do.
In 2008, when flash (formally known as NAND flash memory) started to get into its hyperaccelerated phase, it became the core storage technology of consumer devices. “Usage increased to the point where a lot of both competitors and supply existed,” Walls says. “Suppliers were working on reducing the cost.”
Flash grew so inexpensive that developers realized it could be used in data centers. Because the cost wasn’t as low as HDD storage, it was used primarily for niche or fringe applications that were latency-sensitive and could afford the premium. One early use was automatic tiering. “We could put flash and HDDs together in the same storage enclosure and determine which data went where,” Walls says. “That way, your average response time and I/O operations per second improved, yet the cost didn’t increase much.” Because the total cost of ownership is less for flash than it is for HDDs, IBM has seen wholesale adoption of flash storage for many clients.
Meanwhile, consumer demand continued to drive flash production. Manufacturers kept making flash devices smaller and cheaper. This drove more usage in data center applications. The performance aspects of flash helped drive the adoption rate. But that performance also allowed for data reduction techniques that further reduced the effective cost of flash-based storage. These techniques are particularly slow with HDDs because additional metadata may also be read from slow HDDs. Flash, however, can absorb the additional metadata fetches and still provide excellent performance.
One of the main uses of flash storage is what Walls calls a hyperconverged network, where an organization can put storage in servers that also contain storage. Then, software such as key-value store or Apache Cassandra is used to scale out both the compute and storage. “This new phenomenon has been accelerated by flash,” Walls says. “It’s become cheap enough and has lower cooling requirements than HDDs. IBM Power Systems* servers play into this well because a rich assortment of in-server flash products can be put into servers and scaled in a similar way.”
The second main use for flash-based storage in the enterprise is the external storage market. Organizations can now buy all-flash arrays that have similar total cost of ownership to HDD arrays. An example is the IBM FlashSystem* A9000, which incorporates compression and deduplication. “These all-flash arrays can provide extremely good performance—better than what you’d get with an HDD system, with data reduction that can be as high as 5x, depending on the type of data,” Walls says. This kind of data reduction allows clients to move workloads from HDDs to flash.