The storage segment of the IT industry used to be a tad, well, boring, but big data and cloud computing are spurring innovation. Lance Harris investigates. 

Storage is once again taking centre stage in enterprise technology strategies. The demand for new ways of storing and managing information comes from the growing data volumes generated in a world where there are billions of connected devices. Organisations are drowning in data such as machine data from sensors in plants and factories, geolocation data, social media posts, climate information, financial transactions, voice logs, surveillance video, and more.

The rise of cloud computing and an explosion in big data are helping to drive demand for storage software and hardware as well as spur innovation in this market segment. Market researcher IDC says that the market for big data solutions will be worth $16.1 billion in 2014, growing six times faster than the overall IT market. Infrastructure – including servers and storage – will account for 45% of this market. Against this backdrop, the industry is seeing a range of new technologies emerge, such as in-memory computing, big data appliances, and flash arrays, to help organisations manage the demands of storing and managing reams of data created at a rapid speed. 

The cloud is also changing the landscape by offering end-user companies new ways of buying storage capacity and by creating new storage needs for service providers that offer cloud services and enterprises that run hybrid and private cloud architectures. Increasingly, legacy IT architectures are unable to keep up the capacity and flexibility this new world demands. 

New solutions on the table

Big data is not a challenge merely because of the amount of data, but also because of the need to provide information to the business in real time, says Wayne Dick, business development manager, Hitachi Data Systems. “From an infrastructure perspective, big data demands more throughput than you get from most solutions today,” he adds. Business users don’t want to wait a week to get analytics data, he adds. There are a range of solutions on the market that make it easier to get data to end users without needing to go through a range of complex IT processes. But the challenge is that many of these solutions – flash arrays and in-memory computing, for example – are expensive to implement. “You could sit with all of your information on a costly infrastructure,” says Dick. 

Flash arrays use flash memory drives instead of hard disk drives to store data, with the benefit that data can be accessed and transferred at a faster speed. Even through costs per gigabyte have fallen dramatically, flash remains an expensive option for mainstream storage uses. But some organisations are finding it beneficial for some applications where performance and rapid access to data are important.  

“In-memory and flash array storage is already appearing in data appliances, fast replacing disk spindle storage,” says Mervyn Mooi, director at Knowledge Integration Dynamics. Flash storage is a key component of in-memory computing solutions, which store data in the main RAM of dedicated servers rather than in databases on disk drives. In-memory computing allows companies to analyse huge data volumes on the fly. Though some local retailers are looking at in-memory computing, its complexity means that adoption is likely to be relatively slow, says Dick. “As you move from traditional database structures to technologies like SAP’s HANA, you need different skills,” he says. “You won’t see people jumping into this quickly.”

One way that businesses can manage the challenges of providing real-time access to business data while keeping costs down is to use tiered storage solutions that allow them to assign their most valuable, most often-accessed data to higher cost disc or flash solutions while storing archival data on cheap media like tape. Some vendors provide software that automates the process of moving data between different tiers. 

Converged infrastructure and cloud

There is also a move towards converged infrastructure as companies look to simplify their IT infrastructure, says Dick. Also known as engineered systems, converged infrastructure solutions draw together servers, storage, software, networking and other data centre elements into prepackaged pretested solutions designed to meet a particular business requirement. The cloud is emerging as another option. Many large companies in South Africa want to keep their data under their own control and continue to invest in the infrastructure to store and manage their information in-house, says Mooi. But small businesses are increasingly looking to the cloud for their computing and data storage needs. Storage-as-a-service has already surpassed enterprise storage investments and volumes, says Mooi. Social big data generates more data volume than ERP and other business systems, which means that many data centres will want to consume storage as a cloud service in the future.

But the cloud still has some challenges in the South African market, according to Dick. “Cloud adoption is being driven by the fact that it offers businesses more flexibility and lower costs,” he adds. “But here, bandwidth is still costly and not as readily available as in the rest of the world. There is also still a lot of caution among companies about allowing their data to leave our borders.” Many companies still rely on proprietary and legacy applications they cannot easily move to the cloud, says Dick. Where the cloud is currently most useful is for SMEs who use services such as Amazon and Azure for data protection, he believes.

The 4 Vs of Big Data 

Big data isn’t just about the amount of information that companies need to store and manage. Vendors and analysts define a number of characteristics that make big data so challenging for enterprises:

Volume:  According to IBM, we create 2.5 quintillion bytes of data each, which means that some 90% of the data in the world today has been created in just the last two years alone. The result is that companies need to store massive quantities of data from a range of sources.

Velocity: This refers to the sheer speed at which businesses and consumers create new data—consider, for example, how much text and media people post to social networking sites every second of the day, or how many financial transactions a stock market crunches through in a minute.

Variety: This is about the different types of data we need to manage in today’s world—not just data in traditional structured databases, but also unstructured data like photos, videos, machine sensor data, emails, social media posts and more. 

Value: Big data is valuable because it allows companies to make better decisions at a faster pace. Not only is it invaluable for business analytics, data is also starting to become a tradable commodity in some industries.