Enhancing Cyber Security with Analytics
Cyberspace is today’s new battleground and cyber security continues to be a top imperative for both enterprises and governments. Recently, the U.S. Pentagon announced plans to boost their cyber security team amid a string of attacks, including one that wiped out more than 30,000 computers at a Saudi Arabian state oil company. Earlier this year, hackers from China infiltrated the computer systems of the Wall Street Journal, in the second reported attack on a major US news outlet. The New York Times also reported that Chinese hackers have "persistently" penetrated its systems for the last four months.
Foreign governments, criminals and terrorists’ computer-based intrusions against public and private infrastructure are increasing by orders of magnitude with stolen intellectual property exceeding reaching trillions of dollars.
Finding the cyber-indicators of an impending attack is the proverbial "needle in the haystack" and it has never been more important. Governments and law enforcement agencies are particularly vulnerable due to the rise of international state sponsored cyberwarfare and terrorism. The underground community of hackers and cyber terrorists is vast, well funded and supported by very sophisticated engineers and scientists.
This issue is so critical to the government that the National Institute of Standards and Technology issued a request for information in the Federal Register as the first step in developing a cyber security framework as laid out in President Obama’s cyber security executive order. This framework will create a set of voluntary standards and best practices to guide industry in reducing cyber risks to the networks and computers that support critical infrastructure vital to the nation’s economy, security and daily life.
Cyber security has never been harder as network traffic increases and record counts in the trillions amass over a short period of time. To meet this challenge, cyber security needs to advance technologically to the point where systems are proactive versus reactive. This requires a paradigm shift from rules, signatures and firewalls to automatic threat classification. When attackers can be identified in real-time through machine learning, affected systems can be locked down without consequence and breaches stopped long before they can do harm or interrupt service.
This is where Big Data comes in. The challenges facing cyber security have driven new approaches to analyze cyber data through macro analytics across trillions of records accumulated and stored over months and years. Cyber security platforms use Big Data capabilities as a central part of the solution.
Big Data technology helps keep pace with advanced threats and prevent attacks before they happen. It helps uncover hidden relationships within massive amounts of security data, using proven analytics to reduce billions of security events to a manageable set of prioritized incidents.
The Figure (on the right) shows how Big Data and analytics can improve on traditional cyber security and operations technology
Big Data technology can complement the cyber security solutions in several ways:
- The Hadoop engine can be used to perform custom analytics on transactions and to baseline petabytes of account activity over months and years – sending insights back to the cyber security platform to detect fraudulent activity as it occurs. The platform performs real-time correlation and reporting for rapid threat and risk response and then send enriched security information to Hadoop for additional analysis.
- Hadoop can consume and analyze immense amounts of data from unstructured and semi-structured sources, accommodating both the variety and volume of data needed for advanced security use cases. Hadoop can help improve the accuracy of analysis over time and feed insights back to cyber security platform, providing a facility for closed-loop, continuous learning.
- When the situation requires detection in milliseconds, rather than seconds, streaming technology can be used to perform complex analytics on massive volumes of data in motion, such as text, images, audio, voice, voice over IP, video, web traffic and email content at rates up to petabytes per day.
Data Warehouse Appliances:
- Data warehouse appliances, which combine and optimize hardware and software can simplify and optimize performance of data services for analytic applications, which can improve the very complex algorithms involved with cyber security.
I encourage you to read this white paper “Extending Security Intelligence with Big Data Solutions”, which provides a good overview of the role of IBM Big Data analytics and cyber security.