By: Andy Huckridge

Big Data has been a major business growth driver in the IT community and doesn’t seem to be slowing down; in fact, things seem to be speeding up. 

As the service providers increase the amount of data across their networks, analytics tools are scrambling to keep up with increasing information complexity and generation. This is nothing new; when the tools industry moved from 1Gb to 10Gb line rate speeds, their throughputs were usually quoted between 5-7Gb per 10Gb tool due to computational limitations of appliance-based architectures. Unfortunately, the tools still cost up to 10x more than 1Gb tools and, as a result, the speed upgrade was hardly cost-effective. Another downfall was that the load balancers “front-ending” these tools were not sophisticated enough to identify all the packets associated with a single communication flow. This led to different flows going through completely different tools, resulting in incomplete and imprecise information. 

Help from middleware solutions

Middleware solutions are now necessary to intelligently aggregate all the information and ensure packets are not dropped. Not only can they aggregate all the information at increased speeds, these solutions can also load balance across connected tools and groom the data to allow for an effective increase in throughput processing.

These middleware solutions remove irrelevant traffic and de-duplicate repeated packets to optimize tool performance. This connects back to the bottom line for the service provider, which can then continue to use its existing tools in a Big Data environment - preserving CAPEX and deferring new investment in to more tools. This way the operator can deploy tools and redeploy staff to where they are most needed or most effective.

Analytics have become more and more important in a data heavy world. Mobile carriers are increasingly evolving to 4G, LTE, VoLTE and LTE-Advanced speeds. Meanwhile, new trends like SDN will very likely speed up the explosion in bandwidth requirements. Cisco forecasts 15.9 extrabytes per month of mobile data traffic by 2018: it’s now time for the tool vendors to keep up the pace.

Currently, tools can typically process just 5-7Gb of the 10Gb of traffic offered. How will they be able to keep up with the next level of pipe bandwidths, 100Gb and beyond? The industry’s history shows that there is typically no economy of scale when increasing tool strength, so the industry can expect the cost of tools to rise significantly, to potentially unaffordable levels. 

The future of tools

The tool vendor of the future will need to solve the 100Gb, and potentially even 400Gb, throughput problem by embracing the innovation of traffic visibility solutions. Additionally, the next generation tool vendor must leverage elastic compute architectures for both storage and computers found in existing cloud processing frameworks. The vendor will concentrate on an analytics, diagnostics or derive business insights from their traffic visibility solution.

There are currently a large number of different cloud solutions available from well-known vendors. Each offers an elastic compute platform suitable for massive amounts of analytic data processing. Virtually unlimited storage on demand is also available and frameworks, such as OpenStack, could be used to help transition from appliance-based models to cloud-based architecture.

An industry standard of interfacing tools with a traffic visibility solutions or Network Packet Broker layers is needed. The capability of the tool could be calculated by an Application Programming Interface (API).  This would calculate the terms of processing throughput, types of data, quality of analytic capability, and flow control, using a system that acts as an Analytic Data Arbitration Function (ADAF).

ADAF

An ADAF capability is in many ways similar to a portal and would be largely automated. Service providers would simply identify the amount of data to be analyzed and the price they are willing to pay. Tool vendors would state the amount, characteristics, and price of the data that can be processed. When a match is made, solutions could send the appropriate traffic to the selected tool while providing verification of analytic data records to the ADAF for later arbitration and any later processing or back-end processing such as billing or settling.