It will not only monitor points, but literally everything flowing to and through the network from end to end. This data will be streamed in real-time, rather than polled every five minutes. The consequent richness of this data will allow it to not just be reported, but also analyzed in detail to provide actionable data that allows operators to reach the ultimate goal: integration with software-defined networks alongside real-time analytics that provide the intelligence they need to make them truly dynamic.

All of these possibilities sound fine in theory, but what is an operator really supposed to accomplish with all of this data and why do they need to seriously consider that? In addition to the needs typical of hyper-scale cloud application providers, there are other needs to consider, including solving network outages, instilling proactive customer satisfaction, analyzing encrypted traffic and improving security from DDoS attacks.

The Show Must Go On

Of the 60 percent of internet traffic currently represented by streaming cloud apps and services, binge-watching video content accounts for the majority, and Nokia Bell Labs predicts that it will rise to 80 percent by 2020. Meanwhile, Internet Service Providers (ISPs) have very little visibility into these flows, which is not only a problem for companies such as Netflix and Amazon, whose customers are very intolerant of interruptions to their favorite shows and movies, but also for network operators, whose customers will churn if dissatisfied.

A common problem for ISPs deluged with customer complaints about poor streaming quality is the task of figuring out the cause of the diminishing quality. Sometimes throwing more bandwidth at the problem works, but this approach can be hit or miss, not to mention expensive. A better approach is proper analysis, which can be provided by holistic network analytics. Then, it becomes possible to isolate, for instance, a single router with a poorly configured cache, or a fail-over router that is too distant from the problem to be effective. In such a case, a simple cache reconfiguration and a bit of bandwidth in the right place can translate into happier customers and less churn, and all at a reduced cost to the ISP.

DPI and Encryption

The aforementioned 60 percent of internet traffic represented by streaming cloud apps and services is also increasingly encrypted — and there will come a point in the not-too-distant future when most internet traffic will be encrypted. Although that is good for security, it is not so good for DPI-based analysis.

This is a critical point because many operators have relied on DPI to overcome the shortages of SNMP. One of the key problems with DPI is that it is expensive to implement and most operators can only use it to spot check their networks. However, the root problem with using DPI for network analytics is encryption. Signature-based classification on a packet’s payload can only work if it is in plain text. Otherwise, DPI is simply blind.

For example, one operator had 100 percent more Facebook traffic than it was catching with its DPI-based analytics because over half of it was encrypted. Only a holistic view of its network correlated with data and context gathered from the entire Internet was able to accurately track the encrypted Facebook traffic to and through their network, providing more accurate data for network planning. Across the entire network, its DPI-based analytics solution was leaving 70 percent of top-application traffic flows unclassified.

Using a more comprehensive data set makes it possible to map flows from source to destination. By correlating telemetry from internet endpoints, DNS requests and a host of other information sources, it becomes possible to create a historically rich analysis that identifies up to 90 percent of the traffic — whether it is encrypted or not. This not only avoids all of the privacy issues raised by DPI technology, but also represents the only effective way to understand what is actually flowing to and through the network.