In order to keep up, network visibility solutions must be designed and built on a highly scalable foundation. Without an ability to quickly scale, users will be limited in their ability to ingest data from multiple sources while also processing queries from multiple users.

Successful scaling depends on several factors, including elastic network capacity: fast ingest and query capabilities; and multi-tenancy to serve unlimited clients without creating bottlenecks. Elastic capacity helps to capture the full traffic details for pattern analysis, and to locate root causes.

Several benefits flow from such a highly scalable architecture. First, operators get a fully unified view to track and query all the relevant network information on a single platform. In addition, open access ensures that the platform integrates with complementary systems for business analytics or DDoS protection.

Most current architectures lack such scalability. For instance, open source software has led to many useful innovations, but it has also created some gaps. Most open source programs run on just one computer and they do not cluster, making it impossible to keep up with massive data flows. In addition, it is too inefficient to patch together a collection of open source tools such as SNMP and BGP to create a unified view for network monitoring, or to build an integrated database that can cross-reference sets of data for correlations. Such a patchwork approach quickly becomes too cumbersome to work well.

On the other hand, most current enterprise appliance systems have severely limited compute and storage capacities that do not scale up effectively. When one appliance fills up, users have to add another appliance, creating further problems for load balancing and data sharing.

Also, most enterprise appliances cannot index the full details of every traffic flow. Appliances are hard to integrate with other software solutions, and they can be difficult to provision and deploy. Obviously, such an architecture cannot produce a comprehensive solution for network visibility at full scale.

Getting from Here to There

Clearly, what’s needed today goes well beyond what’s available with open source software or enterprise appliance solutions. Network operators want a clear source of instant visibility to identify network performance problems and to detect incoming attacks. What’s needed is a scalable, big data platform.

Effective, next-generation visibility solutions will be able to handle traffic flows at web-scale. They will be scalable, open, and unified. They will connect a customized big-data back-end to an intuitive user interface on the portal front-end. A single-pane view will quickly reveal current network conditions, including congestion, bottlenecks and attacks.

To be successful in this era of network-centric operations, network visibility solutions need break down knowledge siloes and create opportunities to innovate at the speed of digital business.  With big data approaches, network managers can finally get a handle on network data details at scale, to optimize their resources, improve operational efficiencies, and deliver world class user experience that gains market share and grows revenue and profits. In that sense, it really does pay to know your network.