September 3, 2019

By Cyril Doussau, VP of Marketing, Netrounds

According to a recent survey sponsored by Netrounds, unexpected and unintentional drops in network quality, also called network brownouts, are currently mostly ignored by many IT organizations. The fact is that 61% of Network Brownouts are NOT discovered by IT/NetOps with 42% being discovered directly by customers and employees. 

One would think that the reason they are ignored is that network brownouts are deemed unimportant and not damaging for the business; yet the average cost associated with brownouts is estimated by C-levels at $600K per year. This is quite an astonishing expenditure that could have easily been avoided with the right solutions.

The 400 NetOps managers, end-users and C-levels that replied to the survey revealed that the top costs of brownouts are lost productivity, lost revenue, costs to mitigate the effects of brownouts, and a damaged reputation. 



So why aren’t we talking about this issue more often? Shouldn’t the industry start fixing the problem and working more proactively so that budget wasted on dealing with the damage caused by network brownouts can be redirected toward more intelligent investments?

IT/NetOps Departments are Actually Blind

To provide a fix, we first need to better understand the problem. Network operations are actually “blind” and fail to discover and manage brownouts due to the lack of the right tools.

It is rather easy to understand when there is an outage due to a equipment being down. (On a side note, if you are interested in learning how to proactively mitigate network outages, read our blog post “Were You Affected by the Google Cloud outage? An Analysis, and How Ongoing Monitoring Could Have Helped.”). By contrast, network brownouts are difficult to spot as they primarily are due to misconfigurations or non-optimal configurations in the service chain. 

Now, how do you know for sure if an end-to-end service is performing for all users and/or customers? Is it enough to check that every network infrastructure element performs well? In the era of network virtualization, do you focus only on network functions, or do you need to ensure functions are properly chained? How can your client demonstrate that the issues they are experiencing are rooted in unexpected and intermittent network quality drops rather than in their own applications or data centers/cloud infrastructure?

Today it is clear that many IT organizations rely on traditional monitoring solutions that are unable to mitigate network brownouts, as these solutions are looking at network devices and infrastructure instead of focusing on how employees and customers are able to use network services and corporate applications. There is an elephant in the NetOps room, and we should start talking about it!

The Current Gap in Most Network Monitoring Deployments

At the end of the day, the issue comes down to IT/NetOps not being adequately equipped, and there is a gap in most network monitoring implementations that is quite well explained by our very own, Dr. Stefan Vallin in his blog post “Network Monitoring Systems - What Are They? And What Are the Differentiators?. There are actually four categories of network monitoring systems, and proactive monitoring solutions are yet to be implemented in the majority of Service/Network Operation Centers. The four categories are: 

  1. Classical monitoring systems
  2. Telemetry and analytics-based systems
  3. Passive probes
  4. Active test and monitoring systems (the missing piece in most testing and monitoring systems)

To learn more about their respective usages and benefits, download our Monitoring Cheat Sheet.

The Digital Enterprise Journal (DEJ) in its 2019 research on IT operation revealed that the biggest challenge in IT today is the inability to prevent performance issues before users are impacted. Furthermore, 38% of IT organizations reported insufficient capabilities for performance testing.

The fact that most IT organizations are not yet equipped with active monitoring systems is surprising, as platforms such as Netrounds are easy to install and can bring value in just a few hours.

How the Best-Performing Companies Handle Brownouts

We have identified the network monitoring gap, and documented and quantified IT costs from network brownouts. The obvious question is: What to do about them?

Thanks to the Netrounds brownout survey we also learned that top-performing organizations were 1.5 times as likely to run activation tests before giving users access to a new service or network. Top-performing organizations were also 70 percent more likely to set up a specific monitor for new services once launched. Organizations that run active testing and monitoring see 5 to 10 times fewer brownouts and can identify the cause of brownouts much faster.

While those findings may not be a surprise to Netrounds’ customers, they confirm that there is a relatively easy answer to the network brownout problem. 

Becoming a Best-Performing Organization

The companies willing to tackle this problem can learn from the best-performing organizations by following five easy steps:

  1. Realize that the problem is very common. Severe network brownouts are in a majority of cases reported by end-users and customers not by existing monitoring solutions.
  2. Quantify the impact on the business. Recognize that the consequences of not detecting network brownouts proactively are customer loss and a sharp  reduction of employee productivity.
  3. Analyze the gaps in current service assurance solutions. Accept that classical monitoring solutions focus on device health and cannot detect complex network service issues affecting customers and end-users.
  4. Research innovative solutions. There is a missing component in the assurance/monitoring stack that simulates end-user behavior from the right locations for the relevant network services using synthetic/active traffic. Evaluate solutions that are easy to deploy and use.
  5. Augment current solutions with automated, active assurance. Introduce active synthetic testing and monitoring of network services to complement existing monitoring solutions. This will measure network quality on the data plane, in the same way as customers and end-users consume the network, and will help detect the majority of the issues missed today.

With the adoption of disruptive technologies such as hybrid clouds, SD-WAN and network virtualization, the cost of brownouts will only continue to increase. Organizations that tackle this problem can begin investing brownout costs more resourcefully and embrace their digital transformation journey with peace of mind.