September 12, 2018

Encrypting the Long Haul

  Networking, Technology, TLS

     

This blog post describes an existing function of our service. No change has been made to our network.

The biggest decision we made at the start of NuevoCloud was the decision to develop our edge software from scratch, instead of building on top of Nginx or Varnish (despite their excellent work). The downside of this is that it has taken much longer than expected to develop our software. The upside is that we get to look at the way things currently work on the internet and ask ourselves how we would design it today, given everything that has been learned.

One such problem is HTTP and it's lack of security that makes it vulnerable to hostile actors (including hostile nation states, ISPs, and other network providers). In the real world, we've seen ISPs alter HTTP traffic to insert tracking headers, alter pages to display their own advertisements and strip content. Without encryption, there is no assurance that HTTP traffic is delivered to the browser in the same state it left the server.

"Without encryption, there is no assurance that HTTP traffic is delivered to the browser in the same state it left the server."

 

There are things that can be done to improve this situation. The first is our integration with Let's Encrypt to provide free SSL/TLS certificates for every domain using our service. There are many CDNs that provide free certificates and this is a significant improvement. It encrypts the first leg of the path between the edge server and the browser; but it does nothing to protect the next part of a packet's journey: from the edge server to your origin server. This is what this looks like in practice using a traditional CDN:

In this chart, a browser in Libya (yellow line) connects to the closest edge server in France (blue point), and the edge server connects to your origin server in the USA (red line). The Let's Encrypt certificate protects only the connection between the user in Libya and the edge node in France.

As a CDN, the second leg of the connection--the red line between the edge server in France and your server in the USA--is out of our control. Ideally every server administrator would also install a TLS certificate on their server and configure it properly to ensure communications to their server are protected. But the world is not ideal, and there are many servers that continue to use HTTP. Even if the server uses HTTPS, it may be insecure if, for example, it was configured improperly or was not updated periodically to remove weak ciphers.

Although we can't control how the server communicates with our edge servers, we can minimize its impact by reducing the distance between the edge server and the origin server. How can we do that? Simple, by adding another edge server to the map. Here's how NuevoCloud would transport the same connection above:

In this map, we can see another edge server near the origin server has been recruited to communicate with the origin server (red line). The distance over which the origin server may communicate insecurely has been reduced to the shortest possible distance. These two edge servers communicate over an encrypted persistent connection (blue line).

"The distance over which the origin server may communicate insecurely has been reduced to the shortest possible distance."

 

In addition to encrypting the long haul, this topology has another advantage: speed. NuevoCloud maintains these persistent connections between every PoP. Traffic can be sent over these links immediately, without negotiating a connection or exchanging encryption keys. More about that in a later article.


  2