The future of servers, apps, etc



The future of the cloud is federated across multiple datacenters and corporations. Datacenters will become "dumb pipes". But first we need a good networking foundation.

Kubernetes is nice, but multihost networking has still not been implemented well. We are getting there. Openflow is maturing and nice solutions like openvswitch exist. For high bandwidth businesses, the best architecture is now an edge network of openvswitch containers, forwarding packets to an internal network of backend services. Virtualized switches can now process packets almost as fast as physical switches, and even faster with GPU parallelization (packetshader). Therefore it makes sense to perform routing logic at the edge layer, so the backend services only need to worry about their own application logic. Unfortunately, an edge network is by definition a multi-host network. Until multi-host networking works in something like Kubernetes, this optimal architecture (edge --> backend) is not appealing because orchestration is a nightmare.

(btw, the fact that commodity hardware is now capable of advanced networking is much of the reason companies like cloudflare, highwinds, fastly, etc. are making so much money. Their entire business is based on the "edge network" I described above)

A bigger obstacle in the way of mult-datacenter networking is price. Most cloud companies charge per gigabyte of transfer for bandwdith out of their networks. Upstream providers, which you can connect to when you own your own or rent hardware, charge per gigabyte of transfer, per second, 95% of the time. If you think of bandwidth as a water pipe, Amazon/google/etc charge for how much water you put through the pipe, whereas upstream providers charge you for the invariable size of the pipe. When you do the math, if you assume 95% 24/7 high bandwidth utilization, Amazon gouges you orders of magnitude more than upstream providers. If you are a business with a model depending on networking (any protocol business -- VPN, DNS, CDN, load balancing, proxying, etc) you need to assume that your customers will use max bandwidth. Many times they also do not expect to be charged for bandwidth, so you cannot charge them for it. Therefore you cannot use the cloud because its pricing model is prohibitively expensive.

Thus, bandwidth intensive companies are prohibited from the conveniences of the cloud. They need to build all their own infrastructure in order to take advantage of bandwidth pricing.

This is a huge market gap. The second "cloud revolution" will come in the networking layer.
 
Chatmasta,

How would an application running on 3 different hosts (say Linode, Digital Ocean, AWS), all in a certain area, say London, work?

Latency would be much higher than within one datacenter, would the idea of transactions have to be softened, or would reliance on async database updates have to become more normal?

Price between datacenters is one thing, but also there's very limited bandwidth between them, would the internal components of an application have to try and avoid being chatty?

Great post, I'd never thought much about this stuff before.
 
For web applications like that you are not going to run into an issue. I'm thinking more of businesses that are in the network business, i.e. that process packets in some way for their customers. Think content accelerators, CDN networks, DNS providers, or vpn providers. They are all bandwidth intensive because they need to absorb 100% of their customers' bandwidth while they operate. Most of them can't charge by the gigabyte so they can't use the cloud.

The reason these companies need big multi host edge networks is because requests come into their network from all over the globe and they need to route them to the right datacenter. The cross-datacenter networking happens "IN" when a client request goes to the edge (eg you browse a cloudflare site) and OUT when the return comes back to you.
 
chatmasta said:
The future of the cloud is federated across multiple datacenters and corporations. Datacenters will become "dumb pipes". But first we need a good networking foundation.

Kubernetes is nice, but multihost networking has still not been implemented well. We are getting there. Openflow is maturing and nice solutions like openvswitch exist. For high bandwidth businesses, the best architecture is now an edge network of openvswitch containers, forwarding packets to an internal network of backend services. Virtualized switches can now process packets almost as fast as physical switches, and even faster with GPU parallelization (packetshader). Therefore it makes sense to perform routing logic at the edge layer, so the backend services only need to worry about their own application logic. Unfortunately, an edge network is by definition a multi-host network. Until multi-host networking works in something like Kubernetes, this optimal architecture (edge --> backend) is not appealing because orchestration is a nightmare.

(btw, the fact that commodity hardware is now capable of advanced networking is much of the reason companies like cloudflare, highwinds, fastly, etc. are making so much money. Their entire business is based on the "edge network" I described above)

A bigger obstacle in the way of mult-datacenter networking is price. Most cloud companies charge per gigabyte of transfer for bandwdith out of their networks. Upstream providers, which you can connect to when you own your own or rent hardware, charge per gigabyte of transfer, per second, 95% of the time. If you think of bandwidth as a water pipe, Amazon/google/etc charge for how much water you put through the pipe, whereas upstream providers charge you for the invariable size of the pipe. When you do the math, if you assume 95% 24/7 high bandwidth utilization, Amazon gouges you orders of magnitude more than upstream providers. If you are a business with a model depending on networking (any protocol business -- VPN, DNS, CDN, load balancing, proxying, etc) you need to assume that your customers will use max bandwidth. Many times they also do not expect to be charged for bandwidth, so you cannot charge them for it. Therefore you cannot use the cloud because its pricing model is prohibitively expensive.

Thus, bandwidth intensive companies are prohibited from the conveniences of the cloud. They need to build all their own infrastructure in order to take advantage of bandwidth pricing.

This is a huge market gap. The second "cloud revolution" will come in the networking layer.
Click to expand...

or7GL9q.jpg
 
Jon taught me the best time to respond to forum threads is after eating candy measured in milligrams.
 
mattseh said:
Chatmasta,

How would an application running on 3 different hosts (say Linode, Digital Ocean, AWS), all in a certain area, say London, work?

Latency would be much higher than within one datacenter, would the idea of transactions have to be softened, or would reliance on async database updates have to become more normal?

Price between datacenters is one thing, but also there's very limited bandwidth between them, would the internal components of an application have to try and avoid being chatty?

Great post, I'd never thought much about this stuff before.
Click to expand...

We've yanked some CDN usage because latency went from 22ms on-server to 400ms with a CDN :( (Akamai)
 
mattseh said:
Latency would be much higher than within one datacenter, would the idea of transactions have to be softened, or would reliance on async database updates have to become more normal?
Click to expand...

On one hand you're absolutely right, on the other hand you're probably thinking more long term than is feasible right now.

From the software engineering side, async everything is the long term answer. The idea of async database updates or threading is just a bandaid to think about in the short term. Blocking code itself is a bottleneck that most of the industry is still dealing with. If you're developing from the ground up on Node you're set up to actually think about whether you're going to wait for a callback from the database to send your response, and you really almost never have to do that. When all of your code runs asynchronously it just seems absolutely insane that there are still PHP/MySQL applications running in the wild.

If you want to take it even further, the relational database itself is a bottleneck that can be eliminated in most situations. Just like developing on Node makes you realize how unlikely it is you'll need to wait for a database update to finish before sending a response, using a NoSQL database makes you realize you almost never need a normalized relational database to maintain data integrity. For a long time we used them because they took care of low level stuff, but Javascript is so elegant that it's actually easier to just write good code than configure your DBMS.

Since I'm ranting on these awful bottlenecks, MVC sucks too. It was the best design pattern to run on an Apache server, but now we know that the whole idea of Apache sucks ass. Write middleware to handle HTTP requests and writing an MVC application for the web seems insane.

Now as far as what the servers will be like from an ops perspective, kubernetes is what I think things will be like. If you take it as it is now and use it with some clean engineering like I described, it's actually very simple to manage and scale.

The networking end of it isn't my area of expertise, but I mostly agree with chatmasta. The only thing I would argue is that price is relative. I'm gonna ballpark that the last site I worked for served 50,000,000 pages/month so you can figure that Akamai would cost about the salary of 1 or 2 adequate sysops. Using them we only needed one with a very manageable workload, so it's a no brainer. On the other hand I can see how if you're below a certain level it might seem better to roll your own.
 
Yeah I was referring to companies who want to compete with Akamai, i.e. run their own bandwidth intensive businesses. Akamai helps if your bandwidth intensity is not tied to cost, because users are just downloading a bunch of pictures or something. But if your bandwidth intensity is due to your fundamental service being a network value-add, like DNS or CDN, then you are competing with Akamai and need competitive bandwidth pricing.

Elon Musk advocates analyzing systems by reducing them to their "fundamental truths." In physics this is a strong notion because the universe is governed by laws that are literally, fundamental truths. The "fundamental truth" of the Internet is that it's a network with packets flowing through it . The route of any given packet must cross multiple nodes in the network. Some nodes modify the packet, some nodes forward it, some nodes even drop it.

Fundamentally, the fact is that packets are transferred between nodes, along fiber optic cables. Each cable has a capacity, a supply, and a demand. At 9pm on a weekday, when Netflix is saturating the cables in a region, supply is low and demand is high, so price should be high. To me it seems like a natural market should exist for transit along these cables. The current pricing structure of charging per gb of transfer, or even per capacity, is failing to capture economic value.

My long term vision and the business I hope to own is a real time bandwidth market. Literally, a market order book just like a currency exchange. Each order is a buy or sell for an "edge" in the network graph at a given time, for a given duration, for a given price. For example "buy 10gbps capacity between endpoint A and endpoint B at 6:00pm monday for an hour."

Fundamentally, bandwidth is a scarce resource in realtime. What I am suggesting sounds like the complete opposite of net neutrality, but interestingly, if you read the proposal from the FCC, they specifically disqualify "Internet backbone services" as ISPs. Therefore companies in the backbone routing business are not subject to neutrality rules (and how could they be, when peering already exists at such a high level?). In fact what I am suggesting already exists -- it's called BGP peering and it's essentially traffic trading between large IP transit holders. I simply want to bring it to a wider audience. I think selling bandwidth at the application protocol level is the best way to accomplish this, and I'm excited for the rise of software defined networking. Virtualizing network functions is the obvious next step toward "cloud 2.0" but if we virtualize them then we should also be virtualizing their pricing. Why can't I buy bandwidth the same way I can buy CPU, memory, or storage?

If you like what I'm hearing, this is what I'm working towards long term. Short term my partner and I are trying to build this market for vpn providers. Long term we want to expand beyond vpn to other value-added services on the network.

If you think I'm full of bullshit and can't execute, then I suggest investing in equinix (EQIX). They are basically doing this on an enterprise scale, except they actually own datacenter real estate and then charge to run fiber into them. They just refilled as an REIT. Their next earnings call is this month and I would bet money it's gonna be big.
 
You must log in or register to reply here.
Top Bottom