Migrating from Cloud to On-Prem?
Is this recent trend smart?
While everyone is moving to the cloud, some people are moving OFF the cloud. Naturally this peaks my interest since
has a “do the opposite” mantra, so let’s look at some of the factors that could lead someone to stick with data centers, or leave the cloud in favor of data centers.One of the recent examples that got a fair bit of media buzz was Basecamp’s cloud exit, and the two follow up podcast episodes he did on the subject. Lucky for you I’ve read the article and listened to the two podcast episodes. a16z also put out a piece on cloud spend and moving back to data centers here. As many of you know, I pay my bills by working on the cloud, but I am going into this with an open mind to look at the evidence provided. Decisions on migrating to or off cloud are highly personalized and subject to many caveats.
Exiting the Cloud
Benefits of Cloud
Quick recap on my current position on cloud. To reiterate the way I see it there are 3 primary benefits of cloud: virtually uncapped elasticity/scale, easy geographic distribution, and speed to market
If you’ve worked on-prem or managed your own data center you know how serious of a job capacity guessing is. Too little and customer experience is impacted or even worse, downtime. Too much capacity and your finance counterparts get to dislike you as you are wasting money on idle capacity. The classic quandary on on-prem infrastructure. Nothing new here, but still very frustrating. It is not uncommon to have a team dedicated to guessing capacity.
The second nice thing about the cloud is that you can provision infrastructure virtually anywhere in a couple clicks. This is beneficial because getting closer to your customer can benefit customer experience by reducing latency. In some cases this can also be important for data privacy laws.
Lastly, and in my opinion the biggest benefit of the cloud is speed to market. In my opinion, cloud is still the best option for iterating and bringing a product to market. Cloud platforms are built for that purpose and facilitate the product development process. You can say that a company can build similar around a data center, in terms of the control plane, and APIs to managed infrastructure. Yes that is true but that is a large lift, and labor costs may be greater than entering into a cloud agreement. Additionally taking advantage of fully managed services and/or serverless can greatly increase your speed to market as you get rich features out of the box.
The Cost of Cloud
The number one reason firms want to leave the cloud is the cost. If you are a long time Celt’er you know my first question: what is being done to control your cloud bill? I have written two posts on this matter here and here. And if you’re a really early supporter you’ll remember this post, What Most People Don't Know About Migrating to Cloud. My thesis in that post was that cloud is not just magically cheaper than having a data center. It can be if you make it cheaper, and there are some very specific scenarios where it may not be possible to do that.
You absolutely must be making a concentrated effort to controlling your cloud bill. If you are not please don’t complain about cloud spend. This variability comes with moving off fixed price data center contracts. Yes there are some nightmare scenarios where you can rack up wild costs. You must take efforts control your spend. No excuses.
The author of the Basecamp article, Hansson states that for what they’re paying AWS and GCP they can purchase some beefy servers and rent space at a data center. Hansson finds comfort in this because of upfront cost with no surprises. These are all valid points.
Remember though, there is a tradeoff. Cloud providers are constantly adding and improving services. In addition to that there are some cloud specific services like Amazon Aurora PostGres Compatible database, which offer significantly better features and capabilities than competing services. The way the storage fleet works, and the AWS engine specific optimizations are not available elsewhere. To highlight other features customers get for free is AWS’ cloud optimized hypervisor Nitro or their processor, Gravitron. I can confirm that when we switched our databases to Gravitron instances we saw immediate cost save. It was like literally getting a mail in rebate, if anyone is old enough to remember thoes. Just a couple things to think about, remember everything is a tradeoff.
The other aspect of cloud bills Hansson highlights is the common selling point is that moving to cloud reduces the needs for Operations teams. Hansson says that operating a cloud environment required a same headcount in operations. I can’t really comment on their particular situation but my firm personally repurposed some of our Ops heads to different teams. Lesson here is to not take that mantra at face value.
In the a16z article the writers state that for an early stage company expecting growth, starting on cloud is no brainer. You can take advantage of serverless and fully managed services and scale and save time, ultimately dramatically increasing your time to market and keeping your organization lean. By leveraging fully managed services or serverless you are reducing the amount of people you need to operate. Of course this all comes at a price (remember tradeoffs?!), but this cost is worth it. In addition you don’t face the capacity issue. Missing your virality moment due to crashing servers can be a death mark for a startup.
Size Discounts
If you are a bigger customer with a cloud provider they are going to give you a discount on the platform and possibly some training credits. This comes down to your spend and your negotiation skills. If you are spending 20-30m+ /y on the platform my estimation is <10% discount from AWS. Other platforms will probably throw more discount, and that makes sense since AWS is the market leader. Most of the time these terms are not disclosed and under NDA, so that is all I have on this topic :)
There are also other saving plans that smaller companies can take advantage for including reserved instances, compute savings plans, and more. You must be taking advantage of these programs to reduce your cloud bill. On-Demand price plans are for when you truly do not know the estimated demand, and should not be used for long term production applications. In this case Hansson said this company was taking advantage of these programs, so nothing to criticize.
Support
Hansson repeatedly mentions that AWS and GCP support did not meet his expectations and that they often struggled to get timely support. Hansson said he much preferred going with a data center vendor as they always got back to him in a timely manner and gave him the support he needed. That makes sense. Unfortunately, being one of AWS’ largest customers my support experience is much different. Even though we are a small GCP customer we have a dedicated account team. Can’t add much here besides saying my support experience with AWS has been largely positive with a couple hiccups. One of those hiccups being us designing our data lake very poorly, leading to a complete rebuild in <5 years, which you can read my postmortem here.
Lack of Growth
Hansson points out that his products are not seeing rapid growth or spikey demand, and that the demand is very predictable, and thus there is little benefit to the scalability of the cloud. As mentioned, not having to guess capacity is a strength of the cloud so if you do not need to guess capacity then that is a compelling reason to put this workload on a server. What is the tradeoff? Cloud Service Providers (CSPs) are investing into improving service offerings, so you miss out on any innovation there. Additionally, if you are moving off managed or serverless services, that is another bummer. When using something like an EventBridge, you get A LOT of features out of the box.
The Cloud Paradox
In the a16z article, the authors suggest that cloud adoption poses a paradox, where early stage companies clearly should adopt the cloud due to the scaling and managed services. However, as the company grows and the demand curve levels out, and the company increases headcount, the need for cloud services diminishes. The authors suggest at that point there are benefits to moving back to data centers. At that point though, it would be very difficult to move all workloads back to a data center due to effort and mental lockin. I would also argue this is why GCP chose to focus on data and AI services because of how ‘sticky’ those services tend to be. Once your data is on GCP then you say, ‘maybe we should move this product there, the data is already there’, more on that in this post. My firm is a large fortune 100 financial institution, so by the paradox we should not migrate to cloud. However, we will realize cost savings when our data centers are shutdown. Possible we could be an outlier but our data center contracts were just getting out of control price wise. So again, there can be specific circumstances for each company.
‘Not What the DARPA Researchers Wanted’
Near the end Hansson says something like DARPA researchers would not have wanted the internet to be so centralized into the hands of a couple cloud conglomerates. This is possible, but remember ARPANET (first iteration of an internet) was primarily developed for the US government, specifically the DoD, so I’m not sure their intentions were that pure to begin with. Additionally, DARPA researchers were largely responsible for the Domain Name System (DNS) which is centralized, and a core fundamental function to the internet. If you dont know how DNS works, you’re in luck because another talented avatar,
and I wrote everything you need to know about it on :
Conclusion
Disregarding the last DARPA point, Hansson and a16z have compelling reasons companies should examine if cloud makes sense for them. The factors are highly personalized and specific to the firm, but both put out some general guidelines. Even though it can be damning to be the heterodox in technology, make the best decision for your firm, even if it is doing the opposite.
-celt