Where's the Infrastructure Supermarket?

Question: What's the 1 thing every Internet company has to worry about, takes up lots of their time, and their customers never fully see or appreciate?

Answer: Their infrastructure.

Growing Internet companies all face this same problem. All the sudden your website starts to take off, users are signing up left and right, and then your servers start to get slower and slower. The IT guys are telling the marketing folks to stop their advertising-- all those PPC visitors are killing the servers! It becomes clear... time to upgrade the infrastructure!

Building infrastructure takes an enormous amount of time, resources, and money. Just think about all of the equipment running behind the scenes to power a service like Printfection: servers, data centers, databases, off-site backup locations, clusters, load balancers, switches, routers, etc.

After just about one year on-line we are starting to feel the infrastructure crunch. While our site is still running smooth today, we're preparing for the future (what good retailer doesn't think about Christmas in February!) It's becoming time to upgrade the hardware that powers our website. Simple, right? Just add a few more servers and everything will run faster and all our customers will be happy. WRONG! There is so much more to think about. Buying the servers and installing them is only about 1% of the problem. There are code modifications, database splitting, backup configurations, redundancy issues, and the never ending problem of worrying about how the solution you are designing today will scale as you grow bigger and bigger.

What I find funny (well, not really) is how every Internet company from Flickr to YouTube to Smugmug has gone through the same growing pains and is constantly trying to solve the same problems, yet everyone is doing this alone. Everyone is re-inventing the wheel in their own data centers. Configuring their own redundant clusters of servers, their own SANs, their own backup solutions, and more. This reminds me of pre-industrial America when most people lived on a farm and grew their own food, raised their own cattle, and were for the most part self-sufficient in their own little worlds. How come the Internet is like this now? Why are we all forced to be farmers on top of our normal jobs? How come we can't just buy our basic food (infrastructure) at the supermarket and concentrate on what we're really good at (selling t-shirts). Sometime in the future I hope we can just buy our infrastructure like we buy food at the supermarket. This would allow all Internet companies to concentrate on what they're really good at & would advance web-services at a much faster pace. It's nice to see Amazon Web Services entering this space, and I hope this becomes a mature industry with lots of options sooner rather than later.

Until this happens, we'll continue to worry about our own infrastructure so our website runs fast and reliably. Too bad our customers and store owners will never really see how much time and effort goes into the "behind the scenes" infrastructure projects. Yes, it does slow down development of new features. No, we cannot just forget about our infrastructure-- at least not yet.

Comments

[Eric from Raw Tees]

Speaking as someone with a background in IT, I fully appreciate what you have to do to keep things running smoothly! You're right, though, that 99% of the shopkeepers have no idea of the work involved just to keep the website going (let alone the other aspects of your business). It is discouraging, though, knowing that no one appreciates the work you're doing until the first time there's a glitch...and then all hell breaks loose! LOL. Well, be that as it may...there is a small minority of us who understand and appreciate all that you guys do. New features can wait.

[Candida]

I agree with Eric! New features can wait. Give us more power, faster speeds. People are impatient today, so with out the speed, they will be gone from our shops and the marketplace in the time it takes them to click the mouse.