Hi Jeff, long time fan, first time commenter... I love building servers too and I've managed a small group of servers, I personally use Linode, and my currently company uses AWS and some internal servers...
You would agree that in coding you pick the right tool for the job (scientific computing would use a different technology stack than standard ecommerce startup website)...
AWS is elastic (you pay a premium for being to scale up or down - and there's value to the agility with which you can change or add new services)
AWS RDS is a huge improvement over managing MySQL replication, and they have ELB and lots of other addons that take serious Ops chops to create and maintain
Server operations cost is not the raw hardware:
a. The biggest cost in Ops is people (same as coding), so leveraging Amazon saves on how many people you need to pay to manage your server farm (yes, SysAdmins take holidays and change jobs so cost = N+1 )... you can outsource half way by colocating but the setting up the redundancy, monitoring, auto scaling, etc. becomes a physical pain (you want West Coast and East Coast servers, right).
b. The infrastructure of cooling, UPS, network (bandwidth!), backups, etc. is also a big factor in Operations (does your server room have building security? backup generator?)
My point is that for a stealth mode startup or any internal lab testing buying servers is a no brainer - do it with ESXi or OpenStack and hack away!
BUT for Production you'll need some Cloud strategy (AWS competitors: RedHat OpenShift, RackSpace Cloud, IBM, ATT Compute, Google AppEngine, etc. means lower prices and improved services)
As you've already mentioned if you happen to have hanging around a pile of cash and tech expertise that's underutilized...