ranty, impulsive
Web In A Box
Posts relating to my role at my hosting company, Web In A Box
Startups– Exciting! Awesome! Lots of damn work!
14 years
by Shane Short
in Web In A Box
If you ever find someone who says that doing your own startup is a good idea, listen to them.. if they tell you at any stage it will be ‘easy’, slap them in the face for me.
While on the surface most startups appear to be easy, usually there’s some serviceable market you want to target, you think “easy, lets build something that services those people.. then it’s profit!” but let me stop you right there and share my story with you 🙂
Web In A Box started as an idea while carpooling on the way home from a previous job (and the business name was thought of on a very similar car trip…). That was nearly 4 years ago now and while Web In A Box is a self sustaining business now– it’s still got a long way to go to be as awesome as those little initial thought bubbles said it would be.
It doesn’t help that none of us worked on it full time (until recently) so progress was.. slow. The initial scope of what we were planning to do ballooned out of control and quickly we realised what we’d gotten ourselves into.
This isn’t to say that the trip hasn’t been extremely rewarding, writing a billing system and building a full hosting cluster around it has been challenging and fun, it’s just alot more work than any of us had in mind at the time.
The main reason it’s taken so long is we’ve generally had a stance of “if we can’t fully integrate it in to our management UI and have it completely automated, we’re not doing it.” which has obviously made certain projects take longer than they’d otherwise take elsewhere.
So, go-on, do build your own startup from the ground up– be a perfectionist, but be prepared for the amount of work involved 🙂
External server monitoring with Panopta
14 years
by Shane Short
in Web In A Box
One of the most important things when you’re running a hosting business, is keeping an eye on your infrastructure so you can be on top of any faults that might arise.
At Web In A Box, we run our own internal monitoring using cacti+nagios which keeps an eye on all our stuff.
The problem with running internal monitoring is, it’s not really done from a customer’s perspective– if there’s an upstream fault which would cause your customers problems, your monitoring won’t always catch it.
To combat this, we signed up for a monitoring service from a company called Panopta (http://www.panopta.com). Panopta has a network of monitoring nodes all around the world (20, at last count). You select your primary monitoring point (in our case, Sydney) and Panopta probes the services within your network every 60 seconds.
In the event of a failure, the other monitoring nodes will start actively checking, which means it lowers false positives, while allowing you to gain a bit more perspective on faults, allowing you pin-point geographically where the fault may lie, if it’s upstream.
Panopta also keep historic trending data and also allow you to have a public facing availability report:
An example of our public report
Response times from our three DNS Servers
So if you’re looking for a decent way to keep an eye on your network, give Panopta a look 🙂
For the love of god, update your “Bogon” list.
14 years
by Shane Short
in Web In A Box
For those not familiar, I am a director in a small web hosting company called Web In A Box (http://www.webinabox.net.au).
Earlier this year we were allocated our first shiny, new, IP allocation from the local RIR, APNIC. For what should’ve been a happy happy time for us, turned sour, quickly.
After migrating a good chunk of our infrastructure into our new IP space, it become apparent that something was a bit off with our IP space. We started running into all kinds of connectivity issues, even with some of our own machines overseas!
After some investigation it appeared that our IP space was in fact, quite new and shiny. So new and shiny infact, that a good chunk of the internet still thought the “supernet” (110.0.0.0/8) it came from was still unallocated!
I’ll stop here and provide a bit of backfill so you can understand the situation. Nasty people on the internet (spammers, DoS’ers etc) have long hi-jacked other people’s IP space, or even hi-jacked unallocated space, in an attempt to evade blacklisting/firewalling. In an attempt to thwart the mean men, several “Bogon” lists were published with (at the time) unallocated IP blocks, so people could firewall/blacklist them, so mean men couldn’t use them for the forces of evil.
As the story often goes in IT, the usual churnover of staff meant these bogon lists often went unattended as time went on. This was seemingly fine for a while, until the unthinkable happened. The internet started running low on IPv4 space, so those “Bogon” ranges started being allocated out to people who needed them (like us) and because these filters weren’t being maintained, connectivity to these new IP addresses was being blocked!
Now, back to the story. Because of the very nature of these lists and the fact they’ve often gone unattended and been completely forgotten about, getting them removed is a complete pain in the ass. We’ve had people flat out deny they’ve even got “Bogon” blocks in place, only through insistence on our part have they gone and checked and eventually rectified the problem.
So, where to go from here? Well, there’s plenty of debate on the ‘tubes over the ongoing effectiveness of “Bogon” filtering, but for the people who think it’s a good idea, how do they implement it without it becoming a ticking time bomb next time someone forgets it?
Well, The guys at Team Cymru (http://www.team-cymru.org/) have released a BGP ‘Black-hole’ service which fits the bill quite nicely. We ourselves have turned up a BGP session with them and we’re currently receiving 26 prefixes from their route reflectors. We use a simple route-map to install a null routed next hop, from the routes we receive from them. If you’re worried that the cymru guys could send you some nasty routes and blow your network up, you can simply do what we’ve done– We’ve set up a prefix list with the current “Bogon” prefixes allowed, but nothing more. Because IPv4 is running out, the likelyhood of something being added to the “bogon” list, is slim to none, so doing this prevents the Cymru guys sending us any prefixes we wern’t otherwise expecting, but they’re free to withdraw any they like, as the space gets consumed.
So if you’re currently running “Bogon” filtering within your network please, please, PLEASE consider switching to the Cymru BGP feed, or at worst, set some kind of automatic script, based on the lists they publish. We’d REALLY appreciate it.