At home: Using both Comcast and Monkeybrains for their different strengths.

• Comcast (aka “the man”): I’m using what used to be a 30mbps down/7mbps up DOCSIS 3 connection with an onstensible 250GB/month cap for $74.95/month (ostensibly $62.95/month but with a surcharge for not buying additional services from them). Last year they removed even the threat of overage charges for exceeding the cap (this may change someday but it’s nice for now), and sometime this year they seem to have also stopped shaping my traffic to the 30mbps tier; I now reliably get about 55mbps down and 10mbps up. (DOCSIS cable is a shared medium where you fight your neighbors for bandwidth, but I have never once seen speeds drop below the shaping cap; either Comcast does a good job of provisioning enough capacity or I’m extraordinarily lucky.) I don’t love Comcast’s corporate policies, the fact they’re always trying to upsell me TV services I don’t want, or their “have you tried rebooting it” approach to tech support. But their network works well.
• Monkeybrains (a San Francisco local neighborhood provider): Their plan for non-businesses is $35/month over 5GHz point-to-multipoint wireless links, using roof-mounted Ubiquiti 5GHz dish antennas. I hadn’t realized equipment like this was generally available, or that you could do reliable long-distance wireless links in unlicensed spectrum. It works pretty well. Non-business customers get a shared medium symmetric link; Monkeybrains tells you to expect somewhere around 20mbps in each direction if you’re lucky but less if the link is poor or congested. In practice, it’s been reliable except during one storm, and I do get about 20mbps in each direction most of the time, though in the evenings when everyone is watching Netflix, I’ve seen it drop to about 3mbps (so here, the shared medium effect is often visible). Also, latency is very good (around 4ms for the last-mile hop); far better than I’ve ever seen from cable or DSL. I chose to get the additional Monkeybrains connection for 3 reasons: first to see if it could completely replace cable, second for redundancy and additional capacity especially upstream, and third to support a local business. Given the speed characteristics, it’s a great value for upstream capacity (over the course of a month, ignoring nightly slowdowns, I could transfer more up through Monkeybrains than even a much more expensive Comcast connection), but it’s often not good enough for streaming video in the evening, so I haven’t chosen to replace cable entirely.
• Configuring multiple WAN providers is hard. I have my custom Linux router and I’m configuring the firewall, shaping, and routing via shorewall. I find Shorewall a lot easier to deal with than raw tc and iptables, but even after reading all of Shorewall’s Multi-ISP guide and way too much of LARTC, I still haven’t figured out how to configure things very well. In a perfect world, everything would use Multipath TCP or the equivalent, and adding additional WAN providers would immediately give you more speed, even for a single stream. In the real world, Multipath TCP is not widely deployed, and unless you run the equivalent of a VPN tunnel across all your links to a cooperating router elsewhere, single streams use only a single link. Also, you need fairly smart software watching the links and determining their health and reconfiguring routing if you want the ideal reliability case, where traffic flows as long as at least 1 link is healthy. I haven’t tried all that hard, but I haven’t even gotten this working. In practice, what’s easy to set up leaves you with a combined link that’s only as reliable as the weakest link: if any link goes down, at least some traffic stops flowing. I dabbled for a while in round-robining traffic across links (which is kind of like SMP for your CPU: it doesn’t help at all with the max speed of an individual stream, but helps keep multiple streams from interfering, so with a large number of users, it’s a decent approximation of the ideal case), but given the reliability problems and also the naive load balancing that would often result in slower individual streams by choosing the wrong link, I’m now running only Crashplan backup over Monkeybrains, and everything else over Comcast.¹

Wireless: I switched our family plan from Sprint to Verizon, despairing that Sprint’s promised “Network Vision network upgrades will ever bear fruit here in the Bay Area. I’ve always like Sprint’s focus on less-confusing, lower-priced plans with an emphasis on unlimited data, but Network Vision has been in progress since 2010, their 3G network has been getting slower in my experience ever since then, and their LTE network is still on the horizon. Now that they’re finally converting Nextel’s lower-frequency spectrum for their own purposes, this could still turn out well, but after waiting 3 years for Network Vision to yield results, I was tired of holding my breath, and I sure wasn’t going to sign another 2 year contract under those conditions. Verizon is also The Man, but it turns out they basically don’t charge any more for a basically equivalent² 4-line family plan for 4 smartphones. So now we’re on Verizon’s LTE network, which ranges from awesome (20mbps down, 7mbps up) in downtown SF to barely passable (data barely trickles in each direction) around our house in the Mission District. Most of the time it works pretty well, though. Latencies on both Verizon’s LTE and 3G networks seem substantially better than Sprint’s 3G network (RTT on Verizon is usually around 70ms; with Sprint it was usually 200ms), which might also be a big factor in it feeling generally snappier for data. True, we had to give up unlimited data, but in practice (partly because it was so slow, partly just due to our natural usage patterns) we weren’t using all that much data on Sprint, and given the observed tradeoff, I actually think $10/GB is a fair price to pay for reliably fast data.

At work: A quick mention that at work, we’ve tried both Comcast and WebPass, another local-ish wireless provider with a focus on commercial installations. I like what WebPass is offering, because they sell symmetric connections and can scale up to quite high speeds. Unfortunately the plans that are cost-competitive with Comcast are quite a bit slower downstream, and reliability has not been all that good. The wireless link layer itself seems fine, but I’ve seen more than a few weird routing problems, and their tech support doesn’t reliably fix these quickly when reported or even call back with promised updates.