Subscribe to Newsletter
Tell a Friend
Print this Page
06/12/2007Comparing Service Availability in Philadelphia and Portland
Developing a more precise language and getting more clarity on the metrics for Metro Wi-Fi is not just an academic exercise.
To better define the requirement for “coverage,” we need to know what type of coverage, what type of wireless client, the type of application to be supported, and how much coverage is required.
Type of coverage — Is it indoor coverage or outdoor? If it is indoor coverage in an urban setting, how many stories high should be covered? Indoor coverage will require a higher density of access points in the Metro Wi-Fi infrastructure and a fixed, higher-powered client device.
Type of client device — The performance of Metro Wi-Fi networks is limited by the performance of the Wi-Fi client. The access points in Metro Wi-Fi infrastructure can typically transmit at much greater distances than Wi-Fi client devices. There is a wide variance in the performance of Wi-Fi clients. Small handheld client devices will have lower transmit power and limited antennas. lt is important to specify which type of device is going to be used in the application and for network validation.
Type of application — Public safety and residential broadband present very different requirements for a Metro Wi-Fi infrastructure. Police cars usually have a very high power Wi-Fi radio (or 4.9 GHz) and high-gain antenna. Public safety requires mobility and outdoor coverage throughout the relevant jurisdiction — even in unpopulated areas. Residential broadband has no mobility requirement, but demands indoor coverage in populated areas, which is much more difficult to achieve with Metro Wi-Fi infrstructure.
How much coverage? — To quantify this we will use another term: Service Availability. Service Availability means the wireless service is visible, it is possible to log in, and it is possible to do the desired application. For voice service on the cellular networks this would be: I can see bars on my phone, I can make a call and have a reasonable quality voice conversation. For public access Internet service on a Metro Wi-Fi network, it means: I can see the network in my list of available networks, I can log in to the network, and I can browse the Web or check my e-mail at an acceptable performance level. Service Availability is expressed as a percentage: the percentage of locations tried within the coverage area where Service Availability is true.
This specific notion of Service Availability is perhaps a bit more conservative than what people usually think of as percent coverage. We took an informal straw poll of the User Experience Roundtable participants at Digital Cities Convention in Chicago and came up with the following Service Availability expectations: - Landline Phone – 99.9%
- Cellular Phone – 90%
- Metro Wi-Fi, Public Safety – 95%, outdoor coverage, high power client
- Metro Wi-Fi, Public Access – 85%, outdoor coverage, low power client
How Does This Apply in the Real World? Two very public Metro Wi-Fi network trials were recently accepted and moved into full deployment: Portland and Philadelphia. They could both benefit from more precise language in their RFPs and contracts.
In Portland, the trial network was tested and certified by a third party and it passed the test. However, end-user experience with the network and other third-party testing has shown disappointing coverage. Our own testing of Portland for the Novarum Wireless Broadband Review measured 60% Service Availability (using the same definition here) to a high-power Wi-Fi client outdoors and only around 30% to a standard Wi-Fi client. Clearly this is not the desired result. Was the Portland network certification testing wrong? Probably not. There is, however, a disconnect between the requirements for the network in the RFP and real user experience. The Network test RFP called for testers to locate each access point in the Metro Wi-Fi infrastructure and conduct their testing within a prescribed distance from the access point. Real-life customers don’t use wireless services that way. The network meets the RFP requirements, but doesn’t live up to user expectations.
Philadelphia is a different story. There have certainly been user complaints about the network there, but it is actually one of the better performing Metro Wi-Fi networks deployed so far. On the second test for the Novarum Wireless Broadband Review , we measured 74% Service Availability to a standard Wi-Fi client and 83% Service Availability to a high-powered Wi-Fi client outdoors over the 15-square-mile trial area. There is room for improvement, but these are good numbers for the early Metro Wi-Fi networks. The original RFP for the Wireless Philadelphia network called for “95% in-street (outdoor) coverage” and “90% in-building (indoor) coverage”. Even though “Service Availability” may be a more conservative measure than “coverage” as defined by the Philadelphia RFP, we don’t believe that the network meets the language in the RFP or the EarthLink contract.
The network falls short on another metric from the RFP that is simpler to measure. The RFP called for “average net throughput per subscriber of one (1) megabit per second (Mbps) upstream (client device to network) and downstream (network to client device) transmission”. The EarthLink Feather network delivers more than one megabit per second downstream which is excellent. However, it delivers only 275 kilobits per second upstream. That is fine for public internet access, but not even close to the symmetric service called for in the RFP. Wireless Philadelphia did have the network independently tested. In the end, we believe that they decided to accept the network even though it did not meet the contract language. That seems like the right decision: The Philadelphia network delivers solid performance and continues to improve over time. However, all parties would benefit from more clear language in the RFP and contracts.
If we use the Service Availability notion defined here, a more reasonable coverage requirement for the Philadelphia network might be expressed as: “85% Service Availability outdoors to a standard Wi-Fi client (30 millliwatt) anywhere within the coverage area”. Wireless veteran Phil Belanger is co-principal at Novarum, a strategic consulting firm assessing the performance of citywide wireless networks. He and the Novarum team chair the Wireless Networks User Experience Roundtable at the W2i Digital Cities Convention.
back
Comments
Phil Belanger
Anthony,
Nice comment.
We agree that the testing should be at a detailed technical level. speedtest.net is fine for simple testing, but more detail is better. We use Ixia Chariot for the Novarum Wireless Broadband Review. We’ve developed scripts that allow us to repeat the same exact test from different locations or with different wireless clients. Chariot will also let us simulate voice calls.
We agree that it is important to do acceptance testing with exactly the same client configuration that will be used with the real application. Wi-Fi clients vary widely and the client performance is the limiting factor in the range and coverage of Metro Wi-Fi networks.
The main point I wanted to make with this post is that we need to use more precise language to describe network requirements in RFPs, contracts and acceptance testing. Without clear language mismatched expectations are inevitable.
03:07 AM, 06/16/2007
Anthony Tull
As a city that just recently took back our WIFI network from the ISP and redeployed the assets to better fit our needs and those of our citizens we have some insight to what delivery of service actually means.
First it is very important that the method of measuring network throughput be defined at a very technical level. Are we talking measuring using a network tool similar to what you find on speedtest.net or is this a locally loaded utility on a server connected directly to the network. What platform are we using to measure it with, Windows 2000, Windows XP, or some flavor of linux. Believe it or not all three platforms will render different results regardless of what test you use. What is the power level definition of a high powered WIFI Client. We have tested over 12 differnet models of "supposedly" high power clients and the results were not close at all, even with devices of the same advertised power level outputs.
Ultimately the test needs to be made using what is actually going to be used by the end users regardless of whether this is a consumer or an owner. And the test for acceptance needs to be done using what network infrastrutre will actually be in place when the system is declared operational. It does not do any good to test throughput if you are going to add routers, switches, and gateways in the pipeline after the test.
I do agree with you though that the right decision was made to accept the results. As we have found in Granbury the tuning of the network is an ongoing process that continues as users are added, applications are added, and expansion and growth continue.
08:06 PM, 06/14/2007
Daniel Aghion
Excellent point. I remember attending a roundtable the author chaired in Tempe on the subject of establishing more precise language around user experience concepts. Where can we find the output of that session, which can benefit us going forward?
01:03 PM, 06/14/2007
Post new comment:Only registered users can add comments.
Please Log-in
| 
