Nehemiah Blackburn

CAF performance testing explained

As a vendor member of the Wireless Internet Service Providers Association (WISPA), Minim works with WISPs to deliver advanced WiFi and IoT security services to subscribers. To provide more support for WISPs and rural broadband providers alike, Minim is working on developing toolsets for rural carriers who receive funding through CAF.

CAF Performance Testing Explained

What is CAF?

The Connect America Fund (CAF), also known as the Universal Service Program for High-Cost Areas, is a rural broadband initiative by the FCC proposed in October 2011 as a means for closing America's urban-rural digital divide:

"In urban areas, 97% of Americans have access to high-speed fixed service. In rural areas, that number falls to 65%. And on Tribal lands, barely 60% have access. All told, nearly 30 million Americans cannot reap the benefits of the digital age."

CAF Phase II is currently underway, which is a six-year fund started in May 2015 to provide funding to price cap carriers to expand America's rural broadband deployments. Through CAF Phase II, the FCC will disburse $1.49 billion over a ten-year period directly to eligible telecommunication carriers (ETCs) who can help bring more Americans online.

ETCs are carriers who currently offer broadband services with a minimum 10 Mbps downstream, a minimum 1 Mbps upstream, and latency suitable for real-time applications, like VoIP.

Just recently, the 2nd round of CAF Phase II auction funding was authorized for $166.8 million, targeting more than 60,000 rural homes and businesses within the eligible areas depicted in the map below:

CAF Phase II Auction: Final Eligible Areas

Screenshot taken August 5th, 2019 of CAF Phase II Auction: Final Eligible Areas map.

Requirements for CAF-funded carriers

Last year on July 6, 2018, the FCC Order DA-18-710A1 was released, which outlines that all CAF-funded carriers must meet certain performance criteria when delivering broadband to their CAF-supported subscribers. In the mandate, it states that carriers must show they are using their CAF dollars to deliver acceptable internet speeds to their CAF-supported subscribers.

The way in which carriers are to prove they are meeting these internet speed requirements is through CAF performance testing. Failure to prove delivery of these minimum internet speeds through CAF performance testing puts the carrier at risk of having up to 25% of their monthly CAF funding withheld.

CAF performance testing: Step-by-step guide

CAF performance testing is a process that has been put in place ensure CAF-funded carriers are delivering acceptable internet speeds to their CAF-supported subscribers. To help CAF-funded carriers stay compliant and keep their funding, we've broken down the process into six actionable steps:

CAF Performance Testing: Choose a Testing Method

There are three testing methods for carriers to choose from to conduct their CAF performance testing:

  • Option A: Use the Measuring Broadband America (MBA) testing program
  • Option B: Use an existing network management system or tool
  • Option C: Use a provider-developed self-testing configuration

While all three testing methods allow carriers to conduct their CAF performance testing, there are some important aspects to consider before choosing a testing method, like:

  • What are the upfront installation and long-term support costs?
  • Is a certain hardware required, and if so, who is responsible for installation?
  • How reliable is the testing method (i.e., will it always yield accurate results?)

To provide some guidance on selecting a testing method, we've put together a CAF Performance Testing Method Overview that gives carriers insight into each testing method and the pros / cons to consider.

Download the overview

 

CAF Performance Testing: Choose Test Subjects

Once a testing method is selected, carriers must choose test subjects (i.e., testing locations). Testing locations used for CAF performance testing must be randomly selected every two years from among the carrier’s active CAF-supported subscriber base in each service tier and state combination. The number of testing locations required depends on the size of the carrier’s CAF-supported subscriber base:

Number of CAF-supported subscribers in each service tier / state combo Number of required testing locations
50 or fewer 5
51-500 10% of total subscribers
Over 500 50

 

Carriers can choose to include more testing locations for a given testing window (see Step 3), but cannot exclude any identified testing location during or after the testing takes place.

CAF Performance Testing: Understand the Testing Window

Carriers must conduct their CAF performance testing between the designated testing hours of 6:00 PM and 12:00 AM, which includes weekends. To capture any seasonality effects on a carrier’s broadband performance, carriers are required to conduct one week of testing in each quarter of the calendar year:

  • Q1: January through March
  • Q2: April through June
  • Q3: July through September
  • Q4: October through December

An important note is that testing for all carrier locations in a single service tier / state combination must be done during the same week. Additionally, there are opportunities to request a speed test waiver or extension in cases where a major disruptive event (e.g., a hurricane) negatively impacts a carrier's broadband performance. Alternatively, carriers can choose to reschedule their testing for another time within the 3-month testing window of the impacted quarter.

CAF Performance Testing: Conduct Speed Tests

Regardless of which testing method is chosen in Step 1, all speed tests must be conducted from the testing location at the subscriber premises to (or through) an FCC-designated Internet Exchange Point (IXP). These IXPs are located in 16 metropolitan areas across the US:

New York, NY; Washington, DC; Atlanta, GA; Miami, FL; Chicago, IL; Dallas-Fort Worth, TX; Los Angeles, CA; San Francisco, CA; Seattle, WA; Denver, CO; Salt Lake City, UT; St. Paul, MN; Helena, MT; Kansas City, MO; Phoenix, AZ; and Boston, MA.

Most mainland locations in the US are within 300 air miles of at least one of these IXPs, while all are within 500 air miles one.

Choose a testing method

The speed tests are defined as a single measurement of download or upload speed of a 10-15 second duration between the testing location at the subscriber premises and an FCC-designated IXP.

It is required that 80% of the download and upload speed measurements taken during CAF performance testing be at or above 80% of the CAF-required speed tier (i.e., an 80/80 standard). For example, if a carrier receives CAF funding for a 10/1 Mbps service tier, 80% of the download speed measurements must be at or above 8 Mbps, while 80% of the upload speed measurements must be at or above 0.8 Mbps.

The FCC requires a minimum of one download speed test and one upload speed test per testing hour at each subscriber premises testing location. Carriers who elect to do more than the minimum required number of speed tests at subscriber premises testing locations must include the results from all tests performed during the testing period for compliance calculations. In other words, no cherrypicking!

CAF Performance Testing: Conduct Latency Tests

In addition to measuring upload and download speeds, CAF performance testing is also measuring latency. The latency tests are defined as a measurement of the observed latency, including lost packets, between subscriber premises testing locations and an FCC-designated IXP.

Carriers must test and certify that 95% of the latency measurements taken during the testing hours are at or below 100ms. A minimum of one discrete latency test per minute (i.e., 60 tests per testing hour) at each of the subscriber premises testing locations is required. The results of each discrete test must be recorded separately.

During latency testing, if the subscriber load exceeds the 64 Kbps downstream cross-talk threshold, the carrier may cancel the test for the given minute and reevaluate the subscriber load before retrying the test in the next minute.

As with the speed testing, carriers who elect to do more than the minimum required number of latency tests at subscriber premises testing location must include the results from all tests performed during the testing period for compliance calculations.

CAF Performance Testing: Submit Test Results

The final step in conducting CAF performance testing is for carriers to submit their testing results. Carriers must report their results annually to the Universal Service Administrative Company (USAC), who has the right to audit test results and in some cases, request the carrier to submit results quarterly.

The Wireline Competition Bureau (WCB) is to provide guidance by public notice on how carriers are to submit their data to USAC. As of the publish date of this blog, this information has yet to be publicized.

Carriers must submit their speed test and latency test results, in addition to information on what testing method was used, what technologies were used to provide the broadband services for each service tier / state combination included in the testing, and certification.

Initially, carriers were to be prepared to submit their first set of testing results and certification by July 1, 2020, which would have included data for Q3 and Q4 of 2019. However, on May 30, 2019, the FCC released Public Notice DA 19-490 which announced the delay of CAF performance testing to begin Q1 of 2020. As such, the first set of testing results are to be submitted to USAC by July 1st, 2021. Submissions will continue to follow this same process, wherein July 1st is the submission date for the prior year's data.

Get started


Disclaimer: This is our interpretation of the CAF performance testing process as outlined in the July 6, 2018 FCC Order DA-18-710A1 and does not constitute legal advice.

(Last updated on August 19, 2019)

 

Like this blog?

Subscribe to our newsletter.