Hannes Tschofenig

At the EU Emergency Services Workshop 2013 a regulatory issues panel moderated by Tony O’Brien (Deputy Executive Director, EENA). The panel discussed a couple of topics, including the need for standards, performance indicators, and location. The location slot lead to a heated discussion with the following persons on the panel: Raed Arafat (Secretary of State for Health, Romania), Laszlo Toth (Policy Director, GSMA Europe), John Medland (999/112 Policy Manager, BT), Paulo Pereira (ANACOM, Portal), Florin Dragomir (National Authority for Management and Regulation, Romania), and Gyula Bara (European Commission).

Here is a little bit of background: Emergency calls require location information for three reasons: (1) location is necessary to route the emergency call, (2) to dispatch first responders the location of the person seeking help is needed, and (3) emergency numbers are often interpreted in context of location since different numbers are used in different countries. In the context of this panel discussion the focus is on location for dispatch.

In general, more accurate location is better for emergency services organizations since it allows to dispatch first responders faster, particularly with incidents in rural areas where search can take a longer time and tends to be more costly.

With regard to location accuracy in Europe emergency services authorities receive cell-id location. Cell-id location is rather coarse grained. Better location determination mechanisms are available to-do that offer much better accuracy and those have been deployed in the US for some time already.

The discussion that surfaced again in Europe was whether requirements for better location accuracy should be introduced because industry (mostly telecommunication operators) does not deploy them by themselves. For some reason mobile phone operators have not been able to develop a business model for location-based services that would justify the investments into their networks. Consequently, the technology is not deployed in mobile networks today in Europe. Should the European Commission and national regulators therefore demand the deployment of this location technology? Should other approaches (such as utilizing location available at smart phones) be re-used?

Here is a recording of the discussion and make your own judgement.

Recording of the Location Panel Discussion [ 32:18 ] Play Now | Play in Popup | Download

Tags: EENA Workshop, emergency services, Location | Category: emergency services, workshop | No Comments »

Last week EENA organized their yearly EU emergency services workshop in Riga/Latvia. As in the past many emergency services authorities, regulators, vendors, and service providers showed up to discuss the hottest topics.

I had the pleasure to meet Mark Fletcher again, who arrived in Riga with his podcasting equipment.

Mark interviewed me about the Next Generation 112 work done in EENA as it was again on the agenda of this meeting because of the recently shipped update of the specification.

Here is the podcast:

Standard Podcast Play Now | Play in Popup | Download

Mark regularly interviews folks from the emergency services community in his E911 podcast series.

Tags: EENA, EU Emergency Services Workshop, Mark Fletcher | Category: emergency services, workshop | No Comments »

Dave Täht created a “Realtime Response Under Load (RRUL) Test” specification and he describes it in the following way: “The Realtime Response Under Load (RRUL) test puts a network under worst case conditions, and then measures for fairness, latency, realtime responsiveness, classification based optimization (and preservation), and the relative performance of TCP and UDP streams of varying rates, while under that load. Most importantly, it tests for web page load time performance while under these loads.”

Toke Høiland-Jørgensen decided that it would be a good idea to produce a wrapper for Netperf to implement the test cases that Dave put together. Here is a short description to get the netperf-wrapper working.

First, you might want to check whether you have netperf running on your system already (and which version):

# netperf -V

You have to retrieve Netperf version 2.6. When I tried it the netperf-wrapper did not work with earlier versions.

# svn co http://www.netperf.org/svn/netperf2/tags/netperf-2.6.0

Then, switch to the netperf directory and compile it with

# ./configure --prefix=/usr/ --enable-demo=yes

On Ubuntu I had to use the –enable-demo=yes switch since otherwise netperf wouldn’t work with the wrapper. I also got an error message.

Subsequently, run

# make

# make check

# make install

On one of my Linux machines I had problems doing the make and it turned out executing ./autogen.sh helped to clean up the directory.

Now, netperf -V should then indicate the correctly installed version.

Netperf requires a client installation and a server installation. Hence, you may have to repeat the steps shown above on your second machine. Here for demonstration purposes I run the client and the server on the same machine. This is, of course, not very useful from a bufferbloat point of view.

Btw, I am looking for someone who has a netperf server running on the Internet so I can test my home network. I have a very basic hosting account at the moment, which does not give me ssh access.

Anyway, start the netperf server with:

# netserver

You can check the working installation with an example:

# netperf -P 0 -v 0 -D -0.2 -4 -H localhost -t UDP_RR -l 70

Then, obtain the netpef-wrapper:

# git clone git://github.com/tohojo/netperf-wrapper.git

The wrapper needs a Python graphical library (in case you haven’t installed already):

# apt-get install python-matplotlib

Then, it is time to run the netperf-wrapper by executing the netperf-wrapper file after you switched to the netperf-wrapper directory. You can also install the python files, if you like, with

# sudo python2 setup.py install

Finally, it is time to actually run the netperf-wrapper. In my test example the client and the server are on the same machine. You need to decide where the output of the test should go. It can be either displayed on the screen or dumped into a file. The first example shows the output to the screen:

# ./netperf-wrapper -H localhost -p ping_cdf rrul

Next, we dump the output into a file:

# ./netperf-wrapper -H localhost -p ping_cdf -o rrul-test-file.ps -f plot rrul

(The “-o” flag indicates that you want to produce a file output. If you don’t like Postscript then replace the suffix of the output file, for example to rrul-test-file.pdf)

The -p ping_cdf rrul parameter means the following:

* rrul is the name of test that is executed; there are further tests in the tests directory.

You can also enter the following command to see all the available tests:

# ./netperf-wrapper --list-tests

Here is the list of tests you should see with the fresh installation:

Netperf-wrapper: List available tests

* ping_cdf is the type of plot that is generated. Each test file also has information about the tests that are produced. If you look at the rrul.conf then you will find a number of different options for the plots under the PLOTS section. Alternatively, you can also run the following command to see the available plots for a given test.

# ./netperf-wrapper --list-plots <test_name>

For example, here is the list of plots:

netperf-wrapper: List available plots

An example output of a test may look like this:

Example netperf-wrapper test run

If you would like to perform a different analysis you can either switch to a different test, you could look for different plots, you could also add new test cases yourself. Finally, you can also use a statistic system like R to analyze the data yourself. This is fairly easy since the netperf-wrapper creates a JSON output of the test runs.

In any case double-check the results you had gotten because the graphics produce by the netperf-wrapper sometimes are a bit messed up. I suspect that there are some bugs in there.

Tags: bufferbloat, netperf, netperf-wrapper, network performance | Category: IETF | No Comments »

Wikipedia describes bufferbloat as

“a phenomenon in a packet-switched computer network whereby excess buffering of packets inside the network causes high latency and jitter, as well as reducing the overall network throughput. The phenomenon was detailed at least as far back as 1985, but gained more widespread attention starting in 2009.”

When the IAB and the IRTF organized the congestion control workshop for real-time interactive media in July 2012, Vancouver/Canada participants were investigating questions concerning congestion control algorithms of  real-time interactive services on the Internet. The ongoing standardization work on RTCWeb and its expected widespread deployment gave this topic an interesting spin. RTCWeb is an attempt to extend the capabilities of the Web platform, particularly using JavaScript APIs, to allow Web browsers to natively support browser-to-provider communication as well as other real-time communication features, i.e., functionality that was previously only possible with proprietary plug-ins (or with alternative communication architectures, like SIP or XMPP/Jingle).

As documented in the workshop report two solution directions were investigated, namely:

1. Define a congestion control mechanism that prevents problems with self-inflicting traffic. Following the workshop a new working group was created in the IETF, called “RTP Media Congestion Avoidance Techniques (rmcat)”.
2. Change network infrastructure equipment to prevent excessive buffering.

This write-up focuses on the he write-up mostly focused on #2. A certain amount of buffering is helpful to improve the efficiency of data transmissions to smoothen bursts in data transmissions. However, not dropping packets early enough in the event of congestion leads to increasing delays for interactive real-time communication since widely deployed congestion control algorithms rely on packet loss as a signal for congestion. Not dropping packets (or not in a timely fashion) prevents applications from reacting properly to overload. Real-time communication not only includes voice communication but also gaming, video conferencing, and Web browsing.

Since the congestion control workshop took place a number of activities had happened and there is an enormous amount of interest in detecting bufferbloat related problems as well as developing and deploying active queue management (AQM) techniques. In this short write-up I will provide some pointers to ongoing activities.

In the second part of 2012 the following events are worth pointing out:

• Linux 3.6 received some updates to reduce the buffer sizes in the networking stack, including the TCP small queues and Byte queue limits.
• As many researchers and engineers are trying to reproduce the tests various scripts have been made available to simulate the real-time response under load. One such script worth pointing out is the netperf-wrapper.
• Van Jacobson and Kathleen M. Nichols continued their work on the Controlled Delay (CoDel) Active Queue Management algorithm (slides from IETF#84, a recording, and a presentation to the BITAG are available).
• In addition to the CoDel AQM algorithm Cisco released their own AQM algorithm: it is called PIE.
• Various papers are available that provide different data points, including measurements of cellular providers. For example, Keith Weinstein submitted a position paper to the IAB/IRTF congestion control workshop and published a paper about their Alfalfa video conference system, which contains further information about the data collection in cellular networks. Even the raw data is available.
• A new European research project, called RITE – [Reducing Internet Transport Latency], started last autumn to investigate network buffering issues.
• Finally, it is worth pointing out that the stochastic fair queuing mechanism has gotten a lot of attention. Stochastic fair queuing puts flows into independent queues and therefore allows some degree of flow segregation without introducing the penalty of QoS signaling.

With the IETF#86 meeting in Orlando/Florida a number of sessions covered bufferbloat:

• Participants in the transport area discussed whether an AQM working group should be created.  In response to the discussion a new mailing list was setup.
• The IRTF ICCRG met and the discussions focused on updating RFC 3819 to provide more up-to-date advice on buffer management, on the state of the art buffer management in Linux, performance of the PIE AQM algorithm, and results from AQM performance in DOCSIS 3.0.
• Comcast showed a live demo of the impact of CoDel on gaming at the Bits&Bytes session.

In a nutshell, there is a lot of activity around the bufferbloat topic and with the increased awareness I expect further academic publications and also work continuing at the IETF.

Tags: bufferbloat, CoDel, congestion control, IAB, IETF, IRTF, rmcat, RTCWeb | Category: IETF, workshop | 2 Comments »

Standardization of IP-based emergency services systems is ongoing for more than 10 years and many of the specifications have been completed already in the IETF, 3GPP, NENA and EENA.

Nevertheless, new emergency services standardization efforts are born. One of these newly added activities is the work in the ETSI M493 group (which is now transitioned to the E2NA). You may not have heard about it. This is not your fault: information about the M493 group is hard to find in the Internet. This write-up aims to provide some insight.

The story starts in March 2011 when the EGEA COCOM group [0] met and discussed the current status of emergency services development in European member states. As an outcome of the meeting a new mandate, namely EC M/493 [1], was published.

In this mandate the European Commission asks European Standards Organizations to work on caller location standards for usage with emergency services in an IP environment. ETSI kindly accepted the mandate and created a new group – the M493 group.

Bundesnetz Agentur – the most recent M493 meeting location.

This group has now been working on a stage 2 document on a new emergency services architecture since early 2012. In a nutshell, the architectures describes how emergency calls are initiated from the end device, travel via the VoIP service provider (VSP), through the emergency services network to the PSAP. The VSP interacts with the access network provider to obtain a reference to location information and routing information. The VSP then uses this information to route the call to the PSAP (via a sort-of IP-based emergency services network). The PSAP or the emergency services network use the location reference to exchange it for location.

While there has been progress to develop this architecture document the group ran into a couple of problems. Here is a list of challenges as I see them.

0. A mandate is not a requirements document.

When the work in the M493 group was started the charter of the group was developed based on the text from the mandate. The members in the group had envisioned that the mandate would be used to shed light on specific protocol design requirements or that the European Commission would participate in the meetings to provide additional insight into their thinking.

Unfortunately, the European Commission does not participate in the work directly and attempts to solicit feedback naturally take time. An official letter with the questions has to be written and agreement within the group has to be found regarding the proper wording. Feedback is also not available immediately.

In retrospect it would have been more useful to develop a requirements document that can be discussed with the European Commission.

1. Location from the end host is considered harmful.

The M493 architecture assumes that location information comes from the access network provider only; information from the end device itself is untrusted. The concern is primarily driven by fears about security problems PSAP operators may encounter. The IETF ECRIT working group has worked on a document, called “Trustworthy Location” [2], that illustrates the problems. That document also points to the challenges with a pure location-based view and urges to look at the larger picture (and at other information associated with the call, such as identity information).

There is, however, a lot of missed opportunity here since there is a lot of development in the area of location based services at the end device (based on GPS integration into mobile phones but also based on third party location based services). These location-based services will not be re-usable in the M493 architecture as it currently seems.

The current architecture assumes that location is available for every IP caller everywhere from day one. There is no transition story for how to get to the final deployment stage.

It is also worth noting that location servers not only need to be available with every access network but also within enterprise networks. This is because the VSP uses the source IP address of the incoming call setup message to find a location server. When an end user utilizes a VPN then that lookup will resolve to the IP address obtained by the VPN and does not represent the IP address the end device has obtained from the access network provider. In addition to enterprise networks that are impacted any infrastructure provider who terminates tunnels (like IPv4-IPv6 transition gateways, VPNs, mobility anchor points, or TURN servers) is impacted. It is hard to say how large the number of impacted networks actually is.

In case you are hoping for a shift to better location for emergency services you may be disappointed. In Europe emergency services typically only have access to location at the granularity of cell ids (for cellular services). This is not likely going to change with this architecture either since the development or selection of location determination techniques are not scope of the group.

2. Focus on voice only.

The mandate by the European Commission puts a focus on voice services and the expertise in the group is on voice services only. Any considerations regarding multi-media emergency calling have been excluded and therefore the outcome may not work sufficiently well with media other than voice. This direction is also driven by the main proponents who have an integration with the current PSTN in mind, which does not support multi-media communication either.

3. Regulating VoIP providers is hard.

The current regulation on emergency services uses E.164 numbers to decide whether a company is covered by regulation or not. While this is a reasonable approach for past telecommunication infrastructure many communication services today do not use E.164 numbers. Consequently, they are not covered by regulation either. This regime seems to have been carried over to the IP world. Whenever the VSP is separate from the access network provider, which is the case for many Internet communication services and the VSP offers services outside their own country the situation gets challenging. How should the regulator impose obligations on VSPs that may essentially be located anywhere?

4. Operational complexity added.

While the M493 architecture does not require any location-to-service infrastructure like the IETF proposes with LoST it creates operational complexity.

First, the VSP is likely in the need to establish a relationship between the access network operator for the interaction with the location server. It is not likely that these location servers will be accessible for everyone on the Internet.

The next relationship that is requires is between the VSP and the emergency services provider. Here, it is also likely that a VoIP peering relationship needs to exist in order to route emergency calls from the VSP to the emergency services network. Unlike in many other architectures this emergency services network is envisioned to be operated by telecommunication operators, at least in the countries that participate in the work. This may allow a flow of money from the VSP to the provider of that emergency services network.

Finally, there is a relationship between the access network operator and the PSAP & emergency services network. This relationship is needed establish the necessary security infrastructure (such as the exchange of keys) for retrieval of location information.

5. Access Networks have no call visibility.

While the location retrieval and routing strategy follows the old telecommunication model the interaction between the end device and the VSP for emergency call handling is modern. Since one of the most popular example of an over-the-top VoIP provider is Skype and Skype uses a proprietary protocol the interaction between the end device and its VSP is left unspecified.

This also means that an emergency call itself is not understood by the access network and therefore no QoS treatment or preferential treatment of the call setup will be guaranteed for OTT providers. This also means that telecommunication operators that block certain VoIP services, or execute traffic management practices may accidentally block emergency calls or make them fail in other ways.

6. Current deployments respected.

The mandate points to the importance of considering current deployments and to consider them in the design.

While this is clearly a good idea it leads to various challenges. First, since almost no emergency services authorities are present in the discussion it is almost impossible to argue how the current emergency services deployment in Europe (from a technical point of view) looks like.

Even more important than the current deployment situation is the vision for the future since the transition to IP allows a number of short-comings of the current system to be corrected. Also in terms of finance it would be useful to make changes. In most cases emergency services authorities know about the deployment status and barely anyone else.

Consequently, the current deployment situation is mostly guessed (if considered at all).

7. Stakeholders missing.

The most severe aspect of the entire work is the lack of participation from a large part of the stakeholder community. While the European Commission may have hoped to involve more European stakeholders in the conversation it unfortunately turned out to be far less than available in other forums. This may have many reasons, including the cost of the participation in ETSI, lack of interest, recognition of ETSI in the standardization world, or the missing outreach.

In particular, the following stakeholders are largely absent:

• Emergency services authorities: For most meetings no emergency services authorities are present.
• Emergency services equipment vendors: Only telecommunication equipment manufacturers are present.
• VoIP providers, and OTT players in general: Only Microsoft is present.
• Regulators: only two regulators participate in the work.
• Enterprise networks. No enterprise network operators are interested in the work.
• ISPs

References

[0] EGEA COCOM, available at http://circa.europa.eu/Public/irc/infso/egea/home
[1] EC Mandate 493, available at http://www.etsi.org/images/files/ECMandates/m493.pdf
[2] H. Tschofenig, et al, “Trustworthy Location”, available at http://tools.ietf.org/html/draft-ietf-ecrit-trustworthy-location

Tags: Challenges, ETSI, European Commission, IETF, Location, M493, mandate, Standardization | Category: emergency services | No Comments »

When designing new protocols and technologies it is essentially to get the starting point right – the assumptions of what technology can be used and what cannot be used.

I just saw a great post by Patrik Fältström on the Apps-discuss IETF mailing list. Here is what he said about the design choices for WebFinger:

‘I think it is fascinating that people that love new things like “the web” and new HTML5 features are the most conservative ones regarding other protocols like DNS.
With that attitude, we have no evolution, and no innovation.’

I had shared my views regarding WebFinger already and it is sad to see that many ignore the negative privacy aspect in the design.

Category: Identity Management, IETF, OAuth, Privacy | No Comments »