A major contributor to this article appears to have a close connection with its subject. (February 2011) |
Endace Ltd is a privately owned network monitoring company, based in New Zealand and founded in 2001.[1] It provides network visibility and network recording products to large organizations. The company was listed on the London Stock Exchange in 2005 and then delisted in 2013 when it was acquired by Emulex.[2] In 2016 Endace was spun out of Emulex and is currently a private company.[3]
 | |
| Company type | Private |
|---|---|
| Industry | Network monitoring |
| Founded | 2001 |
| Headquarters | Auckland, |
Key people | Stuart Wilson: CEO |
| Website | www |
In October 2016, The Intercept revealed that some Endace clients were intelligence agencies, including the British GCHQ (known for conducting massive surveillance on network communications) and the Moroccan DGST, likewise known for mass surveillance of its citizens.
Background and history
editEndace was founded after the DAG project at the School of Computing and Mathematical Sciences at the University of Waikato in New Zealand.[1][4] The first cards designed at the university were intended to measure latency in ATM networks.[5]
In 2006, Endace transitioned from component manufacturer to appliance manufacturer to managed infrastructure provider. The company now sells network visibility fabrics, based on its range of network recorders, to large corporations and government agencies.[6]
Endace was the first New Zealand company to list on London's Alternative Investment Market when it floated in mid-June 2005[7] a move which was not without controversy.[8] Poor share price performance in the early years and a seeming failure to attract a broad enough shareholder base lent weight to the criticism that Endace should have focused initially on developing its local profile (via NZX) rather than pushing for overseas investment (via London AIM).
Endace is headquartered in Auckland, New Zealand, and has an R&D centre in Hamilton, New Zealand, and offices in Australia, United States and Great Britain.
Key innovations of the DAG
editThe DAG project grew from academic research at Waikato University. Having found that software measurements of ATM cells (or packets) were unsatisfactory, both for reasons of accuracy and lack of certainty about packet loss, the research group set about developing their own hardware to generate better quality recordings.[5] This hardware and its subsequent iterations introduced two fundamental innovations: hardware timestamping and hardware accounting for packet loss.
Hardware timestamping
editConventionally, each packet or cell is given a timestamp by the host machine's kernel (i.e. in software) when the kernel driver is notified that a new packet has arrived. This approach results in poor quality timestamps for several reasons, among them the considerable latency and jitter between the packet arriving at the network interface and receipt by the kernel driver and uncertainty caused by interrupt coalescing wherein one host interrupt signifies the arrival of several packets. Such poor quality limits what research can usefully be done on network performance and related fields.
To solve this, the DAG generates timestamps in the hardware as close to the network interface as possible. Not only does this obviate latency, jitter and problems caused by interrupt coalescing, the hardware is capable of much greater accuracy and precision than software-generated timestamps. Precision comes from the freedom of custom hardware to assign as many bits to the timestamp as required and accuracy is assured by reference to an external time source such as GPS which is accurate to ± 40 nanoseconds.[9] In contrast, the accuracy of NTP (by which kernel clocks can be corrected over the Internet) is in the order of milliseconds (about 100,000 times less accurate), depending on the conditions involved.
The DAG produces 64 bit timestamps in fixed-point format with 32 fractional bits, giving a potential precision of seconds or 233 picoseconds. The actual precision offered varies with the particular model of DAG, the oldest giving 24 fractional bits (60 nanoseconds) and better precisions offered in DAGs for higher bandwidth networks.[10]
The timestamp is derived from a free-running clock provided by a crystal oscillator but the accuracy of crystals drift with both temperature and age. The DAG's solution is to use direct digital synthesis using the 1 Hz pulse-per-second output that many GPS receivers provide as its reference clock. This mechanism is described in §5.5.3 of Stephen Donnelly's PhD thesis[11] which also describes in detail the pre-commercial era models of DAG.
Crucially, and an academically significant contribution of the DAG, the ability to use an external reference such as globally synchronised GPS makes it possible to do one-way time-of-flight measurements. This is of immense interest to academic researchers because packets flowing between two points on the Internet are neither guaranteed to follow the same path in each direction nor guaranteed to have the same timing characteristics in each direction.
Outside of the academic world, timestamp accuracy has commercial applications in the enforcement and compliance with law such as the EU Markets in Financial Instruments Directive 2004.
Packet loss
editAlmost as important as timestamp accuracy is guaranteeing 100% cell or packet capture and, where loss is unavoidable, knowing not only that packets have been lost but where. The "where" is important because, when analysing a packet trace, it's important to be able to compensate for lost packets when calculating inter-arrival times.
Most commercial NICs keep a count of dropped packets, but they can't indicate where packets were lost. The DAG prepends a header[12] which, amongst other things, indicates how many packets were dropped between that packet and the previously accepted packet.
The DAG is also engineered to deliver recorded packets to the host with the greatest possible efficiency. That, together with the interstitial loss counter, is what makes the DAG so appealing for surveillance applications. The interstitial loss counter also finds application in forensics; a prosecutor needs to be able to prove that the record is complete or, if it is not, where it is not.
Controversy and surveillance
editIn October 2016, The Intercept published an article showing that Endace customers include intelligence agencies, including the GCHQ, Canadian and Australian intelligence agencies, and the DGST (Morocco's domestic surveillance agency).[13] Edward Snowden documents have shown that the GCHQ has installed massive surveillance of network communications in UK, using the over-sea cable between Europe and North America.
References
edit- ^ a b "The DAG Project". Archived from the original on 29 November 2001.
- ^ "ENDACE LTD (EDA:NL): Company Description - BusinessWeek". Bloomberg Businessweek investing database. Bloomberg L.P. Archived from the original on 10 October 2012. Retrieved 9 February 2011.
- ^ "Endace Spins off from Emulex in Management-led Buyout". New Zealand: Endace. 10 March 2016. Retrieved 13 March 2016.
- ^ "Yoke Har Lee: Life's a bit of a DAG for hi-tech firm". The New Zealand Herald. 24 August 2009. Retrieved 11 September 2011.
- ^ a b Cleary, John; Donnelly, Stephen; Graham, Ian; McGregor, Anthony; Pearson, Murray. Design Principles for Accurate Passive Measurement (PDF) (Report). Waikato University. Retrieved 13 May 2017.
- ^ "What is a managed service provider? Strategic outsourcing for IT services". CIO. Retrieved 9 August 2024.
- ^ "Growth Business: Endace poised to take AIM". Archived from the original on 19 November 2005.
- ^ Inder, Richard (5 June 2006). "Endace's performance on UK AIM listing gives fuel to critics". The New Zealand Herald. Retrieved 11 September 2011.
- ^ "§A.4.8 UTC(USNO) Offset Accuracy". Global Positioning System Standard Positioning Service Performance Standard (PDF) (Report) (4th ed.). US Department of Defense. September 2008. p. A-16. Retrieved 13 May 2017.
- ^ Micheel, Jörg; Donnelly, Stephen; Graham, Ian (2001). "Precision timestepping of network packets" (PDF). Proceedings of the First ACM SIGCOMM Workshop on Internet Measurement – IMW '01. Waikato University. p. 273. doi:10.1145/505202.505236. ISBN 1581134355. S2CID 14567389. Archived from the original (PDF) on 25 February 2018. Retrieved 13 May 2017.
- ^ Donnelly, Stephen F. (2002). High Precision Timing in Passive Measurements of Data Networks (PhD). CiteSeerX 10.1.1.136.1730.
- ^ ""Extensible Record Format" header description". WireShark. Retrieved 13 May 2017.
- ^ "The Little-Known Company That Enables Worldwide Mass Surveillance". The Intercept. 23 October 2016. Retrieved 2 November 2016.