Now that the bits are getting themselves flowing in the right direction at Spark, its probably a good time to make some observations.

The first one is that this is the second major Spark/Telecom ‘outage’ media storm that I have covered from the TUANZ perspective (the first was the XT meltdown) and there have been a number of similarities and some major differences. 

I think Spark have an issue in knowing when an issue has arisen, and I think this is down to a couple of things, the first is the sheer size of the beast and the second is the distance between outsourced, call centre based customer service and senior management. 

I know its unfair but i have an image in my mind of a lumbering ‘brontosaurus’ with its tail on fire and it taking a while for the distant head to identify the smell, then look back to see the flames and then finally alter course to find some water to put it out. 

Once they knew they had a problem Spark were faced with three challenges, solving the problem, dealing with customers and dealing with the media. 

The first problem has been aptly described as ‘whack a mole’  at the same time as they trying to find a needle in a haystack. I have the utmost respect for the systems administrators and cyber-security team who did the actual digital fire firefighting, I hope they’re enjoying some well deserved rest and are in line for preformance bonuses.

The next issue is now really hard, customers now rely on the internet being always on and complain loudly when it isn’t. If this had happened ten years ago it wouldn’t have been as big a deal, but now it ranks just below a power cut in terms of disruption and inconvenience.

What’s even more difficult is that it becomes hard to keep your customers up to date when the phones are running red hot and they can’t get on-line to find out why they’re not on-line. Everyone assumes that the problem is isolated in the first instance and just relates to them.

The only disinfectant for confusion and hearsay is the truth or at least as much truth as you know and to Sparks credit they had their comms team on the case on Saturday, I know this because I got caught up in phase three when the media were looking for explanations as to what was going on. 

It was at about this time that the ‘fog of war’ descended and people were coming up with explanations about ‘malware’, ‘cyber crime’ and why just Spark?. The media were looking for newsworthy angles and good stories and by Monday it had turned juicy, the malware was introduced by customers downloading a ‘viewer’ allowing to see the infamous ‘hacked’ Jennifer Lawrence intimate ‘selfies’.  

The story had it all now but still didn’t feel right, so it was with relief that I read this piece on Stuff this morning. 

Mundane as it seems that does make a lot more sense, there is still a ‘cyber warfare‘ angle but it seems Spark were mere pawns in a much bigger game. 

So what lessons have we learnt? 

 Spark need to run a 24/7 NOC that is pretty pro-active in getting alerts out to customers

Spark need a non-internet based method of alerting customers to outages

Spark need to get the media on side fast

Spark have done well sharing details once they are known

Spark need to pay more attention to customers ‘edge’ devices and possibly manage them remotely

I hope this helps. 

I’m not sure too many people are going to vote this year solely based on ICT policies, but if you are involved in the sector its worth being across what’s on offer and it might affect you or your business.

It’s been such a weird campaign so far as policy debate is a distant second from the ‘dirty politics’ saga and the general decline of political commentary into the realm of reality TV and sports reporting. 

So I’ve been really impressed with the work being done by NZ Rise to compare, contrast and question the various parties about their ICT policies. 

NZ Rise if you don’t know them yet, are the collective body of “NZ Owned Digital Technology Companies’ and they represent NZ’s genuine indigenous ICT industry.

Here’s what Don Christie, the chair of NZ Rise had to say about the project so far: 

I think it is fair to say that we have had comprehensive and serious
responses from across the political spectrum and from all the major
parties. This does show how the awareness of the importance of the NZ
digital sector has grown over the last three years.

Many of the policies espoused are ones that NZ digital companies will
welcome. They form the core of the platform that NZRise as been advocating
for over the years; growing the digital technology sector, promoting open standards, supporting local industry through government procurement, and bringing new digital talent into the workforce. We are pleased to see discussion on the role of Government wide technology strategy and upskilling of government branches.

We welcome the emphasis from the Māori Party on engaging Māori with ICT,
particularly in education.

Similarly the procurement policies of the Labour and the Greens
recognise the potential for Government spending to be a real driver of
digital success and future exports for Kiwi owned businesses. The
National Parties support for open standards and open source software is
reflected by others but also recognises the need for government systems
to be able to interoperate freely and avoid capture by old technology
and individual vendors.

Most importantly we see real focus on the need to educate the next
generation of digital technologist, to be active creators and great
products and services rather than mere consumers.

NZRise recognises there is much work to ensure that the promise of these
policies is realised in practice, to the advantage of the NZ economy,
society and business community. But what we have seen in these responses
gives us real confidence that progress is being made.

NZRise will continue to work with the incoming Government to provide updates on developments in digital technology and the needs of our members.

There’s not a lot there that I can disagree with from a TUANZ perspective, it is good to see most of our political parties accept the key role digital infrastructure and technology will play in NZ’s future. 

If there is one policy that I’d like to see implemented it is the idea of having a CTO (Chief Technology Officer) for NZ that reports directly to the Prime Minister and cabinet. This was rebuffed by Steven Joyce as being unnecessary because we already have the GCIO (Government Chief Information Officer) working in the DIA.

Sadly this misses the point, the GCIO has an internal focus on government ICT projects, the focus of the CTO would be external and look at NZ’s overall digital development. 

The NZ Rise project is dynamic and it is being updated, do you think they’ve missed anything? 

One of the fun things about TUANZ is you never know where the next media storm will come from, on Saturday I was in Wellington talking to the real heroes of NZ rural broadband – the regional wireless ISP’s (more about them later)

Here’s the latest update from Spark: 

Cyber criminals based overseas appear to have been attacking web addresses in Eastern Europe, and were bouncing the traffic off Spark customer connections, in what is known as a distributed denial of service (DDoS) attack.

The DDoS attack was dynamic, predominantly taking the shape of an ‘amplified DNS attack’ which means an extremely high number of connection requests – in the order of thousands per second – were being sent to a number of overseas web addresses with the intention of overwhelming and crashing them.  Each of these requests, as it passes through our network, queries our DNS server before it passes on – so our servers were bearing the full brunt of the attack.

While the Spark network did not crash, we did experience extremely high traffic loads hitting our DNS servers which meant many customers had either slow or at times no connectivity (as their requests were timing out). There were multiple attacks, which were dynamic in nature.  They began on Friday night, subsided, and then began again early Saturday, continuing over the day.  By early Sunday morning traffic levels were back to normal and have remained so since. We did see the nature of the attack evolve over the period, possibly due to the cyber criminals monitoring our response and modifying their attack to circumvent our mitigation measures – in a classic ‘whack a mole’ scenario.

Its good to see Spark coming out with this statement as it deals with a lot of the ‘chatter’ that is out there, one question has been top of mind for a lot of people – Why just Spark? 

Here’s their reply: 

Why only Spark?

We can’t say what other networks experienced.  However, cyber criminals often look for clusters of IP addresses to use in any particular DDoS attack.  That makes it more likely that these IP addresses belong to the customers of a single ISP – even more likely with a large ISP like Spark.

How did they get access through the Spark Network?

Since the attacks began we have had people working 24/7 to identify the root causes, alongside working to get service back to normal. During the attack, we observed that a small number of customer connections were involved in generating the vast majority of the traffic. This was consistent with customers having malware on their devices and the timing coincided with other DNS activity related to malware in other parts of the world. 

However, while we’re not ruling out malware as a factor, we have also identified that cyber criminals have been accessing vulnerable customer modems on our network. These modems have been identified as having “open DNS resolver” functionality, which means they can be used to carry out internet requests for anyone on the internet. This makes it easier for cyber criminals to ‘bounce’ an internet request off them (making it appear that the NZ modem was making the request, whereas it actually originates from an overseas source). Most of these modems were not supplied by Spark and tend to be older or lower-end modems.

What remains clear is that good end user security remains an important way to combat these attacks. With the proliferation of devices in households, that means both the security within your device and the security of your modem.

What did Spark do?

We have now disconnected those modems from our network and are contacting all the affected customers. We have also taken steps at a network level to mitigate this modem vulnerability.  We are now in the process of scanning our entire broadband customer base to identify any other customers who may be using modems with similar vulnerabilities and will be contacting those identified customers in due course to advise them on what they should do.

With respect to malware we continue to strongly encourage our customers to keep their internet device security up to date, conduct regular scans and regularly update the operating software and firmware on their home network. We also continue to advise customers not to click on suspicious links or download files when they are not sure of the contents.  

We have also taken steps at the network level to make it more difficult for cyber criminals to exploit the DNS open resolver modem vulnerability and we’re using the latest technology to strengthen our network monitoring and management capabilities. For security reasons we can’t detail these steps, however this is an ongoing battle to stay one step ahead of cyber criminals who are continually using more and more sophisticated tactics.

I spend quite a bit of time watching various technology stories emerge and I’ve learnt that you need to develop a couple of skills if you are going to maintain both your focus and sanity.

It is very easy to believe everything that you read and soon you come to realise that:

a.   A lot of the breathlessly announced new technologies never see the light of day

b.   A lot of progress is incremental and each subsequent release brings an increasing sense of disappointment as products and services approach the boring nirvana of some kind of long term ‘steady state’ existence

c.   How do you pick the winners?

There are a couple of models that I have found really useful over the years, the first is Geoffrey Moore’s ‘Crossing the Chasm’ model which helps to explain why so many products never make it beyond the early adopters and then I came across the Gartner Hype Cycle which I came across through my long term addiction to Wired magazine.

So I’m very curious to what the 2014 edition of the hype cycle contains:

The 2014 Gartner emerging technology Hype Cycle graph

Gartner have actually analysed over 2000 emerging technologies and placed them on this curve and they’ve been doing it for over 20 years so they have pretty unique depth in this space.

I love this model for several reasons, one is that it makes sorting the wheat from the chaff a lot easier, the second is that I now cynically wonder what the real purpose of a lot of announcements truly is. You know that they are usually about money, When concepts are at the ‘innovative trigger’ stage it can be about research grants, then attracting angel investors and venture capital, once the delightfully named ‘peak of inflated expectations’ is reached its often about attracting investors to an IPO or increasingly now part of a ‘crowdsourcing’ push this is followed by the aptly named ‘trough of disillusionment’ where it maybe about keeping grumpy investors or bankers on side long enough to reach the golden shores of the ‘slope of enlightenment’ which lead to the ‘plateau of productivity’ and its then our money that they’re after as consumers.

So what do you think of their picks?

Well I’ve had a night to sleep on yesterday’s announcement and I’m still pretty excited, its a clever piece of policy work, by extending the TDL (Telecommunications Development Levy – successor to the much derided TSO) the Government has given itself an extra $150 Million to play with that is fiscally neutral (I think they’ll be hoping that the Telco’s paying the TDL won’t notice that they’re going to keep paying a wee bit longer.

Politically its very smart as it does 3 things, firstly it gazzumps Labour’s $9.6 million contestable fund for very similar rural initiatives, secondly it deals with genuine heartland concern about broadband and mobile coverage and finally it is actually quite ambitious for rural New Zealand.

To be really successful it needs a target and I think it needs to be an ambitious target – funding projects that will have the capability within 10 years to deliver 1 gigabit/second to rural New Zealand.

Thats not a pipe dream. it can be done today with 2 technologies fibre and point to point wireless and by 2024 5G should be gigabit capable as well. 

As a former rural New Zealander (I’m just currently an economic exile in #gigatownporirua) one thing that really pisses me off is hearing that old technology is good enough for country folk. Its up there with the now 30 year old refrain about farming being a sunset industry, I’m glad its such a long sunset as without the ‘white gold’ boom we would be in a far worse position.

Farming is developing into a very sophisticated data intensive business, it is the starting point for the worlds food based supply chain and consumers increasing demand for more detail about their food & where it comes from. But not only that, rural New Zealanders are entitled to all the good stuff that comes with ultra fast internet, tomorrow see’s the launch of Spark’s ‘Lightbox’ service and I’m sure plenty of rural Kiwi’s are up for a bit of ‘Global mode’ too.

The devil as they say is in the detail and it will be the same with yesterdays announcement, there are many questions to be asked and a lot of capacity to be rebuilt. In its first broadband schemes the Government favoured large monolithic structures run by contract, in fact it believed that contracts would work better than regulation.

Councils and small regional operators were sidelined by both the UFB and the RBI and massive waste and overbuild has occurred, the situation in Palmerston North is a prime example where pioneering work was done by James Watts and his team at inspire.net and he had an effective partnership going with the Palmerston North City Council, James was even running fibre to the farm 7 years ago.

Now the Government recognises that local is often best and will need to re-engage players who still have much to offer. This will require both sides to re-establish dormant relationships and a number of plans can be dusted off and updated.

Who will run the new scheme? is it best suited to MBIE or could it function out of CFH with an expanded mandate? Nearly all of the old team at MED who used to run these projects have moved on or retired.

I think it could well take several years to get back in shape for this challenge, the current RBI advisory structure needs to be revisited and updated to provide genuine governance and oversight.

We also need to make sure that it remains consistent with our market structure, that is that new access networks are structurally separated and open access and the the correct regulatory settings are applied.

I’ll leave with the insights I’ve just gained from the engineer behind the very successful Northpower fibre rollout.