In business sticking to your knitting is usually regarded as a virtue, that is unless your name is Apple in which case you’re just about obliged to go and wreck beautifully designed carnage in other peoples piles of yarn.

I was reminded of this adage this week as I watched the Apple keynote for the second time (sad I know) before returning to playing with Spark’s new ‘Lightbox‘ service. While I was digesting the Apple announcements and browsing the Lightbox catalogue I got to thinking about the challenge Spark are facing .

Spark are trying to get a slice of the on-demand video pie, money thats currently either going locally to Sky or Quikflix or internationally to Netflix or Hulu. We know the network is capable of supporting video streaming at its current levels and for most of us it will only get better with the UFB.

Lightbox is ok but not $15 a month kind of special, running it on my iPad through my Apple TV works fine but it locks up two devices, its very easy to accidentally stop your video and you can’t save your place to start again, I can get more content for the same price on Quikflix and if I was prepared to go down the ‘Global Mode‘ or ‘unblock.us‘ path I can get the real thing, plus if I do that my Apple TV becomes a whole lot more useful.

Then I saw an ad for Spark’s new mobile payment service, again it looks solid, it is competing with the banks and OTT services like ‘square’ and ‘paypal‘. But now that Tim Cook has announced Apple Pay the game has changed, because I already have an iTunes account, I am already able to use Apple Pay whenever it launches in New Zealand.

And there is the dilemma for Spark, whilst they are dominant locally they don’t have the reach or the financial clout to go and play with other peoples yarn so they may need to focus on making their network the best it can be, launching innovative services and making sure that OTT works best on the Spark network for Spark customers.

Maybe instead of putting the money into Lightbox they’d have been better of creating their own version of Global Mode, it would have stopped customers changing their DNS settings and we know what that led to.

Oh and they also need to keep an eye on that Mr Cook and his fruit company, because a minor point in the presentation on IPhone 6 is a new technology to allow cellular calls on wifi networks.

Now that the bits are getting themselves flowing in the right direction at Spark, its probably a good time to make some observations.

The first one is that this is the second major Spark/Telecom ‘outage’ media storm that I have covered from the TUANZ perspective (the first was the XT meltdown) and there have been a number of similarities and some major differences. 

I think Spark have an issue in knowing when an issue has arisen, and I think this is down to a couple of things, the first is the sheer size of the beast and the second is the distance between outsourced, call centre based customer service and senior management. 

I know its unfair but i have an image in my mind of a lumbering ‘brontosaurus’ with its tail on fire and it taking a while for the distant head to identify the smell, then look back to see the flames and then finally alter course to find some water to put it out. 

Once they knew they had a problem Spark were faced with three challenges, solving the problem, dealing with customers and dealing with the media. 

The first problem has been aptly described as ‘whack a mole’  at the same time as they trying to find a needle in a haystack. I have the utmost respect for the systems administrators and cyber-security team who did the actual digital fire firefighting, I hope they’re enjoying some well deserved rest and are in line for preformance bonuses.

The next issue is now really hard, customers now rely on the internet being always on and complain loudly when it isn’t. If this had happened ten years ago it wouldn’t have been as big a deal, but now it ranks just below a power cut in terms of disruption and inconvenience.

What’s even more difficult is that it becomes hard to keep your customers up to date when the phones are running red hot and they can’t get on-line to find out why they’re not on-line. Everyone assumes that the problem is isolated in the first instance and just relates to them.

The only disinfectant for confusion and hearsay is the truth or at least as much truth as you know and to Sparks credit they had their comms team on the case on Saturday, I know this because I got caught up in phase three when the media were looking for explanations as to what was going on. 

It was at about this time that the ‘fog of war’ descended and people were coming up with explanations about ‘malware’, ‘cyber crime’ and why just Spark?. The media were looking for newsworthy angles and good stories and by Monday it had turned juicy, the malware was introduced by customers downloading a ‘viewer’ allowing to see the infamous ‘hacked’ Jennifer Lawrence intimate ‘selfies’.  

The story had it all now but still didn’t feel right, so it was with relief that I read this piece on Stuff this morning. 

Mundane as it seems that does make a lot more sense, there is still a ‘cyber warfare‘ angle but it seems Spark were mere pawns in a much bigger game. 

So what lessons have we learnt? 

 Spark need to run a 24/7 NOC that is pretty pro-active in getting alerts out to customers

Spark need a non-internet based method of alerting customers to outages

Spark need to get the media on side fast

Spark have done well sharing details once they are known

Spark need to pay more attention to customers ‘edge’ devices and possibly manage them remotely

I hope this helps. 

I’m not sure too many people are going to vote this year solely based on ICT policies, but if you are involved in the sector its worth being across what’s on offer and it might affect you or your business.

It’s been such a weird campaign so far as policy debate is a distant second from the ‘dirty politics’ saga and the general decline of political commentary into the realm of reality TV and sports reporting. 

So I’ve been really impressed with the work being done by NZ Rise to compare, contrast and question the various parties about their ICT policies. 

NZ Rise if you don’t know them yet, are the collective body of “NZ Owned Digital Technology Companies’ and they represent NZ’s genuine indigenous ICT industry.

Here’s what Don Christie, the chair of NZ Rise had to say about the project so far: 

I think it is fair to say that we have had comprehensive and serious
responses from across the political spectrum and from all the major
parties. This does show how the awareness of the importance of the NZ
digital sector has grown over the last three years.

Many of the policies espoused are ones that NZ digital companies will
welcome. They form the core of the platform that NZRise as been advocating
for over the years; growing the digital technology sector, promoting open standards, supporting local industry through government procurement, and bringing new digital talent into the workforce. We are pleased to see discussion on the role of Government wide technology strategy and upskilling of government branches.

We welcome the emphasis from the Māori Party on engaging Māori with ICT,
particularly in education.

Similarly the procurement policies of the Labour and the Greens
recognise the potential for Government spending to be a real driver of
digital success and future exports for Kiwi owned businesses. The
National Parties support for open standards and open source software is
reflected by others but also recognises the need for government systems
to be able to interoperate freely and avoid capture by old technology
and individual vendors.

Most importantly we see real focus on the need to educate the next
generation of digital technologist, to be active creators and great
products and services rather than mere consumers.

NZRise recognises there is much work to ensure that the promise of these
policies is realised in practice, to the advantage of the NZ economy,
society and business community. But what we have seen in these responses
gives us real confidence that progress is being made.

NZRise will continue to work with the incoming Government to provide updates on developments in digital technology and the needs of our members.

There’s not a lot there that I can disagree with from a TUANZ perspective, it is good to see most of our political parties accept the key role digital infrastructure and technology will play in NZ’s future. 

If there is one policy that I’d like to see implemented it is the idea of having a CTO (Chief Technology Officer) for NZ that reports directly to the Prime Minister and cabinet. This was rebuffed by Steven Joyce as being unnecessary because we already have the GCIO (Government Chief Information Officer) working in the DIA.

Sadly this misses the point, the GCIO has an internal focus on government ICT projects, the focus of the CTO would be external and look at NZ’s overall digital development. 

The NZ Rise project is dynamic and it is being updated, do you think they’ve missed anything? 

One of the fun things about TUANZ is you never know where the next media storm will come from, on Saturday I was in Wellington talking to the real heroes of NZ rural broadband – the regional wireless ISP’s (more about them later)

Here’s the latest update from Spark: 

Cyber criminals based overseas appear to have been attacking web addresses in Eastern Europe, and were bouncing the traffic off Spark customer connections, in what is known as a distributed denial of service (DDoS) attack.

The DDoS attack was dynamic, predominantly taking the shape of an ‘amplified DNS attack’ which means an extremely high number of connection requests – in the order of thousands per second – were being sent to a number of overseas web addresses with the intention of overwhelming and crashing them.  Each of these requests, as it passes through our network, queries our DNS server before it passes on – so our servers were bearing the full brunt of the attack.

While the Spark network did not crash, we did experience extremely high traffic loads hitting our DNS servers which meant many customers had either slow or at times no connectivity (as their requests were timing out). There were multiple attacks, which were dynamic in nature.  They began on Friday night, subsided, and then began again early Saturday, continuing over the day.  By early Sunday morning traffic levels were back to normal and have remained so since. We did see the nature of the attack evolve over the period, possibly due to the cyber criminals monitoring our response and modifying their attack to circumvent our mitigation measures – in a classic ‘whack a mole’ scenario.

Its good to see Spark coming out with this statement as it deals with a lot of the ‘chatter’ that is out there, one question has been top of mind for a lot of people – Why just Spark? 

Here’s their reply: 

Why only Spark?

We can’t say what other networks experienced.  However, cyber criminals often look for clusters of IP addresses to use in any particular DDoS attack.  That makes it more likely that these IP addresses belong to the customers of a single ISP – even more likely with a large ISP like Spark.

How did they get access through the Spark Network?

Since the attacks began we have had people working 24/7 to identify the root causes, alongside working to get service back to normal. During the attack, we observed that a small number of customer connections were involved in generating the vast majority of the traffic. This was consistent with customers having malware on their devices and the timing coincided with other DNS activity related to malware in other parts of the world. 

However, while we’re not ruling out malware as a factor, we have also identified that cyber criminals have been accessing vulnerable customer modems on our network. These modems have been identified as having “open DNS resolver” functionality, which means they can be used to carry out internet requests for anyone on the internet. This makes it easier for cyber criminals to ‘bounce’ an internet request off them (making it appear that the NZ modem was making the request, whereas it actually originates from an overseas source). Most of these modems were not supplied by Spark and tend to be older or lower-end modems.

What remains clear is that good end user security remains an important way to combat these attacks. With the proliferation of devices in households, that means both the security within your device and the security of your modem.

What did Spark do?

We have now disconnected those modems from our network and are contacting all the affected customers. We have also taken steps at a network level to mitigate this modem vulnerability.  We are now in the process of scanning our entire broadband customer base to identify any other customers who may be using modems with similar vulnerabilities and will be contacting those identified customers in due course to advise them on what they should do.

With respect to malware we continue to strongly encourage our customers to keep their internet device security up to date, conduct regular scans and regularly update the operating software and firmware on their home network. We also continue to advise customers not to click on suspicious links or download files when they are not sure of the contents.  

We have also taken steps at the network level to make it more difficult for cyber criminals to exploit the DNS open resolver modem vulnerability and we’re using the latest technology to strengthen our network monitoring and management capabilities. For security reasons we can’t detail these steps, however this is an ongoing battle to stay one step ahead of cyber criminals who are continually using more and more sophisticated tactics.