Data Scientists – building open data capability

Following on from the previous post, an emerging community of “farmers” are sharing practices and constructing a series of helpful guides on how to approach data management.  If you want to move beyond being a hunter gatherer of data, you should look at the guides that are under construction, and contribute your experience to the community. Here are three sources that you may find useful.


Semantic Community is a wiki dedicated to using and promoting Data Science - “It is not just where you put your data (cloud), but how you put it there." A good entry point is the section on free data visualisation tools. I was fortunate that one of the drivers behind the Semantic Community, Brand Niemann, was willing to be online at 4am to deliver a presentation to a W3C egov conference call, which helped me connect to this rich data source.


The Data Wrangling handbook is a crowdsourced “textbook” from the School of Data, supported by the good folk at the Open Knowledge Foundation. The OKFN blog last month published Managing Expectations  by Rufus Pollock which described the long term evolution of open knowledge; it promised to be the first of two posts, so watch out for the sequel.

The Guardian data blog has been doing some great work on visualising data about the Olympics over the last two week. Last year Tim O’Reilly wrote a short piece for Forbes on the topic of the “World’s 7 Most Powerful Data Scientists." More interesting than the fact that the list actually contains ten names, is the fact that they are all from the USA – just like "World Series" baseball. In the w3c discussion, Brand Niemann confirmed my view that the Guardian data blog is leading the application of data science to data journalism; maybe Simon Rogers should be at the top of the list.

The anthropology of Open Government Data – moving beyond Hunter-Gatherer

RAW DATA NOW was the rallying cry issued by Rufus Pollock from the Open Knowledge Foundation in November 2007. Sir Tim Berners-Lee picked up the call in his landmark TED talk from February 2009, and now, nearly five years on, Open Government and Open Data have become part of government operations for many countries around the world.

In this post, I propose that we are still at an early stage of Open Government Data, and use the stages of evolution of our species as a framework for thinking about the future of Open Data.

Hunter Gatherer

For over 100,000 years, homo sapiens was a Hunter Gatherer and generally nomadic, hunting and foraging for food and moving constantly in the search for sustenance.

The open data community is essentially a hunter-gatherer world – finding food (data) and providing it to our families in the best way possible.  The tribes have ways of sharing information on where good food can be found (#opendata on Twitter is a good source), but in some terrains (governments) food is hard to find, and it takes skill, experience, and cunning to be an effective data hunter.

Fortunately hunters are willing to share their findings, and provide signposts to help hunters find easy to gather food, although a lot of it has tough skin (pdf) and is of low (calorific) value. 

Maps are emerging, but are not authoritative. Interoperability is sharing information on the design of the bow and arrow, through channels such as on Scraperwiki and G_Refine.

Agriculture

Humans first began the systematic cultivation of plants and animals between 7,000 and 10,000 years ago, and the relative security provided by agriculture provided the incentive for most humans to live as farmers in permanent settlements.

The idea of farming and harvesting data is emerging in a few areas; two notable examples are OpenCorporates and the World Government Data Store, where farmers have planted crops from many terrains in one place.

The emergence of data geo-coding and Spatial Data Infrastructure initiatives suggest that more facilities will be available to support agriculture.

Cities, states and empires

The next phase of anthropological evolution saw the establishment of governments, complex economic and social structures with increasing specialisation, sophisticated language and writing systems, and distinct cultures and religions.  The rise and fall of these cities, states and empires has happened across the world for the last 2,000 years.

Open Government Data has not yet moved into this stage, although some people are thinking what it might bring. Government is the world’s largest information business – from global organisations such as the UN to national, regional and local governments.  How will the supply chain change as the internet deconstructs the management and distribution of government information? Other industries have tried to preserve their old business models but have been unsuccessful – artificial scarcity is met by abundance.

What are the specialised roles that will emerge to support this more complex environment - data retailers, data wholesalers, distributors, quality control inspectors, curators and regulators – and what are the new operational models?  Can we expect to see the emergence of the farmers market, specialist stores, department stores and hypermarket chains?

Industrialisation

Just over 200 years ago, the industrial revolution replaced human and animal labour with machines which led to major new modes of mass production, and the related social and economic changes that are the foundation of modern society.

The promise of the semantic web, interoperability, and the 5 star scale of open data may be a pointer to a future. Many have commented that the semantic web is too complex for today’s operational needs; the open data ecosystem may need to evolve through different phases.  The experience of early pioneers can help to ensure that all parts of the ecosystem develop to support high levels of automation - the journey will be much shorter than the evolution of the human race, but will still take decades to reach full potential.

Sector Development 101

Singapore reclaimed a leadership role in e-government and ICT sector development with the announcement this week of eGov2015 and the third call for Cloud Computing Proposals.

Their approach to cloud computing is an outstanding example of sector development – both strategy and execution. Singapore decided to promote the City State as a cloud computing hub by encouraging vendors to establish cloud facilities in Singapore, and then issuing a series of calls for the use of these facilities. Different cloud vendors have participated in each of the three calls for cloud computing proposals.

To ensure the investment is a genuine stimulus to the sector, eligible applicants have to be Singapore-registered companies, and supportable types of projects include trials, proof-of-concepts and testing.  The government had previously identified specific verticals - digital media, life sciences, manufacturing, financial services, retail & tourism, and education. Organisations developing solutions in these areas were selected to take advantage of subsidised cloud computing resources.

The first call in May 2010 outlined the objective to establish Singapore as a Shared Services Hub, focus on the verticals, and enable local users, especially SMEs, and to exploit SaaS (software as a service) for HR, finance, IT & other admin functions. In November 2010, the second call was issued focused on the same verticals. Proposals that were selected include: video hosting and streaming, social media monitoring and analysis, document sharing loud services marketplace, asset traceability and management, Radio Frequency Identification (RFI) technology, commodity trading and investment risk assessment solutions, smart traffic and mobile phone data screening.

The third call, issued last week, continued a focus on “lighthouse projects” that illustrate and promote the use of the infrastructure in the verticals, and added a focus on transportation and construction. At the same time Singapore became the third Asian country to launch an open data portal with more than 5,000 datasets – providing developers with plenty of data to crunch in the cloud.

Other governments should watch and learn.

They are being assessed – also this week a "Cloud Readiness Index" was announced which will analyse 10 key attributes critical to the deployment and use of cloud computing technology across 14 different countries in the region - China, Australia, Hong Kong, India, Indonesia, Japan, Korea, Malaysia, New Zealand, the Philippines, Singapore, Taiwan, Thailand and Vietnam.

Small earthquake in the media: not many dead

Wikileaks took pole position in world headlines in November 2010, becoming an "overnight sensation" after several years under the radar.  Was it, as the Guardian claimed "the first really sustained confrontation between the established order and the culture of the internet"? Is Naomi Wolf right about the ambivalence of US media in their defence of journalists revealing truth?

As the story was breaking, I was travelling between the Middle East, Asia and New Zealand. I followed the story with an obsession that was fed across timezones and political boundaries - accessing information sources without borders. The immediacy of the internet media pushed the circulation of ideas to hyperspeed: Live tweeting from the court hearing in London, commentary flaming in media blogs, and a palpable sense of excitement as the "war" escalated. The DDoS attacks on Wikileaks and the attacks by Anonymous on suppliers who withdrew their services, had all the characteristics of early skirmishes. As Julian Assange remarked sardonically "We now know that Visa, Mastercard, PayPal and others are instruments of US foreign policy. It's not something we knew before".  And the continuing leaks around his court case include the publication of his defense. The events have been great theatre, and rich in irony.

At the same time, Wikileaks has very quickly become part of the journalism landscape. Although only 3,500 of the 251,287 cables have been published, news stories are now regularly accompanied by commentary from Wikileaks - it has become another source of information. 

After the obligatory 15 minutes of fame, has the world changed?

In my view, it has. Many people that I respect have already published their views; here I contribute why I think so.

Before answering, I need to separate the Wikileaks effect from its founder - Julian Assange.  I have no direct knowledge of him, or what caused him to pursue his quest with a clear understanding of his goal and the associated dangers,  but I enjoyed reading Aaron Bady's insights into Julian the man.  Here I look at the structural implications for a 21st century fourth estate.

Since the globalisation of capital in the 1970s, there has been an imbalance between power and accountability. Neo-liberal economic ideology, which should have ended with the 2008 financial crisis, amazingly led to rehabilitation of the very people who got us into this mess, and the perpetuation of the obscene gap between the rich and poor. 

Rudolf Elmer recently used Wikileaks to release financial details of Swiss bank account holders - an indication that Wikileaks is not just a channel for US government leaks. Wikileaks targets power and wealth, without fear or favour, and is a necessary counterbalance to the transnational resources available to power and wealth. Look at the organisations that have been held to account.

Power and wealth seeks privacy and secrecy, and abhors public scrutiny.  Social institutions have evolved, building on the hard work of brave and ethical leaders, to create controls which balance the exercise of power. We look aghast at the excesses of dictators in developing countries who accumulate and hide wealth while their people live in poverty.  However in western democracies, power and wealth have found ways to avoid examination by the traditional fourth estate, which is struggling for survival, recycling press releases from governments and corporations alongside celebrity gossip and reality shows.

Laws and government systems seek to balance the interests of the individual with the interests of the community. Globalisation has removed capital from the control of these laws, and a globalised media has not yet emerged to hold power and wealth to account. Power and wealth take advantage of this opportunity to avoid the oversight of national watchdogs, and use professional advisors, transnational wealth management and offshore funds to minimise their contribution to the collective interest. 

Wikileaks has been described as the first stateless news organisation, operating as a networked organisation, and not subject to the control of any single government.  Power and wealth have understood the value of being stateless; it has probably always been this way, but we thought that good governments controlled abuses. This faith has dissolved as the secrecy is lifted, and behind the veil we see how the elite, including our political leaders, live. The Cayman Islands has the fifth largest banking asset base on the planet (277 banks holding US$2 trillion of deposits in a country with a population of 60,000). Holding power and wealth to account, whether it is through the US embassy cables or the details of offshore wealth held in Swiss banks accounts, is an important task that needs to operate on a global basis in the 21st century.

After the initial publication of the first 191 cables on 28 November 2010, the Wikileaks web site experienced a distributed denial of service (DDoS) attack; in the following week, a range of suppliers withdrew from providing their services to Wikileaks including Paypal, MasterCard and Visa payment services, PostFinance in Switzerland who closed the Wikileaks bank account, Tableau data visualization, EveryDNS domain services, and Amazon hosting services. This line-up indicates the significance of the challenge to the status quo.

The ability of the internet to hold power to account has been praised in western media when applied in countries like China, North Korea, much of the Middle East and a large part of Africa.  The statement by Hilary Clinton in January 2010 seemed to define the US policy in support of "Internet freedom .. {giving} people access to knowledge", but the policy turned out to be fungible when the power of the internet was applied to US government activities. 

What we have found is that technology is value neutral, and the internet can apply scrutiny to all sources of power and wealth, irrespective of ideology.  Operating with integrity and openness is the only way to the moral high ground.

Power and wealth are now facing an equal and opposite force of a different nature - the agility and resilience of the internet. 

• "The Net interprets censorship as damage and routes around it" - John Gilmore's famous quip 
• A mirror of the Wikileaks Cablegate site was seeded as a torrent 20 minutes after its availability was published on Twitter. 

This speed and agility is beyond the ability of any government to control: the Chinese government's Green Dam Youth Escort project was not successful, and currently technology is enabling Egyptians to carry on texting and tweeting during anti-government protests, circumventing the country's internet shutdown.

Even if Wikileaks has a short life, the world of power and wealth has been changed forever, in the same way that Napster deconstructed the music business. Creating a place to "help you safely get the truth out" is an essential part of creating a more balanced world, now that individual governments can no longer do so. The fact that there is such a place will change the behaviours of those holding power and wealth.

From Napster to Pirate Bay, from Wikileaks to Openleaks, the genie is out of the bottle.  Does government really want to start another unwinnable war against an abstract noun?

Or as the Guardian put it (before the separation):

Politicians now face an agonising dilemma. The old, mole-whacking approach won't work. Our rulers have a choice to make: either they learn to live in a WikiLeakable world, with all that implies in terms of their future behaviour; or they shut down the internet.


Places to go to find out more

Wikirebels - a must-watch documentary from STV. Some soundbites:

• "A difference can be made bottom up"
• "Information does not respect borders"
• "Democracy without transparency is not democracy, it's just an empty word"
• "By and large Wikileaks is a force for the good. Wikileaks in very very powerful  ... one has to be cautious about anything that is very powerful"

You see it's not really about Wikileaks. .. every artificial scarcity will be met by an equal and opposite artificial abundance; over time, the artificial abundance will win.

The leaders of Myanmar and Belarus, or Thailand and Russia, can now rightly say to us "You went after Wikileaks domain name, their hosting provider, and even denied your citizens the ability to register protest through donations, all without a warrant and all targeting overseas entities, simply because you decided you don't like the site. If that's the way governments get to behave, we can live with that.

Society is made up of competing goods that can't be resolved in any perfect way - freedom vs. liberty, state secrets vs. citizen oversight

In a world where leaking is easy, secretive or unjust systems are nonlinearly hit relative to open, just systems.

Governments will have to learn what the music and film industries have been forced to learn already, that it's easy to copy and publish digital files

In 1958 a young Rupert Murdoch, then owner and editor of Adelaide's The News, wrote: "In the race between secrecy and truth, it seems inevitable that truth will always win."

Is Open Government absolute?

A new normal has emerged in public policy thinking over the last 18 months: the idea that the reality of Open Government is a “good thing” and that Open Data is an important step in turning the policy of open government an operational reality. The latest event was the Declaration of Open Government last month by the Australian government. So are there boundaries to Open Government? If the mantra is "open by default", what are legitimate reasons for not following the default?

Cultural change involved in implementing Open Government is huge. Despite the fact that we have had the Official Information Act in place in New Zealand for almost 30 years, there are still questions over whether government officials are comfortable operating in accordance with the legislation. The widespread availability of Internet technology removes significant operational barriers from releasing official information, and has exposed more deep-seated systemic resistance. Legislation is a blunt instrument to effect behavioural change.

Last week, the NZ government, led by the Department of Conservation, convened a group of invited participants to brainstorm what data sets should be prioritised across government for release under an open data program. A good idea, I thought, but others did not.

“.. I am concerned about the process here. It's quite legitimate for private individuals to run invite-only events (Foo, for example), and for private companies to do so (Microsoft Open Govt), but IMHO it's a different ballgame when government is involved, especially (ironically) when the topic is open govt or open govt data. ....  We need to scale past this invite-only, pussy-footing to larger, public, open processes as fast as possible."

The full discussion is here

This got me to thinking about the fundamentals of Open Government. Technology has made it possible for every interaction of a government official or politician to be published online. Should every step in the development of ideas take place "on the record" or is there a place for deliberative privacy?

Beth Noveck, Deputy CIO and spearhead of the Obama campaign for open and transparent government, spoke about this when questioned (at 1:20:00) on whether legislating for transparency of meetings simply moved discussions from 'meetings' to the lunch table.  (There was a secondary question about whether secrecy is used as an excuse for not making data open; in this post, I am focused on information that is not subject to privacy or national security constraints).

If everything is totally open, people either grandstand for the cameras or are reluctant to join a conversation for fear of appearing stupid. Deals are not made in front of TV cameras, and locking politicians into early public positions can have unforseen consequences.
 
The White House publishes a list of visitors, and meetings of senior government officials are a matter of public record "so we know there are no secret energy meetings going on, but they are in fact happening out in the open". The problem of institutional reform is hard, and the current approach is to start with lobbying – “ensuring that when we talk with lobbyists we are much more open than when we are going to lunch with our colleagues”.

Beth’s view is also that there are practical constraints on full transparency - "if we had to record every conversation, we would spend so much time being open, there would not be enough time to get the job done". However, this may be a temporary constraint - it is not hard to imagine a future where everything is able to be recorded without any active effort required of participants.

Just because everything can be made part of the public record, does not mean it should be. In the new world of Open Government, we need to understand the legitimacy of government having discussions in private or by invitation only. The questions are:

• the criteria for choosing who is involved in early stage discussion 
• the disclosure requirements about the existence and content of early stage discussions
• at what point should those discussions be opened up to a wider community. 

I personally support deliberative privacy with transparency of participation - publication of who is part of the early stage discussions. The results should be disclosed early enough that positions are not fully formed (and we find ourselves in faux consultation), while allowing enough time to develop ideas so that the subsequent dialogue is effective. In many ways, public officials are caught either way: when ideas are first floated they are criticised for being short on detail, and if they present detailed proposals they are criticised for not consulting widely enough.

Back to the government event last week. I stand by my view that the event was a good idea. People outside government do not always understand the pressures and cultural expectations that constrain what you can do within government. For this event, the organisers made efforts to cast the net wide for community involvement, and published the outcome of the discussions within two working days. As far as I am aware, no-one was excluded from the event, and it is hard to see what more the organisers could have done to operate openly.

As Public Strategist has observed :
The government is an elephant -  It tries to dance, but finds it hard, and the smaller animals around it can get hurt.  The solution may be for the elephant to stand stock still, to do nothing for fear of treading on something more nimble, but more easily hurt. Or it may be to learn to tread more carefully, to place its feet carefully, but to keep moving nevertheless. Or it may be to charge ahead regardless and let others survive as best they can.

When faced with Open Government, the natural response of the elephant is the first option, so we should support and encourage government officials to choose the second option. Encourage them to interact with people and understand how to operate within a community, and join in a rolling conversation in different forums without constraints on who should be involved in individual events. The more conversations there are about Open Government, the more we will make progress. Anyone who champions the cause should be praised and encouraged.

The US government chose the third option, and they did this because of the unequivocal leadership from the top - the very first thing President Obama did, on his first day in office, was to sign the executive memorandum on transparency and open government. With that level of leadership, the elephant can charge ahead.  Without that leadership, the best we can expect from our public servants is that they learn to move forward, and keep moving. Our job is to encourage those that are moving and harangue those that are not.

So praise to the State Services Commission who today announced the formal adoption by Cabinet of the NZ Government Open Access and Licensing framework (NZGOAL).  This provides guidance for State Services agencies to follow when releasing copyright works and non-copyright material for re-use by third parties, standardises the licensing of government copyright works for re-use using Creative Commons licences, and recommends the use of ‘no-known rights’ statements for non-copyright material.

To continue the conversation: OpenLabourNZ is running a public one-day event at the end of August. I am looking forward to the presence of Andrew Rasiej, who will be joining the event by video link. I look forward to further discussion of these issues, which are at the heart of Open Government.

Active with the activists

45 minutes outside Chiang Mai in Northern Thailand, nearly 100 attendees, from NGOs, community media, academics, activists, social workers and government officials, are at the second MekongICT camp.  People have come from Thailand, Laos, Myanmar/Burma, Cambodia, Bangladesh, Malaysia, Korea, Vietnam, India and Sri Lanka; from outside the region there are community-based ICT proponents from the US, Canada, and New Zealand.

My key note covered Government 2.0 and Civil Information Society – why it is important, examples of government 2.0 around the world, and how to move forward.  Coming from a high trust, transparent democracy in New Zealand, I was filled with admiration for activists working in these Mekong countries who have none of the tools that we take for granted - such as Freedom of Information legislation or consultative democracy.

One question that I was asked – “what incentives can be put in place for countries like ours, where government is closed and we cannot get information?” – gave me food for thought.  Apart from anger that people with power, in positions of authority and leadership, do not accept their responsibilities to serve the people who they lead, I could only come up with one answer - find an individual government official who wants to make a difference. If  "Law is the Operating System of Democracy", how do you lead your life without a reliable operating system?

The ideas emerging so far include protecting your information when operating as an NGO, and activism through citizen journalism.There are teams building community radio stations with parts that you used to be able to buy from Dick Smiths and Radio Shack before they put them all away in the drawers at the back and sold you glossy end products. Another group are creating a wifi mesh network to connect communities peer to peer communities.  Along the hall, I am at a workshop designing SMS systems for supporting farmers - "for people in the world that need it most, the only way to access the power of ICT is through SMS.


I keep thinking "Agitprop" without even knowing if that is a word with currency in the 21st century - it meant a lot in 1970. Wikipedia tells me that it is term from Bolshevist Russia, that had negative connotations in the west. It seems to fit well with what is happening here - communities helping each other to help themselves.  Billy Bragg would be proud of them:


Jumble sales are organised and pamphlets have been posted
Even after closing time there's still parties to be hosted
You can be active with the activists
Or sleep in with the sleepers
While you're waiting for the great leap forward

Eat your own dogfood

The New Zealand Labour Party have launched a new way of developing policy – out in the open, involving anyone who chooses to participate. They are starting by developing a policy on open and transparent government. Some might criticise this as a self-referential policy wonk, others would call it eating your own dogfood.

I have a preference for drinking your own champagne.

In Australia, Kate Lundy started her use of ICT and social media tools while in opposition, and last year launched the successful Public Sphere initiative, which built momentum behind the creation of the Gov2.0 TaskForce. In the US, social media tools were used by the Obama campaign to tap into what the public wanted, proving an effective tool for listening to a wide set of stakeholders. However, for yesterday's general election in the UK, social media seemed to be used more as a source of data on public sentiment than as a campaign tool.

It is probably not surprising that many such initiatives start from opposition parties. Being in government creates a lot of work, driven by daily operational imperatives, so there is less time to consider the fundamentals of the democratic process. In addition, the ruling government contains the people upon whom lobbyists spend most of their money and attention, creating a community that is protective of their insider position.

The risks of fake participation can be reduced by ICT technologies. The techniques used to give the illusion of participation have been identified as

• Don’t publicize the meeting to potential opponents.
• Schedule the meeting at an inconvenient time or place.
• Stack citizen representatives on public bodies.
• Signal the futility of participating to those most likely to participate.
• Intimidate potential opponents by forcing them to reveal their identities

With these risks in mind, it is ecouraging that the Labour Party initiative was re-tweeted by the National Business Review, the publication of choice for business leaders. And comments on the policy are coming from across the political spectrum.

In 2007, New Zealand provided world leadership in the use of e-participation, through the Police Act wiki, the Bioethics Council and guidance from the State Services Commission. More recently, New Zealand has been overtaken by developments in London, Canberra and Washington. It is encouraging that the Labour Party has started this initiative and already there are discussions about some fundamental policy points. Should the Official Information Act cover commercial in confidence information, given that government is using public money? During the negotiation phase it is clearly appropriate, but once the contract has been signed, there is no good reason for withholding the details. Opening the operations of our elected MPs up to the same level of scrutiny as other parts of public life is another aspect of Open Government.

We have a way to go. Since early 2009, the Federal US government have led in the implementation of open government and open data. The Australian government have already accepted the recommendations of the Gov 2.0 task force and are proceeding to implementation And despite the absence of social media in the election campaign, the British government have introduced some real innovation in citizen participation.

New Zealand leads the world in good government, which has open and democratic policy-making at its core. We need to broaden and strengthen the dialogue, and accelerate the pace, to retain that position