Open data has emerged as a promising new approach to enable free flow of closely held information in the academic community as well as government and regulators. The open data movement calls for making available all public data in accessible formats. The ambit of “public data” is very broad: it can be defined as all information or data which public authorities collect, produce or pay for. Examples include legislation, judgements, policies, budgets, census data, map data, crime statistics, data on performance of government schemes, public transport timetables and routes, research funded by public funds, historical records and so on.[i]

The benefits of an open data movement in government are manifold, falling under the broad categories of:

1.      Increasing transparency and accountability: Information previously known only to the State is made available to citizens, enabling them to more effectively hold it accountable for its actions and performance

2.      Improving administrative efficiency: Open data enables both government departments and civil society to more effectively monitor and influence public policy formulation and implementation

3.      Economic benefits: Data such as geographic data, traffic data, land use data and so on can be used by private sector firms to improve efficiency of their operations and innovate new areas of business

On the other hand, fully tapping into the transformative potential of Open Data requires new capacities from a range of actors. Governments need to shed their silo-based hierarchical model of information, proactively release usable data and constructively engage with feedback from civil society and the private sector. Civil society needs to build technical capacity to understand and analyse a flood of data, and engage with governments, providing inputs aligned internal departmental processes. 

Open Data in India

Some government data has always been public in India. The legislatures serve as an important clearinghouse of information, since data and reports submitted to them are made available to the public.

An important beginning to Open Data was made in India with the Freedom of Information Act in 2002, later superseded by the much stronger Right to Information Act in 2005. The RTI compels public authorities to release information sought by citizens, subject to a small restricted list of information relating to national security, current investigations and personal data. An interesting non-binding clause of the RTI also asks authorities to identify and release proactive information, though this is observed only in the breach.

While the RTI provides the legal foundation for an Open Data movement, the actual implementation of it falls short of a complete Open Data framework. It is reactive rather than proactive, the government asserts that information shared under the Act can only be used by the applicant rather than the public at large, and information is usually shared as PDFs of scanned copies of printed pages, which significantly hinder accessibility.

Some government agencies, notably the Election Commission, Planning Commission and Reserve Bank of India, voluntarily release large amounts of information; both collected and produced; relevant to their functional areas. Most Central Government agencies also submit an “outcome budget” with a summary of performance indicators. Central government schemes such as the NREGA and the NRHM release regular reports, and also have an inbuilt MIS, of which portions are publicly accessible. The Union Budget and all supporting documents are available online as both data and reports; while many State Budgets for the current financial year are available online only in report format. Obtaining data from lower-level agencies such as state Departments, municipal governments, district administrations, PSUs, State-owned corporations providing public services such as power, transport or insurance continues to be a challenge.

This fragmented nature of data is not unusual to India; and is perhaps inherent in the very nature of a diverse and federal government. These challenges have been addressed in other countries by the creation of Open Data Portals; specialised agencies which actively solicit data from different departments and serve as a centralised repository of all public data. The UK Open Data portal, in particular, is often cited as a pioneer in this aspect.

In India, an Open Data portal was established in 2012, following a series of government reports such as the National E-Governance Plan, which led to the adoption of a National Data Sharing and Accessibility Policy in 2012.  The NDSAP mandates Central Government agencies to assign a dedicated Data Controller, who will submit datasets to a NDSAP cell. These datasets are vetted for compliance with Open Data standards, and then published on data.gov.in. [ii]

NDSAP: successes and failures

Data.gov.in contains nearly 12,000 datasets relating to 3,200 different subjects from Central Ministries. Data on the portal is accessible in a variety of formats, and is of high quality; complying with international best practices. The managers of the NDSAP cell actively solicit public contributions, and engage with civil society organisations such as the Open Data Meet. An example is thehackathon jointly organised with the Planning Commission, soliciting visualisations, applications and short films focussing on the 12^th Five Year Plan. The staff appear to possess the technical knowledge required to successfully establish and maintain the portal as well as add newer features to it.

However, the primary problem with the portal is that the data on it falls significantly short of being comprehensive. Judicial data is missing altogether, and data from State governments is insignificant, amounting to only 92 datasets from 5 States. Even the data available with Central Ministries in their own repositories is often not available on the portal. As an example, the Ministry of HRD has submitted only 6 datasets relating to primary education: the performance of flagship programs such as the Sarva Shiksha Abhiyan is unavailable on the portal. Another example of unavailable public data is geographical data, which is the legal monopoly of the Survey of India. The Survey makes its data available only for restricted uses on payment of high licensing fees despite the NDSAP. It also prevents third-party data collection, as highlighted in the legal action recommended by the Survey against firms such as Google which collect their own map data

What might be the causes of this poor performance? To help explain it, we can draw on the experiences from more successful data portals in the US and UK.  An exhaustive study of these initiatives reveals that the success was because these initiatives received strong support at three levels: bottom-up pressure from civil society organisations, top-down political push, driven either by the political executive in the US or by high-profile individuals such as Sir Tim Berners-Lee (the inventor of the World Wide Web in the US). A critical role, though, seemed to rest with the ‘middle layer’: a set of serving government officials who were receptive to the agenda; and hence advocated for and implement changes within the government machinery.

From this perspective, it is clear that India remains lacking. High-level political advocacy has been missing; anecdotal evidence and informal conversation suggest strongly that the number of serving officials in the “middle level” who are receptive to the movement is limited. A dedicated NDSAP cell and demand pressures from civil society organisations has been unable to generate enough impetus to solve the systemic underlying issues with government agencies. In an off-the-record conversation, an NDSAP cell manager complained that “Even though we are part of the Government, we find it difficult to make other agencies listen to us”. Reflecting this state of affairs, traffic to the portal is stagnating after a jump when it was launched.

Given the systemic issues involved, progress on the Open Data front seems likely to remain a slow incremental process. International progress and the formation of an international and Indian community dedicated to Open Data might serve as a catalyst; as might the current Government’s stated emphasis on “maximum governance”. Till then, though, the lot of the Indian data junkie remains parsing unformatted PDFs from a dozen poorly designed websites.

[i] For more details and a comparative analysis of how countries fare on data accessibility, see the Open Data Barometer Global Report at  http://www.opendataresearch.org/dl/odb2013/Open-Data-Barometer-2013-Global-Report.pdf

[ii] The NDSAP is administered by the Department of Science and Technology, and its full text can be seen at http://www.dst.gov.in/NDSAP.pdf