Public sector: Hadoop, Spark and the 360-degree view of citizens

Director for Watson and AI applications, IBM

The public sector has one single objective: serve citizens in efficient and effective ways to their highest level of satisfaction, despite having limited resources and budgets. Big data is already turning out to be a key competitive differentiator and asset for businesses across many sectors from telecommunications, to banking, to retail.

The lessons of big data that have been learned in the private sector are just as applicable to the public sector. The web of complexities among different public agencies and departments means that the value of governmental data can be latent in nature and require innovative storage and analysis to unleash. But despite their special position in capturing information, public sector entities have been largely ineffective at using it.

Often, the starting point for the commercial sector in gaining value out of this wealth of information is building a single or 360-degree view of the customer using emerging open source technologies such as Apache Hadoop, Apache Spark and machine learning. Similarly, building a holistic, single or 360-degree view of citizens enables public sector organizations to deliver high-quality services in a more efficient and effective manner.

This single or 360-degree view of citizens offers tremendous power to transform the relationships among businesses, citizens, consumers and government. Big data technologies can transform public services delivery with a citizen-centric approach to provide services the government is obliged to provide to citizens in a more compelling, personalized and timely manner. This approach can enable every part of government to serve citizens better than ever.

The challenge of big data for the public sector

Today, public sector departments and organizations need to become data driven more than ever. However, the amount and diversity of available data, and its rate of change, make continually moving around and transforming huge volumes of data in traditional legacy systems such as relational database management systems (RDBMSs) or customer relationship management (CRM) systems unsustainable. As in the private sector, the modern realities of public sector organizations demand a new approach.

Hadoop offers a data store that is well suited to ingesting data about citizens from multiple sources. These sources include taxation departments, healthcare providers, law enforcement and so on. And Spark also provides capabilities for processing and automating the way such agencies can derive insights from the data.

The 360-degree view of the citizen methodology

Through the use of technologies such as analytics, data science, machine learning and Spark, applying the 360-degree view of the citizen methodology helps transform public service delivery. Public entities can gain a single, unified view of citizens by matching and merging electronic information on citizens across all departments or agencies. This unified view unlocks the power of citizens’ data, allowing it to transform the public services delivery processes: 

  • Iterative process: Developing a 360-degree view of citizens is an iterative process that starts from the best versions of citizen profiles and refines them with more data feeds over time. At the same time, it continually delivers results to relevant public sector agencies and divisions. 
  • Ingest time-series data: The key aspect of completing citizen profiles is ingesting granular, time-series citizen interaction data to determine how they are engaging with government agencies through various channels, and building a baseline for measuring improved outcomes. 
  • External data feeds: The 360-degree view of the citizen profile can be further enriched with external data and activity data feeds such as government website clickstreams, feedback forms, healthcare usage, security departments, transport divisions and so on. Traditionally, this data is kept in divisional or departmental silos; however, the 360-degree view of the citizen enables ingesting all the data in one place and making sense of it. 
  • Citizen insights: The ultimate stage is to derive insights from citizen data that specific agencies can act on to deliver high-value, enhanced service and to proactively reach out to citizens. These insights can be further incorporated into the 360-degree view of the citizen profile along with a feedback loop and iterated back to the first step of building this profile. 

The technology path to citizen centricity

The application of the basic principle of citizen-centric thinking in government is straightforward: the more government knows about an individual, the better it can serve that person. In turn, public sector services can be more responsive and ultimately offer less bureaucratically dense services to citizens. The 360-degree view of citizens together with machine learning enabled by Spark allows for improved design for citizen services. It also can enhance the evaluation of economic benefits of government programs and schemes for its citizens to help eliminate the gap between the needs of citizens and government action.