Big Data for Social Impact: A Research Lab

NORRAG Highlights, 2019

This NORRAG Highlights is published by D. Brent Edwards Jr., Assistant Professor of Theory and Methodology in the Study of Education at the University of Hawaii, Manoa. In this post, the author looks at methods used to process and transform education- related data called “impact evaluations”. Big data in the field of global education policy uses these methods, namely regression analysis and randomized controlled trials (RCTs), despite them having serious limitations. The author cautions that although big data and the methods of impact evaluation can be useful inputs for policy discussions, it is important that their limit be understood and that in the context of policy reform and global governance other qualitative strategies be used by decision-makers.

ABSTRACT Toward application by policy and program planners working within the nexus linking public administration and public health in supporting sustainable, built, complex adaptive healthy communities; predictive big data analytics is a series of emerging datascience methods by which critical connections are made and strengthened through the sharing and data mining of massive quantities of data located across diverse public datasets. The days of studying and working in any one discipline or niche are quite likely over; and, the polymath mind and related analytical techniques rules the process of future public policy planning in solving social problems that impact the health and welfare of communities. Big data predictive analytical tools provides communities with the power to make better informed decisions rather than relying on guesswork based on inadequate data access and analysis. Prediction from the massive amount of existing data is empowering, but, not perfect; however, any real time driven prediction remains more powerful and satisfying than merely relying on a public agency’s best guess. The concept of big data reflects the reality today that massive amounts of data are stored in a variety of depositories; and, are awaiting download and analysis by public community planners and others. Big data is characterized by volume, velocity, and variety. Volume is easy to understand. There is so much data stored it is characterized as big or massive and it is now measured in zettabytes (bytes with 20 zeros following). Velocity is also a characteristic since big data moves through the network with lightning speed. Further complicating how big data is downloaded and analyzed is the almost infinite variety characterizing the type of data and its storage format as it arrives and is stored in various massive databases under widely differing categories which makes data mining complicated. These characteristics have driven the development of new statistical analysis tools capable of downloading massive amounts of data (volume) at high rates of speed as new data arrives (velocity) thus providing real-time updates, and mines critical information regardless of how it is stored (variety) while not drilling down to individual identities thus ensuring privacy. This revolution in data management and analysis has created a new kind of professional; the data scientist who combines knowledge of computers with statistics and knowledge of the environment of smart, healthy communities in the 21st century. The benefits are readily available; but, the trajectory and speed of progress are accelerating in the direction of improved prediction in complex adaptive systems where once politically driven agency agenda specific best guesses were the norm with potentially unacceptable failure rates and frequent misuse of scarce community resources invested in a less than optimal direction.

Big data is created every day by the interactions of billions of people using computers, GPS devices, cell phones, censors and medical devices, data-intensive areas such as atmospheric science, genome research and , astronomical studies. Today big data opens huge opportunities to those who can use it effectively. Now realizing the great importance of big data, many analytical companies are engaged in finding hidden information in big data. According to internet experts the present technological advances to collect and analyze massive sets of data is likely to lead to revolutionary changes in business, and society. Till date a lot of work has been done on the tools, software, platforms, analytics etc. applied to big data. Many organizations are giving attention in big data analytics for development, education, disaster management, health care, and natural resource management for benefit of society. Therefore, it is attempted to compile and document the real use cases, benefits, advantages, impact and future challenges of big data. UN Global Pulse has worked on several research projects in collaboration with public and private partners demonstrating the beneficial effect of analytics from monitoring early indicators of unemployment hikes to tracking fluctuations of commodity prices before they are recorded in official statistics. According to thought leaders big data is already showing the potential in areas as genetic mapping and personalized e-commerce. The unprecedented growth in processing power and software technologies such as Hadoop, are allowing organizations " to make decisions that simply could not be made before " having profound impact. Its influence is felt in business planning, research, sales, production and elsewhere. These are considered as new industrial revolution. Scientists used to decode human DNA in minutes, find cures for cancer, accurately predict human behaviour, optimise marketing efforts, prevent diseases and foil terrorist attacks, utilising big data. Finally, concerns about privacy expressed by experts cannot be ignored. As many companies use our private information. The presentation is focused on how Big Data Analytics impact health care and Society.

The discovery of phenomena in social networks has prompted renewed interests in the field. Data in social networks however can be massive, requiring scalable Big Data architecture. Conversely, research in Big Data needs the volume and velocity of social media data for testing its scalability. Not only so, appropriate data processing and mining of acquired datasets involve complex issues in the variety, veracity, and variability of the data, after which visualisation must occur before we can see fruition in our efforts. This article presents topical, multimodal, and longitudinal social media datasets from the integration of various scalable open source technologies. The article details the process that led to the discovery of social information landscapes within the Twitter social network, highlighting the experience of dealing with social media datasets, using a funneling approach so that data becomes manageable. The article demonstrated the feasibility and value of using scalable open source technologies for acquiring massive, connected datasets for research in the social sciences.