One thing will stand out to you right away if you take a short glance at the development of big data: Almost always, our capacity for data processing has lagged behind our capacity for data collection. Processing power once increased exponentially, but this growth has since slowed. The amount of data available, which keeps expanding yearly, cannot be stated to be the same. The statistics are shocking. The amount of data generated between 2014 and 2015 was greater than at any other time in human history, and it is expected to quadruple every two years. Our total digital data was expected to reach 44 zettabytes (or 44 trillion gigabytes) by 2020 and 180 trillion gigabytes by 2025. Despite this focused effort to gather data, only around 3% of it has ever been examined.
More people are conducting every aspect of their personal and professional life online. It can be simple to forget that the "internet revolution" is still very much ongoing if you live in a relatively wealthy region of the world (or, in reality, the United States). There are many people who have not yet joined the internet since internet usage in the USA is still lower than in other nations. They will be stepping into a world where their every move is being watched while they do so. This is mostly done so that adverts can be specifically targeted to them, but it has also resulted in the creation of enormous data banks about specific internet users.
To argue that the technologies currently in use to store and handle this data are antiquated would be an understatement. Up until very recently, open-source ecosystems like Hadoop and NoSQL were mostly used to solve huge data processing problems. However, manual configuration and debugging are necessary for many open-source solutions, which can be challenging for most businesses. This was the main driver behind organizations beginning to move huge data to the cloud about ten years ago. Since then, the storage and processing of large data have been revolutionized by AWS, Microsoft Azure, and Google Cloud Platform. Before, businesses had to physically expand their own data centers when they wanted to run data-intensive programs. Now, cloud infrastructure offers agility, scalability, and ease of use with pay-as-you-go services.
But we should take the time to reflect on the previous transition as this new one takes shape. Big data acquisition systems, which automatically collected and stored trillions of data points on billions of internet users, have only recently come to light as having ethical ramifications. With AI systems, we shouldn't make the same error. There are some encouraging indicators: Giants like Google and IBM are already pushing for greater openness by incorporating tools for bias detection into their machine learning models. However, more than just more powerful AIs and larger data centers will be required if we are to fully realize the benefits of big data. A moral framework will also be necessary to determine when, why, and how to use this data.
For different topics in this field, you can check our Statistic and Data training programs