Big data and data analytics

We produce data at an enormous rate: about 2.5 quintillion bytes every day, and growing fast as more devices join the Internet of Things.

Big data is data so large and complex that traditional tools (such as ordinary database software) cannot process it. Much of it is unstructured: social media posts, chats, photos, tweets and videos, alongside structured data such as financial transactions. It brings challenges in integrating, storing, analysing, searching, transferring and visualising the data. But it may hold great business value.

Big data has five characteristics, the five Vs:

• Volume: its enormous size.
• Velocity: the speed at which it is generated and must be processed.
• Variety: it comes in many forms: structured, semi-structured and unstructured.
• Veracity: how far it can be trusted. Data may be inconsistent, biased or full of errors (noisy).
• Value: the useful patterns and knowledge hidden in it. Before spending money to process big data, a company should check whether it is likely to be worth it.

Data analytics is examining data sets to draw conclusions, using specialised systems and software. Businesses use it to make better-informed decisions (this is how streaming apps decide what to recommend to you), and researchers use it to test their theories. In Python, the Pandas library makes data analysis much simpler.