Big data and data analytics

Data is now produced at a colossal rate: most web traffic comes from smartphones, and around 2.5 quintillion bytes of data are created each day, growing further with the Internet of Things.

Big data is data so large and complex that traditional data processing tools cannot handle it. Much of it is unstructured: posts, messages and chats, shared photos, tweets, blogs, news, polls and comments, audio and video chats.

• It brings challenges of integration, storage, analysis, searching, processing, transfer, querying and visualisation.
• It can hold rich knowledge of high business value, so effort goes into tools to process it.

Five characteristics (the five Vs):

• Volume: its enormous size, too large for traditional DBMS tools; the most prominent feature.
• Velocity: the rate at which it is generated and stored, far higher than for traditional data.
• Variety: structured, semi-structured and unstructured data: text, images, videos, web pages.
• Veracity: its trustworthiness. Big data can be inconsistent, biased or noisy, or badly collected; processing wrong data gives wrong results.
• Value: hidden patterns and useful knowledge. Because processing costs money, check its likely value first, or the effort may be wasted.

Data analytics is examining data sets to draw conclusions using specialised systems and software. Businesses use it to make informed decisions; scientists use it to test models, theories and hypotheses. Pandas, a library of the Python language, makes data analysis much simpler.

Example (new): a video platform receiving hundreds of hours of uploads every minute, in every format from short clips to live streams, shows high velocity and high variety at the same time.