Analysing an open dataset

The book's final example analyses an open dataset from data.gov.in (the Open Government Data Platform India): the 2011 census data on the adolescent and youth population, published by the Ministry of Home Affairs. It is a large file, with over 12,000 rows and over 120 columns.

The steps of the analysis:

• read the CSV file into a DataFrame;
• check its size with shape (rows, columns);
• view the columns;
• filter the rows and columns you need;
• make a new DataFrame from them, and rename the columns to short names;
• group the data;
• plot the results.

Filtering with df.loc[row selection, column selection]: for example, the rows where 'Area Name' is 'INDIA', and only the columns you want.

A warning: know your rows before you add them. In this dataset, the 'Total' rows already equal Rural + Urban. So adding all three rows double-counts the population; the book's own Step 7 makes this mistake.

Reading charts: a '1e8' at the top of a chart's y axis is an offset: it means the axis values are in units of 100 million (10 to the power 8), not that the values are wrong.