Pandas-Shuffling, Grouping and Sorting .
Pandas functions like Reset-Index, sort, group and shuffle explained with examples.
Shuffling a Dataset
There may be information lurking in the order of the rows of your dataset. Unless you are dealing with time-series data, the order of the rows should not be significant. Consider if your training set included employees in a company. Perhaps this dataset is ordered by the number of years that the employees were with the company. It is okay to have an individual column that specifies years of service. However, having the data in this order might be problematic.
Consider if you were to split the data into training and validation. You could end up with your validation set having only the newer employees and the training set longer-term employees. Separating the data into a k-fold cross validation could have similar problems. Because of these issues, it is important to shuffle the data set.
Often shuffling and reindexing are both performed together. Shuffling randomizes the order of the data set. However, it does not change the Pandas row numbers. The following code demonstrates a reshuffle. Notice that the first column, the row indexes, has not been reset. Generally, this will not cause any issues and allows trace back to the original order of the data. However, I usually prefer to reset this index. I reason that I typically do not care about the initial position, and there are a few instances where this unordered index can cause issues.
The following code demonstrates a reindex. Notice how the reindex orders the row indexes.
Sorting a Data Set
While it is always a good idea to shuffle a data set before training, during training and preprocessing, you may also wish to sort the data set. Sorting the data set allows you to order the rows in either ascending or descending order for one or more columns. The following code sorts the MPG dataset by name and displays the first car
Grouping a Data Set
Grouping is a typical operation on data sets. Structured Query Language (SQL) calls this operation a “GROUP BY.” Programmers use grouping to summarize data. Because of this, the summarization row count will usually shrink, and you cannot undo the grouping. Because of this loss of information, it is essential to keep your original data before the grouping.
The Auto MPG dataset is used to demonstrate grouping.
The above data set can be used with the group to perform summaries. For example, the following code will group cylinders by the average (mean). This code will provide the grouping. In addition to mean, you can use other aggregating functions, such as sum or count.
It might be useful to have these mean values as a dictionary
The code below shows how to count the number of rows that match each cylinder count.
