Partition and Sorting
Partitioners
All values with the same key must be sent to the same Reducer.
If there is no Partitioner configured, MapReduce jobs use the HashPartitioner by default. The HashPartitioner uses the hashCode method
of Object (along with the modulus operator) to determine how the records are partitioned.
A group of records from the intermediate key space is assigned to each reduce node. These groups of records are called partitions, anda partition represents the input of a Reducer.
How a record gets assigned to a Reducer:
1.The Mapper outputs <key, value> pairs (records). Once the map task is complete, the partitioning of records can begin.
2.The Partitioner is an object that defines a getPartition method. Each < key ,value > pair is passed into the getPartition method, along with the number of Reducers.
3.The getPartition method returns an int that determines which Reducer the < key ,value > pair is sent to.
The Default Partitioner
public class HashPartitioner<K, V> extends Partitioner<K, V> {
/** Use {@link Object#hashCode()} to partition. */
public int getPartition(K key, V value, int numReduceTasks) {
return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;
}
}
Writing a Custom Partitioner
public class WordCountPartitioner extends Partitioner<Text, IntWritable> {
@Override
public int getPartition(Text key, IntWritable value, int numReduceTasks) {
if (numReduceTasks == 1) {
return 0;
}
return (key.toString().length() * value.get()) % numReduceTasks;
}
}
The TotalOrderPartitioner accomplishes its task by using an external file that defines how the keys are split across partitions.
Involves two main steps:
2.Share the partition file amongst all Mappers.
Create a partition file
- job.setPartitionerClass(TotalOrderPartitioner.class);
- InputSampler.Sampler<Text, Text> sampler = new InputSampler.RandomSampler<Text, Text>(0.1, 200, 3);
- InputSampler.writePartitionFile(job, sampler);
• mapreduce.lib.partition.InputSampler
• -inFormat org.apache.hadoop.mapreduce.lib.
• input.KeyValueTextInputFormat
• -keyClass org.apache.hadoop.io.Text
• -r 3
• -splitInterval 0.1 3 population_data.txt _partition.lst
Distributing the Partition File
•String partitionFile = TotalOrderPartitioner.getPartitionFile(conf);
• URI partitionUri = new URI(partitionFile + "#"
• + TotalOrderPartitioner.DEFAULT_PATH);
• job.addCacheFile(partitionUri);
The getPartitionFile method returns the path to partition file.
The partitionUri is a URI that represents the path that theTotalOrderPartitioner looks up when retrieving the partition file.
The addCacheFile method adds the partition file to the LocalResource so each Container can have access to
it.
Overview of Sorting
The shuffle/sort phaseperforms two key tasks:
1.Keys are sorted in their natural order.
2.Keys that are equal are grouped together.
Recall the key class has to be of type WritableComparable, which forces a compareTo method to be defined. The compareTo method
creates what is called a natural order for the keys.
Grouping Comparator, it decides which keys are equal to each other. If two keys are equal, their values get grouped together.
Secondary Sort
The easiest way to implement a secondary sortis to move part of the value into
the key to form a composite key.
Here are the three main stepsto follow:
1.Write a custom key class that contains the secondary key. While it may be possible to
use a built-in Hadoop key class, typically it must be defined.
2.Write a custom Grouping Comparator to determine how keys are grouped.
3.Write a custom Partitioner that ensures grouped keys are sent to the same reducer.
Writing Custom Keys
A custom key class needs to implement the WritableComparable interface.
public class CustomerKey implements WritableComparable<CustomerKey> {
private int customerId;
private String zipCode;
@Override
public int compareTo(CustomerKey arg0) {
int result = this.zipCode - arg0.zipCode;
return ((result != 0) ? result : (this.customerId - arg0.customerId));
}
@Override
public void readFields(DataInput in) throws IOException {
this.customerId = in.readInt();
this.zipCode = in.readUTF();
}
@Override
public void write(DataOutput out) throws IOException {
out.writeInt(customerId);
out.writeUTF(zipCode);
}
//setters and getters...
}
A custom Group Comparator needs to implement the WritableComparator class.
public class CustomerGroupComparator extends WritableComparator {
protected CustomerGroupComparator() {
super(CustomerKey.class, true);
}
@SuppressWarnings("rawtypes")
@Override
public int compare(WritableComparable a, WritableComparable b) {
CustomerKey lhs = (CustomerKey) a;
CustomerKey rhs = (CustomerKey) b;
return lhs.getZipCode().compareTo(rhs.getZipCode());
}
}