Explain InputFormat in MapReduce.?

InputFormat is a crucial component of MapReduce that defines how the input data is read, processed, and divided into input splits for processing by individual Mapper tasks. It is responsible for creating the InputSplits and providing a RecordReader to read and interpret the data in the MapReduce framework.

There are three main responsibilities for any InputFormat implementation:

1. Validate the input specifications of the job, such as checking input paths, existence of files, and access permissions.

2. Split the input files into logical InputSplits based on some criteria, such as size, number of records, or file boundaries. InputSplits correspond to chunks of data that will be assigned to different Mappers for processing.

3. Provide a RecordReader that can read and parse the data from the InputSplits into key-value pairs that Mappers can consume.

This means that InputFormat controls the way input data is divided and accessed by the MapReduce system. It also allows you to process different types of data formats in a seamless and efficient manner. There are several built-in InputFormat classes in Hadoop, such as:

- TextInputFormat: Default InputFormat implementation that reads lines of text files and returns the data as key-value pairs, where the key is the byte offset of the line from the beginning of the file and the value is the content of the line. It works well for most plain text files.

- KeyValueTextInputFormat: Reads lines from a text file and interprets them as key-value pairs, separated by a delimiter (default is a tab character). It’s useful when your input data is already organized as key-value pairs in a text file, such as CSV or TSV files.

- SequenceFileInputFormat: Used for reading Hadoop’s SequenceFiles, which is a binary key-value storage format optimized for use with MapReduce. SequenceFiles provide better compression and serialization performance compared to plain text formats.

To use a custom InputFormat implementation, you can extend the InputFormat abstract class and define your own methods for input checks, splitting, and RecordReader creation.

Example:

Let’s consider an example of using TextInputFormat with a MapReduce job written in Java to read a simple text file.

The input file ‘sample.txt‘ has the following content:

“‘

apple red

banana yellow

strawberry red

grape purple

“‘

Configure the InputFormat for the MapReduce job:

import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.LongWritable;

public class ExampleJob {
    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "exampleJob");
        
        // Set InputFormat, Mapper, Reducer, OutputFormat classes
        job.setInputFormatClass(TextInputFormat.class);
        job.setMapperClass(ExampleMapper.class);
        job.setReducerClass(ExampleReducer.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        // Set input and output paths
        TextInputFormat.addInputPath(job, new Path(args[0]));
        TextOutputFormat.setOutputPath(job, new Path(args[1]));

        // Set output key and value types
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(LongWritable.class);

        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

As can be seen in the example, the job is configured to use ‘TextInputFormat‘. The Mapper and Reducer classes will receive input data as key-value pairs corresponding to line offsets and line content, following the TextInputFormat specifications.