不使用 JobConf 运行 Hadoop 作业

Run Hadoop job without using JobConf(不使用 JobConf 运行 Hadoop 作业)
本文介绍了不使用 JobConf 运行 Hadoop 作业的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着跟版网的小编来一起学习吧!

问题描述

限时送ChatGPT账号..

我找不到提交不使用已弃用 JobConf 类的 Hadoop 作业的单个示例.尚未弃用的 JobClient 仍然只支持采用 JobConf 参数的方法.

I can't find a single example of submitting a Hadoop job that does not use the deprecated JobConf class. JobClient, which hasn't been deprecated, still only supports methods that take a JobConf parameter.

谁能指出一个 Java 代码示例,该示例仅使用 Configuration 类(不是 JobConf)提交 Hadoop 映射/减少作业,并使用 mapreduce.lib.input 包而不是 mapred.input?

Can someone please point me at an example of Java code submitting a Hadoop map/reduce job using only the Configuration class (not JobConf), and using the mapreduce.lib.input package instead of mapred.input?

推荐答案

希望对你有帮助

import java.io.File;

import org.apache.commons.io.FileUtils;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class MapReduceExample extends Configured implements Tool {

    static class MyMapper extends Mapper<LongWritable, Text, LongWritable, Text> {
        public MyMapper(){

        }

        protected void map(
                LongWritable key,
                Text value,
                org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, LongWritable, Text>.Context context)
                throws java.io.IOException, InterruptedException {
            context.getCounter("mygroup", "jeff").increment(1);
            context.write(key, value);
        };
    }

    @Override
    public int run(String[] args) throws Exception {
        Job job = new Job();
        job.setMapperClass(MyMapper.class);
        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        job.waitForCompletion(true);
        return 0;
    }

    public static void main(String[] args) throws Exception {
        FileUtils.deleteDirectory(new File("data/output"));
        args = new String[] { "data/input", "data/output" };
        ToolRunner.run(new MapReduceExample(), args);
    }
}

这篇关于不使用 JobConf 运行 Hadoop 作业的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持跟版网!

本站部分内容来源互联网,如果有图片或者内容侵犯了您的权益,请联系我们,我们会在确认后第一时间进行删除!

相关文档推荐

Sending a keyboard event from java to any application (on-screen-keyboard)(将键盘事件从 java 发送到任何应用程序(屏幕键盘))
How to make JComboBox selected item not changed when scrolling through its popuplist using keyboard(使用键盘滚动其弹出列表时如何使 JComboBox 所选项目不更改)
Capturing keystrokes without focus(在没有焦点的情况下捕获击键)
How can I position a layout right above the android on-screen keyboard?(如何将布局放置在 android 屏幕键盘的正上方?)
How to check for key being held down on startup in Java(如何检查在Java中启动时按住的键)
Android - Get keyboard key press(Android - 获取键盘按键)