org.apache.hadoop.mapred
Interface JTProtocols

All Superinterfaces:
AdminOperationsProtocol, GetUserMappingsProtocol, RefreshAuthorizationPolicyProtocol, RefreshUserMappingsProtocol, org.apache.hadoop.ipc.VersionedProtocol
All Known Implementing Classes:
JobTracker

public interface JTProtocols
extends RefreshUserMappingsProtocol, GetUserMappingsProtocol, RefreshAuthorizationPolicyProtocol, AdminOperationsProtocol


Field Summary
static int TRACKERS_OK
           
static int UNKNOWN_TASKTRACKER
           
static long versionID
          version 3 introduced to replace emitHearbeat/pollForNewTask/pollForTaskWithClosedJob with heartbeat(TaskTrackerStatus, boolean, boolean, boolean, short) version 4 changed TaskReport for HADOOP-549.
static long versionID
           
 
Fields inherited from interface org.apache.hadoop.mr1security.RefreshUserMappingsProtocol
versionID
 
Fields inherited from interface org.apache.hadoop.mr1tools.GetUserMappingsProtocol
versionID
 
Fields inherited from interface org.apache.hadoop.mr1security.authorize.RefreshAuthorizationPolicyProtocol
versionID
 
Fields inherited from interface org.apache.hadoop.mapred.AdminOperationsProtocol
versionID
 
Method Summary
 void addJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)
          Registers a JobInProgressListener for updates from this TaskTrackerManager.
 void cancelDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
          Cancel a delegation token.
 void failJob(JobInProgress job)
          Fail a job.
 JobStatus[] getAllJobs()
          Get all the jobs submitted.
 java.lang.String getBuildVersion()
          Returns the VersionInfo build version of the JobTracker
 TaskReport[] getCleanupTaskReports(JobID jobid)
          Grab a bunch of info on the cleanup tasks that make up the job
 ClusterStatus getClusterStatus()
           
 ClusterStatus getClusterStatus(boolean detailed)
          Get the current status of the cluster
 org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> getDelegationToken(org.apache.hadoop.io.Text renewer)
          Get a new delegation token.
 java.lang.String getFilesystemName()
          The task tracker calls this once, to discern where it can find files referred to by the JobTracker
 java.lang.String getFilesystemName()
          A MapReduce system always operates on a single filesystem.
 JobInProgress getJob(JobID jobid)
          Obtain the job object identified by jobid
 Counters getJobCounters(JobID jobid)
          Grab the current job counters
 JobProfile getJobProfile(JobID jobid)
          Grab a handle to a job that is already known to the JobTracker.
 JobStatus[] getJobsFromQueue(java.lang.String queue)
          Gets all the jobs submitted to the particular Queue
 JobStatus getJobStatus(JobID jobid)
          Grab a handle to a job that is already known to the JobTracker.
 TaskReport[] getMapTaskReports(JobID jobid)
          Grab a bunch of info on the map tasks that make up the job
 JobID getNewJobId()
          Allocate a name for the job.
 int getNextHeartbeatInterval()
          Return the current heartbeat interval that's used by TaskTrackers.
 int getNumberOfUniqueHosts()
           
 org.apache.hadoop.mapred.QueueAclsInfo[] getQueueAclsForCurrentUser()
          Gets the Queue ACLs for current user
 org.apache.hadoop.security.authorize.AccessControlList getQueueAdmins(java.lang.String queueName)
          Get the administrators of the given job-queue.
 JobQueueInfo getQueueInfo(java.lang.String queue)
          Gets scheduling information associated with the particular Job queue
 org.apache.hadoop.mapred.QueueManager getQueueManager()
          Return the QueueManager which manages the queues in this TaskTrackerManager.
 JobQueueInfo[] getQueues()
          Gets set of Job Queues associated with the Job Tracker
 TaskReport[] getReduceTaskReports(JobID jobid)
          Grab a bunch of info on the reduce tasks that make up the job
 TaskReport[] getSetupTaskReports(JobID jobid)
          Grab a bunch of info on the setup tasks that make up the job
 java.lang.String getStagingAreaDir()
          Get a hint from the JobTracker where job-specific files are to be placed.
 java.lang.String getSystemDir()
          Grab the jobtracker system directory path where job-specific files are to be placed.
 java.lang.String getSystemDir()
          Grab the jobtracker system directory path where job-specific files are to be placed.
 TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid, int fromEventId, int maxEvents)
          Get task completion events for the jobid, starting from fromEventId.
 TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid, int fromEventId, int maxEvents)
          Get task completion events for the jobid, starting from fromEventId.
 java.lang.String[] getTaskDiagnostics(TaskAttemptID taskId)
          Get the diagnostics for a given task in a given job
 java.lang.String getVIVersion()
          Returns the VersionInfo version of the JobTracker
 org.apache.hadoop.mapred.HeartbeatResponse heartbeat(TaskTrackerStatus status, boolean restarted, boolean initialContact, boolean acceptNewTasks, short responseId)
          Called regularly by the TaskTracker to update the status of its tasks within the job tracker.
 void initJob(JobInProgress job)
          Initialize the Job
 JobStatus[] jobsToComplete()
          Get the jobs that are not completed and not failed
 void killJob(JobID jobid)
          Kill the indicated job
 void killJob(JobID jobid)
          Kill the job identified by jobid
 boolean killTask(TaskAttemptID taskId, boolean shouldFail)
          Kill indicated task attempt.
 boolean killTask(TaskAttemptID taskid, boolean shouldFail)
          Mark the task attempt identified by taskid to be killed
 void removeJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)
          Unregisters a JobInProgressListener from this TaskTrackerManager.
 long renewDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
          Renew an existing delegation token
 void reportTaskTrackerError(java.lang.String taskTracker, java.lang.String errorClass, java.lang.String errorMessage)
          Report a problem to the job tracker.
 void setJobPriority(JobID jobid, java.lang.String priority)
          Set the priority of the specified job
 JobStatus submitJob(JobID jobName, java.lang.String jobSubmitDir, org.apache.hadoop.security.Credentials ts)
          Submit a Job for execution.
 java.util.Collection<TaskTrackerStatus> taskTrackers()
           
 
Methods inherited from interface org.apache.hadoop.mr1security.RefreshUserMappingsProtocol
refreshSuperUserGroupsConfiguration, refreshUserToGroupsMappings
 
Methods inherited from interface org.apache.hadoop.mr1tools.GetUserMappingsProtocol
getGroupsForUser
 
Methods inherited from interface org.apache.hadoop.mr1security.authorize.RefreshAuthorizationPolicyProtocol
refreshServiceAcl
 
Methods inherited from interface org.apache.hadoop.mapred.AdminOperationsProtocol
refreshNodes, refreshQueues
 
Methods inherited from interface org.apache.hadoop.ipc.VersionedProtocol
getProtocolSignature, getProtocolVersion
 

Field Detail

versionID

static final long versionID
version 3 introduced to replace emitHearbeat/pollForNewTask/pollForTaskWithClosedJob with heartbeat(TaskTrackerStatus, boolean, boolean, boolean, short) version 4 changed TaskReport for HADOOP-549. version 5 introduced that removes locateMapOutputs and instead uses getTaskCompletionEvents to figure finished maps and fetch the outputs version 6 adds maxTasks to TaskTrackerStatus for HADOOP-1245 version 7 replaces maxTasks by maxMapTasks and maxReduceTasks in TaskTrackerStatus for HADOOP-1274 Version 8: HeartbeatResponse is added with the next heartbeat interval. version 9 changes the counter representation for HADOOP-2248 version 10 changes the TaskStatus representation for HADOOP-2208 version 11 changes string to JobID in getTaskCompletionEvents(). version 12 changes the counters representation for HADOOP-1915 version 13 added call getBuildVersion() for HADOOP-236 Version 14: replaced getFilesystemName with getSystemDir for HADOOP-3135 Version 15: Changed format of Task and TaskStatus for HADOOP-153 Version 16: adds ResourceStatus to TaskTrackerStatus for HADOOP-3759 Version 17: Changed format of Task and TaskStatus for HADOOP-3150 Version 18: Changed status message due to changes in TaskStatus Version 19: Changed heartbeat to piggyback JobTracker restart information so that the TaskTracker can synchronize itself. Version 20: Changed status message due to changes in TaskStatus (HADOOP-4232) Version 21: Changed information reported in TaskTrackerStatus' ResourceStatus and the corresponding accessor methods (HADOOP-4035) Version 22: Replaced parameter 'initialContact' with 'restarted' in heartbeat method (HADOOP-4305) Version 23: Added parameter 'initialContact' again in heartbeat method (HADOOP-4869) Version 24: Changed format of Task and TaskStatus for HADOOP-4759 Version 25: JobIDs are passed in response to JobTracker restart Version 26: Added numRequiredSlots to TaskStatus for MAPREDUCE-516 Version 27: Adding node health status to TaskStatus for MAPREDUCE-211 Version 28: Adding user name to the serialized Task for use by TT. Version 29: Adding available memory and CPU usage information on TT to TaskTrackerStatus for MAPREDUCE-1218 Version 30: Adding disk failure to TaskTrackerStatus for MAPREDUCE-3015 Version 31: Efficient serialization format for Framework counters (MAPREDUCE-901) Version 32: Adding version methods for HADOOP-8209

See Also:
Constant Field Values

TRACKERS_OK

static final int TRACKERS_OK
See Also:
Constant Field Values

UNKNOWN_TASKTRACKER

static final int UNKNOWN_TASKTRACKER
See Also:
Constant Field Values

versionID

static final long versionID
See Also:
Constant Field Values
Method Detail

heartbeat

org.apache.hadoop.mapred.HeartbeatResponse heartbeat(TaskTrackerStatus status,
                                                     boolean restarted,
                                                     boolean initialContact,
                                                     boolean acceptNewTasks,
                                                     short responseId)
                                                     throws java.io.IOException
Called regularly by the TaskTracker to update the status of its tasks within the job tracker. JobTracker responds with a HeartbeatResponse that directs the TaskTracker to undertake a series of 'actions' (see TaskTrackerAction.ActionType). TaskTracker must also indicate whether this is the first interaction (since state refresh) and acknowledge the last response it recieved from the JobTracker

Parameters:
status - the status update
restarted - true if the process has just started or restarted, false otherwise
initialContact - true if this is first interaction since 'refresh', false otherwise.
acceptNewTasks - true if the TaskTracker is ready to accept new tasks to run.
responseId - the last responseId successfully acted upon by the TaskTracker.
Returns:
a HeartbeatResponse with fresh instructions.
Throws:
java.io.IOException

getFilesystemName

java.lang.String getFilesystemName()
                                   throws java.io.IOException
The task tracker calls this once, to discern where it can find files referred to by the JobTracker

Throws:
java.io.IOException

reportTaskTrackerError

void reportTaskTrackerError(java.lang.String taskTracker,
                            java.lang.String errorClass,
                            java.lang.String errorMessage)
                            throws java.io.IOException
Report a problem to the job tracker.

Parameters:
taskTracker - the name of the task tracker
errorClass - the kind of error (eg. the class that was thrown)
errorMessage - the human readable error message
Throws:
java.io.IOException - if there was a problem in communication or on the remote side

getTaskCompletionEvents

TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid,
                                              int fromEventId,
                                              int maxEvents)
                                              throws java.io.IOException
Get task completion events for the jobid, starting from fromEventId. Returns empty aray if no events are available.

Parameters:
jobid - job id
fromEventId - event id to start from.
maxEvents - the max number of events we want to look at
Returns:
array of task completion events.
Throws:
java.io.IOException

getSystemDir

java.lang.String getSystemDir()
Grab the jobtracker system directory path where job-specific files are to be placed.

Returns:
the system directory where job-specific files are to be placed.

getBuildVersion

java.lang.String getBuildVersion()
                                 throws java.io.IOException
Returns the VersionInfo build version of the JobTracker

Throws:
java.io.IOException

getVIVersion

java.lang.String getVIVersion()
                              throws java.io.IOException
Returns the VersionInfo version of the JobTracker

Throws:
java.io.IOException

getNewJobId

JobID getNewJobId()
                  throws java.io.IOException
Allocate a name for the job.

Returns:
a unique job name for submitting jobs.
Throws:
java.io.IOException

submitJob

JobStatus submitJob(JobID jobName,
                    java.lang.String jobSubmitDir,
                    org.apache.hadoop.security.Credentials ts)
                    throws java.io.IOException
Submit a Job for execution. Returns the latest profile for that job. The job files should be submitted in jobSubmitDir.

Throws:
java.io.IOException

getClusterStatus

ClusterStatus getClusterStatus(boolean detailed)
                               throws java.io.IOException
Get the current status of the cluster

Parameters:
detailed - if true then report tracker names and memory usage
Returns:
summary of the state of the cluster
Throws:
java.io.IOException

getQueueAdmins

org.apache.hadoop.security.authorize.AccessControlList getQueueAdmins(java.lang.String queueName)
                                                                      throws java.io.IOException
Get the administrators of the given job-queue. This method is for hadoop internal use only.

Parameters:
queueName -
Returns:
Queue administrators ACL for the queue to which job is submitted to
Throws:
java.io.IOException

killJob

void killJob(JobID jobid)
             throws java.io.IOException
Kill the indicated job

Throws:
java.io.IOException

setJobPriority

void setJobPriority(JobID jobid,
                    java.lang.String priority)
                    throws java.io.IOException
Set the priority of the specified job

Parameters:
jobid - ID of the job
priority - Priority to be set for the job
Throws:
java.io.IOException

killTask

boolean killTask(TaskAttemptID taskId,
                 boolean shouldFail)
                 throws java.io.IOException
Kill indicated task attempt.

Parameters:
taskId - the id of the task to kill.
shouldFail - if true the task is failed and added to failed tasks list, otherwise it is just killed, w/o affecting job failure status.
Throws:
java.io.IOException

getJobProfile

JobProfile getJobProfile(JobID jobid)
                         throws java.io.IOException
Grab a handle to a job that is already known to the JobTracker.

Returns:
Profile of the job, or null if not found.
Throws:
java.io.IOException

getJobStatus

JobStatus getJobStatus(JobID jobid)
                       throws java.io.IOException
Grab a handle to a job that is already known to the JobTracker.

Returns:
Status of the job, or null if not found.
Throws:
java.io.IOException

getJobCounters

Counters getJobCounters(JobID jobid)
                        throws java.io.IOException
Grab the current job counters

Throws:
java.io.IOException

getMapTaskReports

TaskReport[] getMapTaskReports(JobID jobid)
                               throws java.io.IOException
Grab a bunch of info on the map tasks that make up the job

Throws:
java.io.IOException

getReduceTaskReports

TaskReport[] getReduceTaskReports(JobID jobid)
                                  throws java.io.IOException
Grab a bunch of info on the reduce tasks that make up the job

Throws:
java.io.IOException

getCleanupTaskReports

TaskReport[] getCleanupTaskReports(JobID jobid)
                                   throws java.io.IOException
Grab a bunch of info on the cleanup tasks that make up the job

Throws:
java.io.IOException

getSetupTaskReports

TaskReport[] getSetupTaskReports(JobID jobid)
                                 throws java.io.IOException
Grab a bunch of info on the setup tasks that make up the job

Throws:
java.io.IOException

getFilesystemName

java.lang.String getFilesystemName()
                                   throws java.io.IOException
A MapReduce system always operates on a single filesystem. This function returns the fs name. ('local' if the localfs; 'addr:port' if dfs). The client can then copy files into the right locations prior to submitting the job.

Throws:
java.io.IOException

jobsToComplete

JobStatus[] jobsToComplete()
                           throws java.io.IOException
Get the jobs that are not completed and not failed

Returns:
array of JobStatus for the running/to-be-run jobs.
Throws:
java.io.IOException

getAllJobs

JobStatus[] getAllJobs()
                       throws java.io.IOException
Get all the jobs submitted.

Returns:
array of JobStatus for the submitted jobs
Throws:
java.io.IOException

getTaskCompletionEvents

TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid,
                                              int fromEventId,
                                              int maxEvents)
                                              throws java.io.IOException
Get task completion events for the jobid, starting from fromEventId. Returns empty aray if no events are available.

Parameters:
jobid - job id
fromEventId - event id to start from.
maxEvents - the max number of events we want to look at
Returns:
array of task completion events.
Throws:
java.io.IOException

getTaskDiagnostics

java.lang.String[] getTaskDiagnostics(TaskAttemptID taskId)
                                      throws java.io.IOException
Get the diagnostics for a given task in a given job

Parameters:
taskId - the id of the task
Returns:
an array of the diagnostic messages
Throws:
java.io.IOException

getSystemDir

java.lang.String getSystemDir()
Grab the jobtracker system directory path where job-specific files are to be placed.

Returns:
the system directory where job-specific files are to be placed.

getStagingAreaDir

java.lang.String getStagingAreaDir()
                                   throws java.io.IOException
Get a hint from the JobTracker where job-specific files are to be placed.

Returns:
the directory where job-specific files are to be placed.
Throws:
java.io.IOException

getQueues

JobQueueInfo[] getQueues()
                         throws java.io.IOException
Gets set of Job Queues associated with the Job Tracker

Returns:
Array of the Job Queue Information Object
Throws:
java.io.IOException

getQueueInfo

JobQueueInfo getQueueInfo(java.lang.String queue)
                          throws java.io.IOException
Gets scheduling information associated with the particular Job queue

Parameters:
queue - Queue Name
Returns:
Scheduling Information of the Queue
Throws:
java.io.IOException

getJobsFromQueue

JobStatus[] getJobsFromQueue(java.lang.String queue)
                             throws java.io.IOException
Gets all the jobs submitted to the particular Queue

Parameters:
queue - Queue name
Returns:
array of JobStatus for the submitted jobs
Throws:
java.io.IOException

getQueueAclsForCurrentUser

org.apache.hadoop.mapred.QueueAclsInfo[] getQueueAclsForCurrentUser()
                                                                    throws java.io.IOException
Gets the Queue ACLs for current user

Returns:
array of QueueAclsInfo object for current user.
Throws:
java.io.IOException

getDelegationToken

org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> getDelegationToken(org.apache.hadoop.io.Text renewer)
                                                                                     throws java.io.IOException,
                                                                                            java.lang.InterruptedException
Get a new delegation token.

Parameters:
renewer - the user other than the creator (if any) that can renew the token
Returns:
the new delegation token
Throws:
java.io.IOException
java.lang.InterruptedException

renewDelegationToken

long renewDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
                          throws java.io.IOException,
                                 java.lang.InterruptedException
Renew an existing delegation token

Parameters:
token - the token to renew
Returns:
the new expiration time
Throws:
java.io.IOException
java.lang.InterruptedException

cancelDelegationToken

void cancelDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
                           throws java.io.IOException,
                                  java.lang.InterruptedException
Cancel a delegation token.

Parameters:
token - the token to cancel
Throws:
java.io.IOException
java.lang.InterruptedException

taskTrackers

java.util.Collection<TaskTrackerStatus> taskTrackers()
Returns:
A collection of the TaskTrackerStatus for the tasktrackers being managed.

getNumberOfUniqueHosts

int getNumberOfUniqueHosts()
Returns:
The number of unique hosts running tasktrackers.

getClusterStatus

ClusterStatus getClusterStatus()
Returns:
a summary of the cluster's status.

addJobInProgressListener

void addJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)
Registers a JobInProgressListener for updates from this TaskTrackerManager.

Parameters:
jobInProgressListener - the JobInProgressListener to add

removeJobInProgressListener

void removeJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)
Unregisters a JobInProgressListener from this TaskTrackerManager.

Parameters:
jobInProgressListener - the JobInProgressListener to remove

getQueueManager

org.apache.hadoop.mapred.QueueManager getQueueManager()
Return the QueueManager which manages the queues in this TaskTrackerManager.

Returns:
the QueueManager

getNextHeartbeatInterval

int getNextHeartbeatInterval()
Return the current heartbeat interval that's used by TaskTrackers.

Returns:
the heartbeat interval used by TaskTrackers

killJob

void killJob(JobID jobid)
             throws java.io.IOException
Kill the job identified by jobid

Parameters:
jobid -
Throws:
java.io.IOException

getJob

JobInProgress getJob(JobID jobid)
Obtain the job object identified by jobid

Parameters:
jobid -
Returns:
jobInProgress object

initJob

void initJob(JobInProgress job)
Initialize the Job

Parameters:
job - JobInProgress object

failJob

void failJob(JobInProgress job)
Fail a job.

Parameters:
job - JobInProgress object

killTask

boolean killTask(TaskAttemptID taskid,
                 boolean shouldFail)
                 throws java.io.IOException
Mark the task attempt identified by taskid to be killed

Parameters:
taskid - task to kill
shouldFail - whether to count the task as failed
Returns:
true if the task was found and successfully marked to kill
Throws:
java.io.IOException


Copyright © 2009 The Apache Software Foundation