org.apache.hadoop.mapred
Class TaskLog

java.lang.Object
  extended by org.apache.hadoop.mapred.TaskLog

public class TaskLog
extends java.lang.Object

A simple logger to handle the task-specific user logs. This class uses the system property hadoop.log.dir. This class is for Map/Reduce internal use only.


Nested Class Summary
static class TaskLog.LogName
          The filter for userlogs.
 
Field Summary
static org.apache.hadoop.fs.PathFilter USERLOGS_PATH_FILTER
          Path filter that filters out userlogs directory.
 
Constructor Summary
TaskLog()
           
 
Method Summary
static java.lang.String addCommand(java.util.List<java.lang.String> cmd, boolean isExecutable)
          Add quotes to each of the command strings and return as a single string
static java.util.List<java.lang.String> captureDebugOut(java.util.List<java.lang.String> cmd, java.io.File debugoutFilename)
          Wrap a command in a shell to capture debug script's stdout and stderr to debugout.
static java.util.List<java.lang.String> captureOutAndError(java.util.List<java.lang.String> cmd, java.io.File stdoutFilename, java.io.File stderrFilename, long tailLength)
          Wrap a command in a shell to capture stdout and stderr to files.
static java.util.List<java.lang.String> captureOutAndError(java.util.List<java.lang.String> setup, java.util.List<java.lang.String> cmd, java.io.File stdoutFilename, java.io.File stderrFilename, long tailLength)
          Wrap a command in a shell to capture stdout and stderr to files.
static java.util.List<java.lang.String> captureOutAndError(java.util.List<java.lang.String> setup, java.util.List<java.lang.String> cmd, java.io.File stdoutFilename, java.io.File stderrFilename, long tailLength, boolean useSetsid)
          Wrap a command in a shell to capture stdout and stderr to files.
static java.util.List<java.lang.String> captureOutAndError(java.util.List<java.lang.String> setup, java.util.List<java.lang.String> cmd, java.io.File stdoutFilename, java.io.File stderrFilename, long tailLength, boolean useSetsid, java.lang.String pidFileName)
          Deprecated. pidFiles are no more used. Instead pid is exported to env variable JVM_PID.
static java.util.List<java.lang.String> captureOutAndError(java.util.List<java.lang.String> setup, java.util.List<java.lang.String> cmd, java.io.File stdoutFilename, java.io.File stderrFilename, long tailLength, java.lang.String pidFileName)
          Deprecated. pidFiles are no more used. Instead pid is exported to env variable JVM_PID.
static void createTaskAttemptLogDir(TaskAttemptID taskID, boolean isCleanup, java.lang.String[] localDirs)
          Create log directory for the given attempt.
static java.io.File getJobDir(JobID jobid)
          Get the user log directory for the job jobid.
static java.io.File getJobDir(java.lang.String jobid)
          Get the user log directory for the job jobid.
static java.lang.String getTaskAttemptLogDir(TaskAttemptID taskID, java.lang.String cleanupSuffix, java.lang.String[] localDirs)
          Get attempt log directory path for the given attempt-id under randomly selected mapred local directory.
static java.io.File getTaskLogFile(TaskAttemptID taskid, boolean isCleanup, TaskLog.LogName filter)
           
static long getTaskLogLength(JobConf conf)
          Get the desired maximum length of task's logs.
static java.io.File getUserLogDir()
           
static void syncLogs(java.lang.String logLocation, TaskAttemptID taskid, boolean isCleanup, boolean segmented)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

USERLOGS_PATH_FILTER

public static final org.apache.hadoop.fs.PathFilter USERLOGS_PATH_FILTER
Path filter that filters out userlogs directory.

Constructor Detail

TaskLog

public TaskLog()
Method Detail

createTaskAttemptLogDir

public static void createTaskAttemptLogDir(TaskAttemptID taskID,
                                           boolean isCleanup,
                                           java.lang.String[] localDirs)
                                    throws java.io.IOException
Create log directory for the given attempt. This involves creating the following and setting proper permissions for the new directories
{hadoop.log.dir}/userlogs/
{hadoop.log.dir}/userlogs//
{one of the mapred-local-dirs}/userlogs/
{one of the mapred-local-dirs}/userlogs//

Parameters:
taskID - attempt-id for which log dir is to be created
isCleanup - Is this attempt a cleanup attempt ?
localDirs - mapred local directories
Throws:
java.io.IOException

getTaskAttemptLogDir

public static java.lang.String getTaskAttemptLogDir(TaskAttemptID taskID,
                                                    java.lang.String cleanupSuffix,
                                                    java.lang.String[] localDirs)
                                             throws java.io.IOException
Get attempt log directory path for the given attempt-id under randomly selected mapred local directory.

Parameters:
taskID - attempt-id for which log dir path is needed
cleanupSuffix - ".cleanup" if this attempt is a cleanup attempt
localDirs - mapred local directories
Returns:
target task attempt log directory
Throws:
java.io.IOException

getTaskLogFile

public static java.io.File getTaskLogFile(TaskAttemptID taskid,
                                          boolean isCleanup,
                                          TaskLog.LogName filter)

syncLogs

public static void syncLogs(java.lang.String logLocation,
                            TaskAttemptID taskid,
                            boolean isCleanup,
                            boolean segmented)
                     throws java.io.IOException
Throws:
java.io.IOException

getTaskLogLength

public static long getTaskLogLength(JobConf conf)
Get the desired maximum length of task's logs.

Parameters:
conf - the job to look in
Returns:
the number of bytes to cap the log files at

captureOutAndError

public static java.util.List<java.lang.String> captureOutAndError(java.util.List<java.lang.String> cmd,
                                                                  java.io.File stdoutFilename,
                                                                  java.io.File stderrFilename,
                                                                  long tailLength)
                                                           throws java.io.IOException
Wrap a command in a shell to capture stdout and stderr to files. If the tailLength is 0, the entire output will be saved.

Parameters:
cmd - The command and the arguments that should be run
stdoutFilename - The filename that stdout should be saved to
stderrFilename - The filename that stderr should be saved to
tailLength - The length of the tail to be saved.
Returns:
the modified command that should be run
Throws:
java.io.IOException

captureOutAndError

public static java.util.List<java.lang.String> captureOutAndError(java.util.List<java.lang.String> setup,
                                                                  java.util.List<java.lang.String> cmd,
                                                                  java.io.File stdoutFilename,
                                                                  java.io.File stderrFilename,
                                                                  long tailLength)
                                                           throws java.io.IOException
Wrap a command in a shell to capture stdout and stderr to files. Setup commands such as setting memory limit can be passed which will be executed before exec. If the tailLength is 0, the entire output will be saved.

Parameters:
setup - The setup commands for the execed process.
cmd - The command and the arguments that should be run
stdoutFilename - The filename that stdout should be saved to
stderrFilename - The filename that stderr should be saved to
tailLength - The length of the tail to be saved.
Returns:
the modified command that should be run
Throws:
java.io.IOException

captureOutAndError

@Deprecated
public static java.util.List<java.lang.String> captureOutAndError(java.util.List<java.lang.String> setup,
                                                                             java.util.List<java.lang.String> cmd,
                                                                             java.io.File stdoutFilename,
                                                                             java.io.File stderrFilename,
                                                                             long tailLength,
                                                                             java.lang.String pidFileName)
                                                           throws java.io.IOException
Deprecated. pidFiles are no more used. Instead pid is exported to env variable JVM_PID.

Wrap a command in a shell to capture stdout and stderr to files. Setup commands such as setting memory limit can be passed which will be executed before exec. If the tailLength is 0, the entire output will be saved.

Parameters:
setup - The setup commands for the execed process.
cmd - The command and the arguments that should be run
stdoutFilename - The filename that stdout should be saved to
stderrFilename - The filename that stderr should be saved to
tailLength - The length of the tail to be saved.
Returns:
the modified command that should be run
Throws:
java.io.IOException

captureOutAndError

@Deprecated
public static java.util.List<java.lang.String> captureOutAndError(java.util.List<java.lang.String> setup,
                                                                             java.util.List<java.lang.String> cmd,
                                                                             java.io.File stdoutFilename,
                                                                             java.io.File stderrFilename,
                                                                             long tailLength,
                                                                             boolean useSetsid,
                                                                             java.lang.String pidFileName)
                                                           throws java.io.IOException
Deprecated. pidFiles are no more used. Instead pid is exported to env variable JVM_PID.

Wrap a command in a shell to capture stdout and stderr to files. Setup commands such as setting memory limit can be passed which will be executed before exec. If the tailLength is 0, the entire output will be saved.

Parameters:
setup - The setup commands for the execed process.
cmd - The command and the arguments that should be run
stdoutFilename - The filename that stdout should be saved to
stderrFilename - The filename that stderr should be saved to
tailLength - The length of the tail to be saved.
useSetsid - Should setsid be used in the command or not.
Returns:
the modified command that should be run
Throws:
java.io.IOException

captureOutAndError

public static java.util.List<java.lang.String> captureOutAndError(java.util.List<java.lang.String> setup,
                                                                  java.util.List<java.lang.String> cmd,
                                                                  java.io.File stdoutFilename,
                                                                  java.io.File stderrFilename,
                                                                  long tailLength,
                                                                  boolean useSetsid)
                                                           throws java.io.IOException
Wrap a command in a shell to capture stdout and stderr to files. Setup commands such as setting memory limit can be passed which will be executed before exec. If the tailLength is 0, the entire output will be saved.

Parameters:
setup - The setup commands for the execed process.
cmd - The command and the arguments that should be run
stdoutFilename - The filename that stdout should be saved to
stderrFilename - The filename that stderr should be saved to
tailLength - The length of the tail to be saved.
useSetsid - Should setsid be used in the command or not.
Returns:
the modified command that should be run
Throws:
java.io.IOException

addCommand

public static java.lang.String addCommand(java.util.List<java.lang.String> cmd,
                                          boolean isExecutable)
                                   throws java.io.IOException
Add quotes to each of the command strings and return as a single string

Parameters:
cmd - The command to be quoted
isExecutable - makes shell path if the first argument is executable
Returns:
returns The quoted string.
Throws:
java.io.IOException

captureDebugOut

public static java.util.List<java.lang.String> captureDebugOut(java.util.List<java.lang.String> cmd,
                                                               java.io.File debugoutFilename)
                                                        throws java.io.IOException
Wrap a command in a shell to capture debug script's stdout and stderr to debugout.

Parameters:
cmd - The command and the arguments that should be run
debugoutFilename - The filename that stdout and stderr should be saved to.
Returns:
the modified command that should be run
Throws:
java.io.IOException

getUserLogDir

public static java.io.File getUserLogDir()

getJobDir

public static java.io.File getJobDir(java.lang.String jobid)
Get the user log directory for the job jobid.

Parameters:
jobid - string representation of the jobid
Returns:
user log directory for the job

getJobDir

public static java.io.File getJobDir(JobID jobid)
Get the user log directory for the job jobid.

Parameters:
jobid - the jobid object
Returns:
user log directory for the job


Copyright © 2009 The Apache Software Foundation