Abstract:Different Job requires different memory resource, it is difficult to assess the rationality for a memory allocation to a MapReduce Job. Trying to solve this problem, we present an assessment method and recommendation for memory settings of JVM where Job’s tasks run. Firstly, we introduce some important GC metrics based on our analyzing JVM’s memory allocation and GC workflow in-depth; Then, we introduce three kinds of indicators and memory allocation rationality evaluation method based on the three indicators; Finally, we introduce two kinds of optimal JVM configuration recommendations: By using of K-means algorithm and statistical information to estimate the threshold value of the object size which should have been allocate in old generation; By modeling GC pause time and using search algorithm to predict the size of young generation and the old generation. Experimental results show that the proposed approach can automatically find insufficient of memory configuration of a Job. Compared with using machine learning methods, the proposed method does not need to run a large number of test cases, so the proposed method can apply to production cluster of MapReduce.