In this research, the effect of nonlinear behavior of concrete on seismic performance of roller compacted concrete (RCC) is discussed. Nonlinear behavior of material shows the behavior close to reality due to use of all structural capacity and it is considered the most appropriate method for design. So in this paper, by using the appropriate definition of nonlinear behavior of RCC, the Ansys software based of finite element method is used for modeling and analysis considering dam-reservoir-foundation interaction. Consider to behavior of roller concrete dam, two-dimensional model is selected and the dynamic analysis is carried out in time domain applying the Northridge earthquake components for linear and nonlinear cases. For evaluation of the effect of nonlinear behavior on the output results, the maximum hydrodynamic pressure, dam crest displacement, tensile stress in heel and compressive stress in the toe of dam are selected as critical responses. Finally, for accurate evaluation of the effect of nonlinear behavior on output responses, the results has been obtained as time history curves for both linear and nonlinear cases and the difference of responses has been determined.