Background modeling is one of the most challenging and time consuming tasks in moving object detection for video surveillance. In this paper, we present a new algorithm which does not require any background model. Instead, it utilizes three most recent consecutive frames to detect the presence of moving object by extracting moving edges. In the proposed method, we introduce an edge segment based approach instead of traditional edge pixel based approach. We also utilize an efficient edge-matching algorithm which reduces the variation of edge localization in different frames. Finally, regions of the moving objects are extracted from previously detected moving edges by using an efficient watershed based segmentation algorithm. The proposed method is characterized through robustness against the random noise, illumination variations and quantization error and is validated with the extensive experimental results.