In this paper, we present an automatic method based on context-dependent characteristics for the detection and classification of arterial vessels and venous vessels in retinal fundus images. It provides a non-invasive opportunity and effective foundation for the diagnosis of several medical pathologies. In the proposed method, a combination of shifted filter responses is used, which can selectively respond to vessels. It achieves orientation selectivity by computing the weighted geometric mean of the output of a pool of Difference-of-Gaussian filters, whose supports are aligned in a collinear manner. We then configure two combinations of shifted filters, namely symmetric and asymmetric, that are selective for bars and bar-endings, respectively. We achieve vessel detection by summing up the responses of the two filters. Then we extract the morphology and topological characteristics based on the vessel segmentation, and specifically present context-dependent features of blood vessels, including the shape, structure, relative position, context information and other important features. Based on these features, we use JointBoost classifier to construct potential function for conditional random fields (CRFs) model, and train the labeled samples to classify arteriovenous blood vessels in retinal images. The training and testing data sets were prepared according to the results based on DRIVE dataset provided by Estrada et al. The experimental results show that the accuracy of the proposed method for vein and artery detection is 91.1% and 94.5%, respectively, which is superior to that of the state-of-the-art methods. It can be used as a clinical reference for computer-assisted quantitative analysis of fundus images.