We present a robust two-level architecture for blur-based segmentation of a single image. First network is a fully convolutional encoder-decoder for estimating a semantically meaningful blur map from the full-resolution blurred image. Second network is a CNN-based classifier for obtaining local (patch-level) blur-probabilities. Fusion of the two network outputs enables accurate blur-segmentation using Graph-cut optimization over the obtained probabilities. We also show its applications in blur magnification and matting.