Detecting unknown objects such as lost cargo is essential for improving the safety of self-driving cars. This is the first work focusing on reducing the computational cost of discrepancy networks for unknown object detection on monocular camera images. We propose an efficient discrepancy networks based solely on semantic segmentation, which has 50% fewer parameters and is 140% faster inference speed compared to an existing method, while improving detection performance by a large margin. In a major departure from prior work, we remove GANs from discrepancy networks. While previous studies have used GANs as a necessary component, our model outperforms them without using it. We further improve detection performance by analyzing intermediate representations and introducing feature selection and deep supervision. Our experiments on three datasets for obstacle detection show significant improvement of more than 5% in AUROC.