Segment Everything with SAM#
In this section we
Use Napari to visualize the overlapping labels generated by SAM
Filter the (potentially) hundreds or thousands of labels generated by SAM
What is SAM ?#
SAM stands for Segment Anything Model. SAM is a prompt based generalist image segmentation system. A prompt is a hint as to what object(s) to segment. A prompt can be a point (click on an object), a bounding box, or even a pixel mask (first guess segmentation).
The input to sam is a prompt and an image and the output is a pixel mask. There are many variations of SAM. Mobile SAM for example uses a custom YOLO detector to generate a bounding boxes prompt.
SAM is very good at detecting overlapping objects (think spots on the lady-bugs). We can give SAM multiple prompts and it will return multiple segmentation masks. If we want a conventional ‘2D’ segmentation we have to transform the mask collection to a 2D image (for example by projecting).
