Selecting & Masking - a little S&M Tutorial

[Tom V]

I appologize in advance.

Any object in an image or part of an image can be selectively edited (filtered, sharpened, deleted, retouched, duplicated, etc.) once you have selected it. The pixels inside the selection can be modified, while the pixels outside the selection remain untouched. The selection area can be modified to have a feathered edge, expanded in area by a number of pixels, added to, removed from, intersected, etc.

A selection in Photoshop is shown by the flashing marquee or "dancing ants" line that overlays your image on the screen. The marquee or "ants" do not print. The marquee line defines the area in which the enclosed pixels are available for editing. You can hide / unhide the selection marquee line by [keyboard H] or in the menu: VIEW > SHOW > EXTRAS. If you have guidelines or a grid showing, they too are hidden this way. Even if your selection marquee is hidden, it acts just as if it were visible, allowing editing to occur only within its borders.

I use Photoshop's "Quick Mask" feature as well as ordinary layer masking to the make my selections. I find it far easier to make a mask that I turn into a selection, than it is to do direct selecting using the Lasso, Magnetic Lasso, Eraser, or even the "automatic masking" features built into Photoshop. I have tried the third-party masking plug-ins like Mask Pro, and found my method easier to learn, easier to control, more intuitive, faster, more forgiving, cheapest, and... ah, therefore, best.

I find that the Lasso tool, and other related tools make a razor sharp selections that are very unnatural looking in the middle of a photographic image. You can feather the selection, but I find that sort of a guessing game as to how soft the selection edge is going to be. By making a mask, you can not only see how sharp the edge of the selection will be, but you can alter it, and even have one part of the selection edge sharper than other parts of the selection. You can also have varing degrees of transparency.

I wanted to see how this tutorial would help Elements users so I installed Elements today and learned a few things. There is no Quick Mask or direct way of doing masking, but there is a way to do something similar with Layer Fills. There is no Curves adjustment (YIKES! - I use that all the time in Photoshop, but there is a Fill-Flash adjustment that can substitute somewhat. About the worst features to use in Photoshop are Brightness and Contrast, which are also in Elements. Without Curves, I am afraid Elements users will rely on Brightness and Contrast too much trying to save marginal images, or enhance good ones. Elements users should learn Levels and Fill-Flash to correct images, and should avoid Brightness and Contrast for most images.

For this tutorial, I will use the viola that was offered for our Forum's first Image Editing Challenge. ( See http://www.s2pro.com/forums/showthread.php?s=&threadid=1977 ) This tutorial will show you how I selected the viola, and was able to work with it independantly of the background. Once the viola is selected, it is easy to open up the shadows, replace the background, make drop shadows, dodge and burn highlights and shadows, retouch the obscured tuning knob, etc.

This tutorial goes through some extra steps because the original is so dark on the near edge. Some extra steps are required to see the edge before the mask / selection can be made. Since the shadow edge of the viola is nearly blended in with the background, it is amazing when you see the viola removed from the background so cleanly. Not all jobs are this hard to mask and select.

[Tom V]

Words in all caps are listings in the program's menus. MENU > SUBMENU > ITEM. Letters in brackets [W] are keyboard shortcuts.

I use the Brush Tool all the time, and made several organized sets of uniform, custom brushes that I find better than the default Photoshop brushes. You can add the brushes I use to your Brushes palette by downloading the .abr file as talked about in: [http://www.fmount.com/forums/showthread.php?t=1989] I would also suggest you set your Cursors and Display preferences as I discuss in that thread. Not being able to see your brush size diameter as you work makes all of the following nearly impossible.

I downloaded the image and opened it in Photoshop 7. I rotated the image [IMAGE > ROTATE CANVAS > 90° CCW]. The file is small at 900 x 600 pixels, so I resized it to work on IMAGE > IMAGE SIZE ( • Resample Image, Bicubic / • Constrain Proportions and type 1800 in the top Pixel Box.) When I am done with the image, I will resample the image down to 900 x 600.

In order to select JUST the viola, you have to be able to see it. By lightening it up sufficiently to really get a good look at it, you would have to nearly ruin the picture. So let's "ruin" a copy of it. LAYER > DUPLICATE LAYER Click OK. I check the Levels IMAGE > ADJUSTMENT > LEVELS and note that the image contains a huge mountain of black and nearly black pixels, and a trickle of mid-tones, light tones, nearly whites, and whites. Just below the histogram, click on the gray triangle in the middle and slide it to the left, under the mountain of dark pixels. As you move the slider, the image's mid-tones will lighten, but the pure blacks stay black, and the pure whites stay white. The number in the top center box is changing too. Make it 4.00 (you can even type it in).

[Tom V]

Get into Quick Mask Mode. In the Tool Palette, click on the rightmost icon below the Foreground / Background Color boxes, -or- [Q] to toggle in and out of Quick Mask Mode. Quick Mask Mode (QMM) makes a temporary extra channel that is used for masking. Channels contains the equivalent of 'grayscale' information, so it can have 256 shades of gray, or density. When you paint in QMM, your image is obscured by a colored mask. When you click out of QMM, a selection is created whereever you have painted. Click back into QMM and whatever selection was active, now is the mask. It works both ways: Selection to Mask, Mask to Selection. To set up your QMM Options, double-click on the QMM Icon in the Tools Palette. Photoshop's default QMM color is red, but since the viola is red, change it to green for better visual contrast. Click on the color box, and the Color Picker dialog box opens. Select • R=0, • G=255, •B=0 and close the window. In the Quick Mask Options, • Selected Areas, Opacity 80%, and close the window. You can always change the settings in these windows to suit the image's needs.

[Tom V]

Start Masking.
Zoom in on the image to 400% so you can really see the finest details in the image. I started at the viola's waist, where the wood has relatively sharp points. I find that I have to pick out my brush size initially based on how "sharp" the subject's corners are. Use the Brush Tool [B] and pick out a 7 pixel brush size with 100% hardness. If you use "Tom's Uniform Brushes.abr" it is in the first set of 1-300 pixel brushes. Click and drag around in the viola area to get a feel for the brush size. Start your serious brushing on the corner of the wood by clicking the round brush so it fits the contour of the wood. Move the cursor a few brush diameters, hold [SHIFT] and click again. A solid line will draw from the initial click to where you [SHIFT]-clicked. You will need to go around the entire viola like this.

You can download, install, load, and use the brushes I use. Go to http://users.foxvalley.net/~tomvoegeli/Brushes/index.html (link updated Feb. 23, 2004)

[Tom V]

Keep the brush just inside the edge of the viola so you do not cover any background. If the very edge of the viola gets cut off, nobody will notice. If you leave just a little background in, everyone will notice. Just be smoooooth.
On straighter sections, you can make bigger line segments, on curves you need shorter segments. You can do the bow hair in just a couple [SHIFT]- clicks - one on each end of each side

If you do cover background accidently, or have a section you want to redo, you can un-paint just as easy. Switch the Foreground/Background Colors in the Tool Palette with [X]. Instead of adding to the mask, it takes it away.

[Tom V]

When you get to the chin rest, note that you can see background between the viola and the metal rods. Paint the viola, then go back and do the metal rods with a 5 pixel brush (7 is too fat). When you get to the strings, ignore them and continue with the viola body & finger board shape - come back to the straight strings later. Where the bow tip sinks into the carpet, fake it - make a tip shaped like a laundry iron. Where the stand obscures the viola, just imagine where the viola is and use "that" line. Now go back and do the strings. The strings are straight, so you only need to click at the ends. I used a 3 pixel brush that I try to center on the string. The jpg artifacts make this tricky, because it makes the strings look fat. I start my [CLICK]-shift lines well 'inside' the viola shape. All the extraneous brush marks get covered up next.

[Tom V]

Start filling it in. Select a fatter brush, like 20 pixels and [SHIFT]-click around the inside of your mask, making sure to get in the corners without "going outside the lines." Use a smaller brush for the bow, as 20 is too fat. Sometimes I just click dots in the corners, rather than [SHIFT]-clicking. Then go around again with a fatter brush and get everything filled in. The more you can do with a fatter brush, the quicker you will be. But be careful you don't miss any little spots. You can easily move from one brush to the next with [ [ ] and [ ] ] (the brackets that I type these brackets with).

You may be tempted to use the Bucket Fill tool. Don't. It causes a bunch of little problems that are hard to correct. It does not fill in good enough, and too many attempts to fill everything in is bound to give the overall shape a jagged edge. The paint bucket hasn't a soft edge like a brush, and it is hard to make it compatible with brush softness.

[Tom V]

Once it is all filled in, it should look like this.

[Tom V]

Once it is all filled in, get out of QMM [Q] and all the painted area becomes your selection. There is a marquee around your viola and bow. If you hit [DELETE], the viola will be gone from the layer, and you will see the original viola on the bottom layer through the hole. Try this to see what happens, then undo it [Z]. To remove the background, inverse the selection to get rid of everything except the viola. SELECTION > INVERSE. You will still have the marquee around the viola, but notice now that the marquee also goes around the edge of the image - the "background" (everything but the viola) is selected. Now get rid of it [DELETE]. Now you have the really light viola on its own layer, and the dark original background showing below. You can now deselect your selection [D] and the marquee goes away. Your work on making the selection is not lost because the remaining shape on the "really light" layer can be made into a selection with just a click at any time.

[Tom V]

At this time, I like to check my work by hiding the distracting background. Make a new layer in the middle and fill it with white. Make the top layer active, and examine the viola shape closely and erase any background edges that might have gotten in. There are ways to get back into QMM and do it all in masking, but the Eraser is fine for a few little things here and there. On my example, I erased some fat edges on the bow, the strings, and a little off the top edge of the body. When you are happy with the shape of the viola, delete the middle layer. If all worked good, there is now a perfectly shaped viola & bow extracted from the background. The too-light layer we have been working on has served its purpose.

Let's make a good viola!
The idea is to make a selection out of the too-light viola, make the dark viola's layer active, and copy the dark viola in the selection area to bring it to a new layer. (How did I ever think I would explain this?)
[COMMAND]-click on the upper layer thumbnail in the Layers Palette. (On a Mac, [COMMAND] is the key next to the space bar. Some people call it the Apple Key. Beats me what it is on Windows. This makes the shape on that layer into a selection. The marquee is once again around the viola. Hide the top layer by clicking on the Eye icon to the left of the layer thumbnail in the Layers Palette. Activate the bottom layer. Copy the contents of the selection [COMMAND]+[C]. Now Paste [COMMAND]+[V], and a new layer is formed with just the pasted "dark" viola. Hide the bottom layer so the only thing showing is the original "dark" viola (on the middle layer). Make sure this new dark viola layer is active.

Check the Levels [COMMAND]+[L], and you will see that the viola has a good range of pixels. Close the Levels window and go to the Curves adjustment [COMMAND]+[M]. Click the cursor on the viola, and you will see and indicator on the Curves graph. Note that clicking on the dark viola wood puts an indicator in the lower left corner. We need to move the indicator straight up to make those tones lighter. Click on the indicator and move it up, the curve changes and you can read what is happening in the Input: / Output: readouts. If you just move the shadows and midtones up, they will start washing out, and the color will wash out of the light tones. I like to keep the very darkest and lightest tones about where they are, to keep what richness is there, and because the original already has enough contrast. The dark tones and midtones will be adjusted independently to bring out the detail.

The steeper the line, the more contrast - the flatter the line, the less contrast.

I "pin" the very darkest and very lightest tones where they are. I put a dot on Input: 10 / Output: 10, and on Input: 240 / Output: 240. To lighten the shadow side of the viola, I move a dot to: Input: 30 / Output: 60, which puts a wild curve in the midtones. The steepness of the curve means that there is increased contrast in the shadows and midtones, and the flat highlights means less contrast. With the highlights so high, a lot of color is washed out in my opinion. Reel the wild curve back in with a dot at Input: 62 / Output: 112, which straightens out the curve. The further the curve is from straight, the fewer of the original 256 tones (per channel) are retained. Less tones means less detail overall. I am happy with the Curves, so hit OK.

(I keep my black pixels to the lower left, if your window shows them on the right, you can switch it with a click on the little double arrowhead in the center of the lower "gray value" bar below the graph.)

The bow goes from nice to dingy. I fixed this with the Dodge Tool [O] in the Tools Palette. I used a very soft 100 pixel brush from the lowest set of my brushes. I set the Dodge controls to "Midtones" and 25% Exposure. I dodged back and forth a few times to lighten up the dingy end of the bow.

I will omit the retouching of the tuning knobs and bow tip. This is getting long. There are dozens of ways to retouch.

[Tom V]

Make the background. (Moving along faster now)
Make a new layer beneath the good viola. Select two colors for the background "clouds" by clicking first on the Foreground Color Swatch on the Tools Palette. This brings up the Color Picker window which I ignore. Instead, click on the viola in the rich tone under the chin rest. Click OK to close the Color Picker window. Switch colors [X] and repeat for the other color, this time choosing a lighter color, like the one on the far side of the finger board. Make the program render the clouds using the colors FILTER > RENDER > CLOUDS. Try several renderings with [F] (redoes last filter) to get a feel for the variations. I picked one that didn't look like it had too much of a pattern. Since the pattern looks uniform top to bottom, it looks like a flat wall to me. Put some depth in it by distorting it. Select All [COMMAND]+[A], EDIT > TRANSORM > PERSPECTIVE and pull the bottom corner handles (of the selection) out about 200%. Hit [ENTER] to set the transformation.

Make a new layer. Set the Foreground/Background colors to default [D], and use the Gradient Tool [G] with a linear Foreground to Background blend, dragging from top to bottom. This layer goes over the clouds, and below the viola. Set the gradient layer's opacity to 50% and change the Mode to Multiply (all in the Layers Palette). Now the background has perspective, and depth.

Make the viola float.
Activate the viola layer and click on the ƒ icon at the bottom of the Layers Palette. (Or, LAYERS > LAYER STYLE > DROP SHADOW). In the Layer Style dialog box, set Blend Mode to Multiply (the default), Opacity: 25%, Angle 90°, Distance: 175 px (pixels), Spread: 0, Size: 50.

Do some sharpening if you wish. I use USM at 100%, 1, 0 for this web quality image.

Save and Finish Up.
If you want to save the file layers (with the selection, editable shadow, original and all the other layers to work on later), you can save it as a .psd file. To make the file useable and viewable in other places, you should flatten it LAYERS > FLATTEN IMAGE (discard hidden layers) and save it however you want. I flattened mine, and resized it down in IMAGE > IMAGE SIZE.

[dhani2489]

Thank you Tom,

Thanks for your valuable time and superb talent.

:) :)
dhani

[Tom V]

Originally posted by dhani2489
Thank you Tom,

Thanks for your valuable time and superb talent.

:) :)
dhani

dhani,

You're a thousand times welcome!

-----

By the way, if anyone is inerested in copying what I have written, I have posted the text file I used for all these tutorial posts online in my hack webspace. http://users.myexcel.com/tomvoegeli/selecting/
All I ask is that if you turn my manuscript into a movie screenplay, I get 10% of gross.

[sandman]

Well detailed tutorial Tom , i could'nt have explained all that over the net in a million years.
Just one thing to add , all you out there with ELEMENTS there is a selection brush in ELEMENTS 2, just change the mode to ''mask'' and the colour to green (if you want) and you can use it like Tom suggested.

Brian

[toners]

Oh! Now I see how your eyes got that way.

Levity aside, Tom, a thousand times thanks.

[Andre]

Thanks Tom.
Another example of what makes this forum so great - someone such as yourself taking the time out to spell out something like that.

[okidoki]

I love PS, something new every day...

I did a somewhat different tutorial here:

http://www.timojulku.com/selection-tutorial/index.html

As many ways to do this as opinions...

Thanks Tom, gotta try it out...

Timo

[Swampy]

great job Tom. You could have masked a little more and a little better, but you got the job done, and you probably got it done a while lot quicker than anyone else could have here, so I'll cut some slack there. hahah.

Well done though. We, even me, all appreciate it!

[LinhLe]

Thank you Tom.

Lính Lệ

[RacerX]

Nicely, nicely, done. Thanks for doing that, Tom!
Plus thanks to Timo for showing a different way.

Thanks guys!

Rick

[jknights]

Tom,

As ever. Thanks for your expert PS tutorial.

Now I just have to practise to get somewhere near as good as you ! ;-)

[Linda G]

Tom,

I do this on a daily basis and since there are as many ways to do it as opinions, this one is a great one.

I use whichever method I'm in the mood for and some methods work better with different subjects.

My BEEF here is:

It's NOT dancing ants! It's MARCHING ants.

:mad:

;)

Honestly, a more thourough and complete tutorial couldn't be found elsewhere on the net. Great job! And big thanks!

Now, got one for hair and feathery treelines? (another daily headache)

Linda

[Melody]

Just another thank you Tom, this is more complete and easier to follow on a very detailed subject then any book I have (which is several :lol: ), I know I've mentioned it however you write tutorials very well. :)

Now back to my cursing and going blind, uh I'm not going to do this long enough to get 4 eyes! :eek: ;)

Melody

[S_Leeper]

I'm not really into S&M...
But I do want to Thank You for such an effective tutorial. I've had little time to invest in learning PS & this is one area that I want to start exploring.

[HairyHaggis]

Cool, this is a question I have always felt embarrased to ask, as like most of my lack of photoshop skills. Thanks Tom :goldcup:

[Melody]

Doesn't Tom do a fabulous job, and if you'll notice this tutorial is 2yrs OLD, just use the handydandy search feature on the board put in his name and tutorial and woohoo there you go more great stuff! This is rather nice since it can be used for folks who haven't upgraded to PS CS2 maybe if we are patient we can see what he's got in mind for us on that too!

Melody

[LiVe]

Tom you help me out so much, it inspires me too. thanks for that tutorial

[Serge]

An excellent Tutorial document Tom, I'm going to print it out in color, even though it's 4 years down the track :D

[Tom V]

I think this method still works best. I still use it to mask motorcycles, people, and nearly anything. The Extraction tool in Photoshop still cannot correctly mask chrome, black on black, textures at the edge on some materials, etc. Using the appropriate brush on a quick mask is still the best way.

[sandman]

Have you tried the new quick selection brush in CS3 yet Tom .
I've only had it 2 weeks so i'm still learning the new techniques involved but it's certainly easier than the extract tool IMO .

Brian

[Tom V]

Have you tried the new quick selection brush in CS3 yet Tom .
I've only had it 2 weeks so i'm still learning the new techniques involved but it's certainly easier than the extract tool IMO .

Brian

Brian,

Nope. Not yet. I should amend my comment 2 posts previous that Photoshop still cannot..." I should say that As far as I know, and with CS2, Photoshop cannot...

I do plan on getting the full boat Master Collection of CS3, along with an upgrade to my computer power: Apple MacPro with 16GB of ram, in February (New MacPros will be coming out in mid-January). Finally.