Let me say at the outset that is is perhaps not a particularly original method. It is a combination of existing methods, plus a few things I have gleaned from "looking over" the Gimptalk Manager saulgoode's shoulder as he answered questions posted by others in the Gimptalk Help Forum.

The two extant methods of doing "renders", or removing backgrounds from images, the Paths Method (by the legendary fencepost) and the Quickmask Method (by Manager swmiller6), while quite workable for some subjects, have certain drawbacks which I find troublesome at times.

The Paths Method is perhaps perceived as being the easiest by most people trying to do renders, and for many subjects, especially those with less complex outlines, it is quite easy. However, many who use it do not take full advantage of the ability of paths to form curves and instead use an excessive number of control nodes which basically can amount to a LOT of little straight lines that may not accurately cut the contour of the image they are trying to render. I am also not sure what the effect of that many control nodes has on filesize and/or computer response time.

The QuickMask Method allows, in my opinion, for more control of fine detail. I personally also find that "painting" things in is easier than trying to bend the lines between the control nodes when using the Paths Method so as to conform the line to the contour of the image. Indeed, perhaps it is the sometime trickiness of this bending that causes so many to use lots of control nodes when doing renders.

As I see it, both methods suffer from two problems. One, you don't get to see your render until after you convert the path to a selection and actually cut the image, or convert the Quickmask to a selection and do the same. If you missed an area or maybe didn't quite get things right, then you have to either go back and clean things up with the eraser tool, or do "undo" a couple of times to get back to your original rendering setup and then tweak the control nodes or adjust the Quickmask.

The second issue is that if you have something you are rendering that has a combination of soft and hard edges, you are pretty much limited (without a lot of manual hassling) in how you can deal selectively with these edges. This perhaps applies more to rendering photographs than things like in-game screen shots or line art. So you do your render, do the shrink and feather thing and hope that you don't have any really ragged edges, or an edge that looks "unnatural".

What if there were another way? What if the method allowed you to see your render in "realtime" as you rendered, so you knew exactly how it was going to look without having to cut and then tweak or "undo" and tweak the final image? What if it allowed you to deal with both soft and hard edges as you rendered? To actually "feather" your render on the fly as you worked and see it against a potential background while you were rendering?

This is what the Layer Mask method allows you to do.

Layer masks certainly aren't new and many of you probably already know something of how to use them. Maybe you have even been using them to render stuff already.

But if you haven't, this tutorial may be of interest.

First, we need to do a little preparation.

Step 1. In the brushes dialog. Click on the New Brush icon and make a new circular brush. I use circular for everything, though you may have need for other shapes occasionally. Make the brush fairly large and set the hardness to 95-98 (this isn't really an absolute, as you will see later, but it does allow you to see what the brush will look like at this hardness level in the brush preview window).

Step 2. Now we are gonna make some changes in your user interface. We are going to make it so that you can dynamically resize the brush as you are working and also dynamically change the hardness of the brush as well. I got the method of making dynamically resizeable brushes from the Gimptalk FAQ, and the idea for dynamically changing the hardness from the Gimptalk Manager, fotocomics.

This is what to do:

In the main toolbox window, click on File>Preferences. A dialog opens up. Go to the part that says "Input controllers". Click on it. The right hand pane will change to a view with two tabs. One says Main Mouse Wheel, and the other says Main Keyboard. The Main Mouse one should be the default. Scroll down through the list in the lower pane until you get to an entry that says "Scroll Up (Shift)", the space to the right of it should be blank. Mine already has something because I have already configured this option like I am describing here. Click on it, and it will highlight in blue.

Edit: Things have changed in 2.4.1. It is still possible to change the size and hardness of the brush. However, to change the size in the input controllers menu, double click on the mousewheel, then pick the key/mousewheel combination I mentioned for changing the size of the brush. Pick Edit. Instead of using the context menu to change the size, pick the tools menu. There will be an entry for scaling the brush up. Choose that. Do the same for when you follow the steps for scaling down.

To change the hardness, that is still under the context menu. And the directions for that are still good. Note: ONLY BRUSHES CREATED IN THE BRUSH EDITOR HAVE THE CAPABILITY FOR VARIABLE HARDNESS!

Below shows the sequence of menus to go through. Note that the fourth image is simply the third window "dragged out" to show more options. The final window shows all the changes that have been made.



This next series shows the steps to create a "render brush" in the brush editor.




If you follow the sequence of steps shown above, you can change the behavior of brushes created in the brush editor so that you can change the size and hardness "on the fly".

Edit>Preferences>Input Devices>Input Controllers>Main Mouse Wheel

From here, choose the option that you want to assign to modifying the option you wish to change, i.e., brush size or hardness. When you pick an option, such as shift with scroll up, you click the edit button and then choose the Context menu. In the context menu, pick "Increase Radius" to make the brush larger when you do <shift-scroll-up>.

Duplicate the step for the Main Mouse entry for decreasing the brush radius and also for the brush hardness and softness. You will be changing a total of four entries as summarized below.

Summary:

When you get done, you should have things configured like this:

Scroll Up (shift) = Increase Radius
Scroll Down (shift) = Decrease Radius

Scroll Up (control) = Increase Hardness
Scroll Down (control) = Decrease Hardness

To see the effect of what you have just done, you can exit out of the Preferences dialog by clicking "OK". Open a new image and select the brush tool. Pick the brush you made earlier in the brush editor. (It is probably already selected) Start brushing. Now, hold down your <shift> key and scroll the mouse wheel. Your brush should get bigger/larger, depending on how you scroll the wheel. Now try it with <ctrl> and see how the brush edge gets harder or softer in the brush display window.

The animated GIF below shows how the brush preview works with increase/decrease brush hardness.



The last thing to do is to set the spacing of the brush to "1" by moving the slider at the bottom of the brush selection window to the left, with your chosen brush selected. FYI, if you didn't know, the number is a percentage of the brush's width, "1" sets the spacing to 1 % of the brush's total width, 50% to half, and so on.



Okay, now we have our "rendering brush" set up.

Why did we do this? Perhaps you have already figured it out. But for those that haven't, let me explain.

One of the gripes that most people have with the Quickmask Method of doing renders is that you have to sometimes have a pretty steady hand, not to mention that the default Gimp brushes are kinda small. Being able to scale the brush goes a long way toward getting around this limitation. It is a lot easier to "have a steady hand" when using a really large brush, especially if you zoom in on the image you are rendering. Being able to scale the brush "on the fly" allows you to adjust for tight quarters without having to constantly switch brushes.

In addition, being able to control the hardness of the brush allows you to "dynamically feather" your render as you work. Got a soft edge? Soften your brush a bit. Need a harder brush for fine detail (like hair, for instance...?), Make the brush smaller and harden it up a bit. Need to paint large areas? Make the brush bigger.

Step 3. Open the image you want to render and duplicate the layer with the image. (Mine sample is an image I grabbed from the Creative Commons area of flickr.com. Here is the link: http://flickr.com/photo_zoom.gne?id=174361527&size=o This just provides an added level of protection so that you don't mess up the base layer, plus it allows you to mess with the levels, etc. if you want to try to make things stand out a bit to make things easier to render.

Right now, save the file with an .xcf extension. This is a further protection of the original, plus it saves the image with your layers, channels, paths, masks, image quality, etc. fully preserved.

Step 4. Do a rough selection around the image using the lasso tool. Don't worry about being a bit messy, we can easily clean that up later.



Step 5. Right click on the layer in the Layers dialog and pick "Add Layer Mask". In the dialog that pops up, pick "Selection". A black on white silhouette image will appear next to the image preview on the selected layer. This is your mask. Turn off your selection and then turn off the layer underneath. You will now see the area selected by your lasso tool against a transparent alpha background.



Now then, the image has not actually been modified. Unlike the Paths or Quickmask Methods, nothing is "cut" until you actually apply the layer mask. Indeed, you do not even need to apply the layer mask for this to be useful. All the background information is still there in the image, it is just blocked off by the mask.

A word about selecting the image vs. selecting the mask. Notice how the icon of the silhouette next to the actual image icon in the layers dialog has a white border. This means that the mask is selected. Whatever you do on the canvas happens on the mask, not on the image itself. If the image is selected, IT will have a border. Don't get the two confused.



Step 6. I like to add a white layer underneath the masked layer because trying to see things clearly against the gray checkerboard of the alpha channel is sometimes not easy. You also might want to add a black layer as well, you can toggle back and forth between the two to see how your render is gonna look against both types of backgrounds. Below see the image with no background, and against white with just the "lasso" mask.



When you get done rendering, if you use this method right, you won't have any areas where there is a dark or light fringe around the render. It should be nearly perfect, depending on the size of your source image. Indeed, with this method, you could actually render your image "in place" against your intended background.

Now we start actually rendering...phew!

If you click on the image in the layers dialog, you will paint on the image. We don't want to do that.

If you click on the mask, you will paint on the mask. As you paint on it in either black or white, you will either non-destructively erase or reveal parts of the image. Painting in black "erases" and painting in white reveals.

Tip: You don't even need to keep clicking on the foreground/background colors while you paint. You can toggle back and forth between the two by pressing your "X" key. When I am rendering, I have one hand on the lower left corner of the keyboard, controlling the brush characteristics with the <shift> and <control> keys and the mousewheel, and toggling the colors with the <X> key.

Now all you have to do is to paint around the image with the brush, "erasing" the background. Got an area with a soft edge? Soften up the brush. Indeed, you rarely want to have the brush all the way hard. I use a slightly soft edge as I paint, this allows me to "feather as I go" and bypass the step used by the other two rendering methods where you shrink/grow and feather the selection before cutting. Don't forget to save your image often as you work (and always in .xcf format).

Here is also another trick. Zooming in is helpful when rendering, no matter what method you are using, but it sometimes makes it hard to judge the overall effect of what you are doing on the full image.

If you click on View>New View in your canvas window, you can create a copy of your image. I use one that is zoomed and one that shows the full image. You can see the effect of your rendering in the full view while zooming in on part of the image.

A further trick is to do Dialogs>Navigation from the menus across the top of your canvas. This will open a dockable small window showing your full image with a small frame in it. You can drag the frame around on the image to control what your "zoomed in" window displays. It also has a slider that allows you to control the level of zoom as well as some icons which do the same sort of thing. Obviously to take advantage of these tricks, you need a bit of screen real estate.



The image I chose for this is perhaps a bit small. This method lends itself well to full size images from digital cameras, smaller images tend to look a bit jaggy sometimes. Of course this can be true with both the Path and Quickmask Methods.

The hair sticking up on top of the baby's head is something of a problem in an image this size. On the other hand, imagine trying to render hair with the other two methods. You generally end up giving the subject a haircut. Even if you try to single out thinner hair groups, you will lose them when you shrink and feather your selection. With this method, you can to some extent preserve the detail.

Further thoughts: You can also use things like the blur or smudge tools on the mask itself to help to blend your render with the background, very handy if you are "rendering" in place.

Below, see the effect of switching to a dark background . Notice the lighter areas that were missed from rendering against the white background. Conversely, if you rendered against a dark background to begin, you might see black "fringe" where you missed different areas. It's a snap to go in and correct either with this method. With the other two methods, it's bring out the eraser tool. Effective, yes, but are you "handling" the image more than necessary?



You could also use the path method in conjunction with this method. Use paths to outline larger areas, or even simply use the "conventional" path technique and click around your subject, convert to a selection and then, instead of on the image itself, but on the mask, just bucket fill with white or black, depending on what you are trying to do, then go back in and clean up the fine detail with a brush.

You can do gradients on the mask (yes, you can do this on a Quickmask as well...but can you see the results right away? ....and then undo and redo as necessary until you get what you want?)

Here is the finished product, shown against both white and black backgrounds, some smudging of the mask was done using our wonderful dynamic brush in soft mode.



Once you get things the way you want, then right click on the layer and click "Apply Layer Mask" (this will make your changes permanent) and remove/turn off any background layers. Do File>Save a Copy and save as a PNG file.

Another alternative is to simply drag the layer from the Layers dialog to the Layers dialog of a new image, this will preserve the layer mask.

*hugs to death* YOU ARE MY HERO!!!! It worked! Yippeekiyay!!

A good and useful tutorial! I rarely cut things out, but learning more about the layer mask was very interesting.
.
Griatch

:h:layer masks is one of the useful feature i often overlook , thank for sharing.

About quickmask mode the bigger problems is that the first time quickmask looks baffling...if the user switch to quickmask with NO selection active.

On the other hand is much more intuitive if a even rough selection is done first , and is useful switch during the work between normal and quickmask mode (very little selected pixel are often more visible in "normal" mode...).

But i fear that often 2 useful trick for quickmask are overlooked
1) when you think to have finished if there is any reddish colour in the image :
right click and change quickmask colour in something as blue or green...so any problem on reddish areas will become evident.

2 As you suggest for layermask use interactive brush

but again is very interesting tut...will be fine if you add the link to the sticky on help page on cutting renders

OMGosh this is great....I <3 u man..

Excellent CO! I hadn't thought about doing it that way but it looks like a great idea.

Thank you so much! This was the best and most effective way to render an image for me. :h:

Indeed, a most excellent tutorial!

For those who are comfortable with the alternative methods (paths, quickmask, lasso), it should be noted that they can be combined with the layermask method quite readily. The only difference is to replace the last step of those methods -- which is usually inverting the selection and performing an "Edit->Clear" -- with a "Edit->Fill With FG/BG" (CTRL-comma or CTRL-period) with the default (B&W) colors. This will permit you to keep the original image data intact.

:h:
@Clayogre
I must admit that the first time i had not much more then a glimpse on this tut:in half to notice that was very well done and original and that was necessary move it on this board...not in half to appreciate all the tips you give.
I'm going to follow it in pratice step by step.


since you work on a duplicate you can scale up as much needed , edit as you say (but withouth the final cut) and then scale again to the original size

then ...if you notice that in some way scaling up+ down decreased image quality you can just export the layermask
and apply it on the original (or a dup layer of the original)for the final cut.

PS i can't remember if there is something tricky about "copy and paste" or export layer mask , but it seems not likely.
If so i'm sure Saulgoode already wrote a script to do it in few seconds

Is there something with gnome or ubuntu? I cant get the mouse wheel scroll to work for size and hardness (insert joke here). Has anyone had this problem?

Yes Arron, I have winXP and the mouse wheel does not anything, and I promise, I made the context-changes. Even worse: now, after the changes as soon as I push the CTRL, I get the color-picker instead of brush. I use version 3.19. Maybe something wrong with this?

On xp i have similar problem, mouse wheel + key works but not always and no smoothly

And for some reason i too get at first the color picker only if i insist i can rescale the brush

I image i must change short key but is a pity because mouse wheel seems the best for the task...in theory.

Maybe somebody solved better on win?

After I've made changes (or tried to) I recognize unwilling changes in the keyboard functions, the path-tool keyboard funcitons doesn't work, alt shift and ctrl doesn't effect the cursor. Teh reset button doesn't help. The reset button doesn't help.

Ivan
Mouse wheel + key will work only with brush created with the brush editor, not with all other brush
(just in case you overlook this point)

PhotoComix, have you checked 2.4? It works fine there, mousewheel and shift can scale all the brushes. But there is not any regulator for hardness. Hardness can be selected or not. I cannot see any difference.

I know...but that could be added to all brush only by developers

Anyway the Procedural brush (created with brush editor, then could be saved)could be edit in any way with keys...you can add control for shapes, number of spikes and so on

Then actually control size and hardness to me seems sufficient for most of the task i can image

Now I have a newinstalled 2.4 and this brush control by scrolling doesn't work there. It worked when I had 2.4 before the same version and they behave different. Very frustrating. Have you any explanatin or - even better - a sollution?

Question.

I follow the method fine until I was done with creating the layer mask and selection and all that, then begin painting. I tried painting both black and white but nothing really happened, why's that? There's not even a paint mark on the image...

EDIT:

Nvm, I'm an idiot, lol. I set Brush Opacity to 0% (don't remember when) so it didn't paint out anything XD

i dt kno if ill b able to do this one

Just take it step by step. There may be some issues with creating the variable hardness brush in 2.4.1, however, supposedly the option is still there, but has been moved to a different section of the mouse configuration setup.

Well, I'm back again just to comment that this is a nice tutorial. Ever since I've learn about this, I use Layer Mask on all my rendering so far (even though I haven't rendered out many yet). Although just recently I decide to stop painting and do more Pathing, but of course along with Layer Mask, since I find myself able to control nodes and curve paths more properly than painting steadily(which I believe is the purpose of this tutorial, to learn Layer Mask, as it works well with other methods, and said so in the tut as well). I just wish I'm a better painter since I believe certain things just give greater result or simply work better with painting. Overall, though, I like the feeling of being secure when using Layer Mask. Thank you for introduce me to this tool. But, I'm pretty sure there're more functionalities for this tool, although I haven't figure them out yet. According to the manual, it's all about Selection and stuff... actually, nvm

Glad to hear it was useful for someone...

Note: I have made an edit to the original tutorial with some added instructions for 2.4.1. Instead of using the option for size under the context menu, use the one under the tools menu. This will work for scaling any kind of brushes, not just ones made in the brush editor.

The hardness control is still under the context section. Hardness control only works with brushes made in the brush editor.

i'm going to need to read this again

Hi, ClayOgre,
I run gimp 2.4.6 for windows XP.
The "renderbrush" controls work as described for method prior to 2.4.1 . ie;- The following sequence works!!
File/preferences/input Controllers/Main Mouse wheel/scroll to desired Key Action ((eg Scroll up (shift)) / Edit/Context/Scroll to desired Action (eg increase brush radius).
Again, as mentioned in prior posts, the Ctl key causes the "eye dropper" to appear so you cannot see the action of increase/decrease hardness. BUT the brush does increase /decrease hardness.

I was weary of using masks, because I find it difficult to always understand their actions in different circumstances, but this is a good way to combine with quickmask and other methods to cut a good render.
Thanks Wbool63