15 Replies Latest reply on Mar 31, 2014 12:39 AM by dgolberg

# How to determine if layers overlap?

I'm trying to find a way to determine if different layers have overlapping pixels.  The layers are randomly shaped and need to fit as closely together as possible, so using the bounds to check for the overlap would not work.  I'm basically looking for a way to detect the overlap based on pixels that are 100% opac.  Is there any easy way to do this through scripting?

• ###### 1. Re: How to determine if layers overlap?

So from your post, I take it that the shapes are irregular and not rectangles?  The bounds comand would work with rectangles, but to try and do this with pixels using extendscript would be painfully slow.

• ###### 2. Re: How to determine if layers overlap?

I kind of figured it would be unless there was some kind of option I'm not aware of that already does this (though it sounds like there isn't).  Regardless, it would still be faster than doing the task this will be used with, manually.

Right now, I'm building some code that will find 8 points on the layer (the layers are convex in shape and the points are basically laid out like a compass; ie: N,NE,E, etc.) that I can use to check if any other layers intersect those points.  Not perfect, but better than nothing I figure; though I will need a way to detect all layers that intersect the specific point.

So I guess an alternate question would be; is there an easy way to detect all layers that intersect a specific pixel coordinate?  Preferrably without having to cycle through them all and check bounds vs the point.

• ###### 3. Re: How to determine if layers overlap?

I would expect it should work with something along the lines of selecting the pixels of the first layer (as in ctrl-click layer) and then intersecting the selection with the second layer (as in ctrl-alt-shift clicking on the second layer) and then testing if the resulting selection is not empty.

• ###### 4. Re: How to determine if layers overlap?

Just thought of something that might work.  Create a snapshot of your file in history.  Make a selection of the reference layer, then select the layer you want to check. Apply a gaussian blur or some other function.  If an error is thrown saying that no pixels were selected, then you know that the layers don't overlap.  Use the snapshot to remove the blur.

Edit:  Of course if there is even the slightest filled pixel in the selection from anti aliasing, it will show that the layer's overlap.

• ###### 5. Re: How to determine if layers overlap?

Alright, I believe this should work.

Select the pixels for layer A

Save selection.

Select the pixels for layer B

Do a Threshold operation at whichever value you deem apropriate to ignore semi-transparent pixels.

Check if a selection remains (I believe I did that once with a Try/Catch block when trying to get selection bounds)

• ###### 6. Re: How to determine if layers overlap?

Interesting suggestions, I'll have to give them a try and see how it goes.  This would certainly be quicker (and easier) than my current plans using mathematical equations, if they work as expected.

• ###### 7. Re: How to determine if layers overlap?

or are they flat and solid?

I would change the opacity and see if they are overlapping.  the odd color would be your overlaps

• ###### 8. Re: How to determine if layers overlap?

They're a solid shape with varying color inside of that shape to show the detail of the item in question.  Currently, the only transparency is the anti-aliasing of the edges, which is fine if it causes a small gap as a small gap is desired, so long as it doesn't become too large.  A 10+ pixel gap all around is becoming too large.  It's fine if the gap goes beyond this in a couple spots, just so long as this isn't the universal minimum gap between layers.  (hope that doesn't make things more confusing)

Anyway, all good suggestions that are getting the old PS-Scripting wheels turning in my head again (been a while).

• ###### 9. Re: How to determine if layers overlap?

Following a co-worker's expression of "Show me, don't tell me", I've created 2 examples of the issue at hand.  Below are 2 images of the correct and incorrect result of what I'm trying to do based on bounds-based placement (incorrect) and transparent pixel detection placement (correct).  In the examples, the red and green boxes are there to represent the objects' bounds.

Desired outcome:

Bounds based outcome (incorrect for this project):

Note: The mock-up uses solid colors and hard-shapes, but the actual items being used are more detailed inside the shape with generally softer corners (by softer, I mean more rounded).

Anyway, due to interruptions at work I'm still setting up to try the methods suggested by csuebele and EnsilZah.  I'll let you know how those pan out, though I'm still open to other suggestions in the meantime.

Edit: figured I better clarify.  The overlap detection was requested due to their odd shapes, the actual placement is being handled by some code I've been working on; which can account for most, but not all object shapes.  For those that fall into the extra category, the overlap detection will tell me if I need to have the script nudge one layer away from the other a little bit.

• ###### 10. Re: How to determine if layers overlap?

I had some free time at work so here's my implementation, only problem is I don't know how to suppress the 'no pixels selected dialog.

function checkOverlapping()

{

var selectedLayers = getSelectedLayersIdx();

if (selectedLayers.length != 2) {alert ("Please select the two layers you would like to check"); return null;}

makeActiveByIndex(selectedLayers[0]);

selectActivePixels();

makeActiveByIndex(selectedLayers[1]);

intersectActivePixels();

activeDocument.activeLayer.threshold (128);

try

{

activeDocument.selection.bounds;

}

catch (e)

{

return false

}

activeDocument.selection.deselect();

return true;

}

function cID (inVal) { return charIDToTypeID(inVal);}

function sID (inVal) { return stringIDToTypeID(inVal);}

function selectActivePixels()

{

var desc15 = new ActionDescriptor();

var ref8 = new ActionReference();

ref8.putProperty( cID( "Chnl" ), cID( "fsel" ) );

desc15.putReference( cID( "null" ), ref8 );

var ref9 = new ActionReference();

ref9.putEnumerated( cID( "Chnl" ), cID( "Chnl" ), cID( "Trsp" ) );

desc15.putReference( cID( "T   " ), ref9 );

executeAction( cID( "setd" ), desc15, DialogModes.NO );

}

function intersectActivePixels()

{

var desc18 = new ActionDescriptor();

var ref13 = new ActionReference();

ref13.putEnumerated( cID( "Chnl" ), cID( "Chnl" ), cID( "Trsp" ) );

desc18.putReference( cID( "null" ), ref13 );

var ref14 = new ActionReference();

ref14.putProperty( cID( "Chnl" ), cID( "fsel" ) );

desc18.putReference( cID( "With" ), ref14 );

executeAction( cID( "Intr" ), desc18, DialogModes.NO );

}

function makeActiveByIndex( idx){

var desc = new ActionDescriptor();

var ref = new ActionReference();

ref.putIndex(cID( "Lyr " ), idx)

desc.putReference( cID( "null" ), ref );

desc.putBoolean( cID( "MkVs" ), true );

executeAction( cID( "slct" ), desc, DialogModes.NO );

};

function getSelectedLayersIdx(){

var selectedLayers = new Array;

var ref = new ActionReference();

ref.putEnumerated( cID("Dcmn"), cID("Ordn"), cID("Trgt") );

var desc = executeActionGet(ref);

if( desc.hasKey( sID( 'targetLayers' ) ) ){

desc = desc.getList( sID( 'targetLayers' ));

var c = desc.count

var selectedLayers = new Array();

for(var i=0;i<c;i++){

try{

activeDocument.backgroundLayer;

selectedLayers.push(  desc.getReference( i ).getIndex() );

}catch(e){

selectedLayers.push(  desc.getReference( i ).getIndex()+1 );

}

}

}else{

var ref = new ActionReference();

ref.putProperty( cID("Prpr") , cID( "ItmI" ));

ref.putEnumerated( cID("Lyr "), cID("Ordn"), cID("Trgt") );

try{

activeDocument.backgroundLayer;

selectedLayers.push( executeActionGet(ref).getInteger(cID( "ItmI" ))-1);

}catch(e){

selectedLayers.push( executeActionGet(ref).getInteger(cID( "ItmI" )));

}

var vis = app.activeDocument.activeLayer.visible;

if(vis == true) app.activeDocument.activeLayer.visible = false;

var desc9 = new ActionDescriptor();

var list9 = new ActionList();

var ref9 = new ActionReference();

ref9.putEnumerated( cID('Lyr '), cID('Ordn'), cID('Trgt') );

list9.putReference( ref9 );

desc9.putList( cID('null'), list9 );

executeAction( cID('Shw '), desc9, DialogModes.NO );

if(app.activeDocument.activeLayer.visible == false) selectedLayers.shift();

app.activeDocument.activeLayer.visible = vis;

}

return selectedLayers;

};

• ###### 11. Re: How to determine if layers overlap?

Very nice!  It works perfectly on my end!  No dialog boxes or anything (aside from the obvious alert, that is).  And thank you for putting the code together.  It never seems to fail; when I post a request for assistance on something, I suddenly get pulled in 50 other directions at work, lol.

I wish I could mark both yours and csuebele's as correct, as they both appear to work and are very similar, but since yours accounts for the transparent pixels (not that it's really an issue in this case) and you've provided some working code, I figure it only fair that you get the full credit.  Thanks a ton!

• ###### 12. Re: How to determine if layers overlap?

It might be faster to foregoe the threshold and evaluating the bounds and instead evaluate the QuickMask channel’s histogram[0].

And instead of selecting the Layers, then loading their transparency (and thus losing the original Layer selection) selecting the transparency via the index directly should also be a bit faster.

• ###### 13. Re: How to determine if layers overlap?

Ooo, that would speed it up a bit.  This process will be getting repeated several times per image (up to several hundred times), so every little bit helps.  The whole point of the script is to turn a manual process that takes roughly 60-80 minutes into a semi-automated process that takes 20 or less.

Anyway, I'll see if I can get your suggestions to work.

• ###### 14. Re: How to determine if layers overlap?

The difference in the time it takes is actually not tremendous, maybe something like 0,2 seconds.

And whether the histogram[0] value fits your needs (the two layers would both have to be fully solid at some overlapping pixels to achieve that) you’ll have to figure out yourself.

Anyway, here is an adapted version based on EnsilZah’s code.

// 2014, use it at your own risk;

#target photoshop

var time1 = Number(timeString());

////////////////////////////////////

var theCheck = checkOverlapping();

////////////////////////////////////

var time2 = Number(timeString());

////////////////////////////////////

function checkOverlapping()

{

var theDoc = app.activeDocument;

var selectedLayers = getSelectedLayersIdx();

if (selectedLayers.length != 2) {alert ("Please select the two layers you would like to check"); return null;}

selectLayerTransparency(selectedLayers[0], false);

selectLayerTransparency(selectedLayers[1], true);

var theCheck = theDoc.channels[theDoc.channels.length-1].histogram[0];

activeDocument.selection.deselect();

if (theCheck == 0) {return false}

else {return true};

};

////////////////////////////////////

function cID (inVal) { return charIDToTypeID(inVal);}

function sID (inVal) { return stringIDToTypeID(inVal);}

// select layer transparency;

function selectLayerTransparency (theIndex, intersect) {

// =======================================================

if (intersect == false) {

var idsetd = charIDToTypeID( "setd" );

var desc2 = new ActionDescriptor();

var idnull = charIDToTypeID( "null" );

var ref1 = new ActionReference();

var idChnl = charIDToTypeID( "Chnl" );

var idfsel = charIDToTypeID( "fsel" );

ref1.putProperty( idChnl, idfsel );

desc2.putReference( idnull, ref1 );

var idT = charIDToTypeID( "T   " );

desc2.putReference( idnull, ref1 );

var idT = charIDToTypeID( "T   " );

var ref2 = new ActionReference();

var idChnl = charIDToTypeID( "Chnl" );

var idChnl = charIDToTypeID( "Chnl" );

var idTrsp = charIDToTypeID( "Trsp" );

ref2.putEnumerated( idChnl, idChnl, idTrsp );

var idLyr = charIDToTypeID( "Lyr " );

ref2.putIndex( idLyr, theIndex );

desc2.putReference( idT, ref2 );

executeAction( idsetd, desc2, DialogModes.NO );

}

else {

var idIntr = charIDToTypeID( "Intr" );

var desc3 = new ActionDescriptor();

var idnull = charIDToTypeID( "null" );

var ref3 = new ActionReference();

var idChnl = charIDToTypeID( "Chnl" );

var idChnl = charIDToTypeID( "Chnl" );

var idTrsp = charIDToTypeID( "Trsp" );

ref3.putEnumerated( idChnl, idChnl, idTrsp );

var idLyr = charIDToTypeID( "Lyr " );

ref3.putIndex( idLyr, theIndex );

desc3.putReference( idnull, ref3 );

var idWith = charIDToTypeID( "With" );

var ref4 = new ActionReference();

var idChnl = charIDToTypeID( "Chnl" );

var idfsel = charIDToTypeID( "fsel" );

ref4.putProperty( idChnl, idfsel );

desc3.putReference( idWith, ref4 );

executeAction( idIntr, desc3, DialogModes.NO );

};

};

function getSelectedLayersIdx(){

var selectedLayers = new Array;

var ref = new ActionReference();

ref.putEnumerated( cID("Dcmn"), cID("Ordn"), cID("Trgt") );

var desc = executeActionGet(ref);

if( desc.hasKey( sID( 'targetLayers' ) ) ){

desc = desc.getList( sID( 'targetLayers' ));

var c = desc.count

var selectedLayers = new Array();

for(var i=0;i<c;i++){

try{

activeDocument.backgroundLayer;

selectedLayers.push(  desc.getReference( i ).getIndex() );

}catch(e){

selectedLayers.push(  desc.getReference( i ).getIndex()+1 );

}

}

}else{

var ref = new ActionReference();

ref.putProperty( cID("Prpr") , cID( "ItmI" ));

ref.putEnumerated( cID("Lyr "), cID("Ordn"), cID("Trgt") );

try{

activeDocument.backgroundLayer;

selectedLayers.push( executeActionGet(ref).getInteger(cID( "ItmI" ))-1);

}catch(e){

selectedLayers.push( executeActionGet(ref).getInteger(cID( "ItmI" )));

}

var vis = app.activeDocument.activeLayer.visible;

if(vis == true) app.activeDocument.activeLayer.visible = false;

var desc9 = new ActionDescriptor();

var list9 = new ActionList();

var ref9 = new ActionReference();

ref9.putEnumerated( cID('Lyr '), cID('Ordn'), cID('Trgt') );

list9.putReference( ref9 );

desc9.putList( cID('null'), list9 );

executeAction( cID('Shw '), desc9, DialogModes.NO );

if(app.activeDocument.activeLayer.visible == false) selectedLayers.shift();

app.activeDocument.activeLayer.visible = vis;

}

return selectedLayers;

};

////////////////////////////////////

////// function to get the date //////

function timeString () {

var now = new Date();

return now.getTime()

};

This process will be getting repeated several times per image (up to several hundred times

Arew you selecting the Layers pairs manually or will you automate that, too?

• ###### 15. Re: How to determine if layers overlap?

Very interesting.  And to answer your question, the layer selections will be automated as well.  The only thing that will be done manually is any touch-up that's necessary to finalize the look of the output.  This would include, but not be limited to, cleaning up unintentional color patterns, offsetting the image to make it tileable (though I might look at possibly automating this as well), and/or adding foreground/background objects.

I also played around with your histogram idea a bit (never really used it before) and came up with the following code that, while not perfect, does work in some cases; and in 0.01s, no less:

function overlapCheck() {

myDoc.activeLayer.visible = false;

selectOpac();

//myDoc.activeLayer = myDoc.layers.getByName("Layer 0");

var overlap = false;

var histo = app.activeDocument.histogram;

for(var i in histo) {

if(histo[i] > 0 && histo[i] < 255) {

overlap = true;

break;

}

}

return overlap;

}

function selectOpac() {

var idsetd = charIDToTypeID( "setd" );

var desc4 = new ActionDescriptor();

var idnull = charIDToTypeID( "null" );

var ref2 = new ActionReference();

var idChnl = charIDToTypeID( "Chnl" );

var idfsel = charIDToTypeID( "fsel" );

ref2.putProperty( idChnl, idfsel );

desc4.putReference( idnull, ref2 );

var idT = charIDToTypeID( "T   " );

var ref3 = new ActionReference();

var idChnl = charIDToTypeID( "Chnl" );

var idChnl = charIDToTypeID( "Chnl" );

var idTrsp = charIDToTypeID( "Trsp" );

ref3.putEnumerated( idChnl, idChnl, idTrsp );

desc4.putReference( idT, ref3 );

executeAction( idsetd, desc4, DialogModes.NO );

}

This will consistently return whether there is an overlap or not as long as one of 2 requirements are met:

• The document must have a white or transparent background
• This also has a sub-requirement that you cannot be attempting to check for overlap with one object while ignoring overlap with another.

or:

• The histogram must have its source set to "Selected Layer"
• This one requires that no more than 1 layer be selected at any one time during the script or it reverts to "entire document" mode, at which point the above requirement comes into play.
• This also requires that the commented line in the code above be uncommented and the appropriate layer be set as active (Layer 0 was used for my test).

Unfortunately, these flaws make it nearly useless in most cases.  If there were a way to set the histogram's "source" through scripting, the second issue could be averted and this would likely be the fastest way to check.  From what I've seen, though, it's not possible to change that via scripting, and the amount of time saved would be minimal compared to what's already being saved.

Regardless, both yours and EnsilZah's methods are a vast improvement over the manual method, so I thank you both for your assistance!