Stop Indexing Duplicate Content

[Simon]

I just looked at one of my pligg sites today, loads of supplemental results stopping the stories from ever having a chance of indexing well. Categories and tagging = duplicate story content.

In index.php, added in the following code, near to where pagename is defined:

$the_page = get_current_page();
if($the_page == 1) {
    $main_smarty->assign('indexit', 1);
} else {
    $main_smarty->assign('indexit', 0);
}

Then added into meta.tpl:

{if $indexit eq "1" OR $pagename eq "story"}
<meta name="robots" content="index,follow" />
{else}
<meta name="robots" content="noindex,follow" />
{/if}

This will only index the home page and story pages, since everything else is duplicate content to the search engines. Story links on non-indexed pages will still be followed though. Some people may say this is too much, but duplicate content will really hurt your rankings, so removing it completely (other than the home page content) should help.

You can always change the code to allow more pages to be indexed, such as user profile pages.

[P1mpPanther]

hey simon, in which file did you add this code?

[Simon]

Just updated the code to make it easier to try out. Files that need changing are index.php and meta.tpl.

[P1mpPanther]

thanks, much better!

i just implemented this, so we'll see...

[Simon]

No problem. The reason I thought of this now is because I've been doing some SEO work on my new wordpress blog, and one of the main problems was with categories, archives, and tag pages producing heaps of duplicate content.

By default, pligg is just as bad on this front, if not worse. All the main pages (not counting things like user profiles) produce duplicate content of the story pages. Once that happens, you're never likely to rank that highly in the search engines.

But be warned that this is only a theory. Some people will tell you that categories / tag pages should be indexed because of their higher keyword density. I doubt too many really know the best way to let a site be indexed (especially by google), but I'm just using the simple premise that less duplicate content is a good thing.

http://www.google.com/support/webmas...y?answer=66359

That's what google has to say on duplicate content. It will always try to pick "the most appropriate" version of the same content on a site. So you could argue that as long as the duplicate stuff is just on your site, then it probably won't be perceived as malicious. If it is perceived as malicious, then rankings will definitely suffer. The question is how often google gets it right.

[P1mpPanther]

id say this is a good place to start and if you need more content to be indexed, you can add more pages, right?

[Simon]

That's what I'm thinking. I actually just blogged about it, although in regards to my wordpress setup. I'm thinking that I will just allow indexing of articles, home page, and the 1st page of each main category (because of the richer keyword potential) and go from there.

Not sure about pligg categories yet, but for now I'm keeping them non-indexed...

[wanderingmind]

It might or might not be duplicate content.

For example, say a category or tag page.

These pages would have titles and descriptions from several stories. Say 10 titles and descriptions.

This page then would not be identical to any other story page for the duplicate content problems to arise. If you have ten stories, compared to any other readmore story on the site, the page would be 90 % different. That is more than enough to take care of dup content issues.

However, having identical / no meta descriptions in every page is considered quite dangerous... and in pligg, we have no way to insert unique meta descriptions....

[Simon]

You can insert unique meta descriptions for story pages, but not for tag and category pages I believe. That's how I have it set up at any rate, along with a unique description for the home page.

I have been reading around the whole duplicate content issue quite a lot lately, and to be honest there is very little consensus on what actually constitutes duplicate content. Some will class it as you have above, some will say that if you have the same story content on multiple pages (regardless of what other content is on the page) then that can be a trigger.

Then you have the view that it is not worth the effort for a search engine to check for duplicate content in the first place. Billions of pages - that's a lot of checking to be done. Then you've got the fact that articles are re-published all over the internet, should the original authors be specifically penalised for writing successful articles?

The only conclusion I can easily draw is that it is an unknown - although it probably makes little difference in the long run. Building links to your inner pages will do far more good for a webmaster.

[P1mpPanther]

I was about to chime in something similar - it seems that if you ask 10 people about this matter, you will get 10 different opinions.

More than likely, you won't run into any issues - however, I personally would rather monitor things and add pages once I realize there is no negative impact rather than getting nailed by Google and spend the next couple years trying to get out of their sandbox!

Great discussion - SEO and Pligg!!