Stop Indexing Duplicate Content

[Simon]

While we're on the subject, I just built my first official wordpress plugin, which I've named the "Dilution Factor". It uses the classic noindex,follow meta tag on certain pages, but the twist is that it places the tag on all category / tag pages that have fewer posts than a set minimum:

Check out the full explanation, plus download, here.

Unashamedly taken from what wanderingmind said earlier in the thread. As we've already said, it *may* make no difference, but then again it just might! I might later also add unique meta description tags per page (although its already been done plenty of times for wordpress) - and then maybe think about porting it over to a pligg module.

[noyp]

how many pages will be indexed when using Simon's recommendation?

how much to gain with this?...please share your results.

thanks
noyP

[cmlawson]

Originally Posted by Simon

I just looked at one of my pligg sites today, loads of supplemental results stopping the stories from ever having a chance of indexing well. Categories and tagging = duplicate story content.

In index.php, added in the following code, near to where pagename is defined:

$the_page = get_current_page();
if($the_page == 1) {
    $main_smarty->assign('indexit', 1);
} else {
    $main_smarty->assign('indexit', 0);
}

Then added into meta.tpl:

{if $indexit eq "1" OR $pagename eq "story"}
<meta name="robots" content="index,follow" />
{else}
<meta name="robots" content="noindex,follow" />
{/if}

This will only index the home page and story pages, since everything else is duplicate content to the search engines. Story links on non-indexed pages will still be followed though. Some people may say this is too much, but duplicate content will really hurt your rankings, so removing it completely (other than the home page content) should help.

You can always change the code to allow more pages to be indexed, such as user profile pages.

Great idea! Just curious then, what does your sitemap.xml file look like then?

As an aside, is there an tutorial or post on how to create another page on the pligg site installation. I can't figure out how to get one with the template and all. (Sorry, I'm new. )

~cml

[Simon]

I don't use a sitemap for pligg yet - its actually been indexed pretty quickly, so I never bothered creating one.

Page Creation: Simple Static Page Support...

@noyp - what I set out in the first post should cover indexing of the home page, individual story pages, and the first page of all categories (since they run through index.php too). Whether you want to add other pages is up to you.

What we really need is a way to produce unique meta descriptions for category and tag pages, then I could combine it for what I've done in my wordpress plugin, and stop indexing of pages that are potentially "too similar", based on the number of articles on a page.

[noyp]

Simon

thanks for the reply

another question, how can i have "search result pages" indexed from my site?
Since i sometimes use "keyword results" as "categories", i'd like google to index these.

A better question would be how to add pages for indexing using your code

thanks

noyP

[Simon]

For search pages, you'd want to use $pagename eq "search" I think (in the if condition). You may also want to put in the "page one only" code into search.php as well as index.php, depending on your preferences.

[Simon]

Just wanted to bring people's attention to a few posts, made by Aaron Wall, and ShoeMoney. If Aaron Wall says something, it's usually best to follow his advice!

http://www.seobook.com/archives/002021.shtml
http://www.shoemoney.com/2007/02/01/...ve-new-months/
http://www.shoemoney.com/robots.txt

In summary, Aaron talks about not spreading your link authority across too many pages. Less is more, is the feeling I get from his post. He also gave some advice to ShoeMoney, which resulted in a 1400% increase in traffic to some pages. It seems to have been a combination of not allowing indexing of "fluff" pages, combined with the acquiring of a few choice links to get pages out of the supplementals.

If you look at ShoeMoney's robots file, you will see that category pages have been dis-allowed. I found that interesting. Anyone got any comments or analysis on what Aaron / SM have said in the above posts?

[P1mpPanther]

thanks for continuing this discussion simon and sharing these great resources...

[abcdefgary]

this part is really insightful:

Many people believe that having more pages is always better, but ever since Google got more aggressive with duplicate content filters

I used to own a beginner website which was a photo gallery and used many includes such as the header, footer, sidebar, similar to the yget layout.

I noticed that many of my pages were recognized by google as "the same" although they had different titles, meta information, and main contents. And, as you know, many of these pages resulted in ending up in Google's supplementary results.

So, after reading this, I think I'm just going to allow spiders to crawl the index, the upcoming stories and any stories that can be accessed via these pages.

[betagirl]

Originally Posted by Simon

You can insert unique meta descriptions for story pages, but not for tag and category pages I believe. That's how I have it set up at any rate, along with a unique description for the home page.

I have been reading around the whole duplicate content issue quite a lot lately, and to be honest there is very little consensus on what actually constitutes duplicate content. Some will class it as you have above, some will say that if you have the same story content on multiple pages (regardless of what other content is on the page) then that can be a trigger.

Then you have the view that it is not worth the effort for a search engine to check for duplicate content in the first place. Billions of pages - that's a lot of checking to be done. Then you've got the fact that articles are re-published all over the internet, should the original authors be specifically penalised for writing successful articles?

The only conclusion I can easily draw is that it is an unknown - although it probably makes little difference in the long run. Building links to your inner pages will do far more good for a webmaster.

Simon - how did you get unique metas on your home page? I've gotten them on other pages I've added to the system, as well as some already there, but I've not been able to sort out getting the home page unique. Thanks for any tips!