Feature and Google being an idiot

[xezno]

I just clicked on "visitors" in the Members section and saw this:

1. 
   Robot: Google
   Viewing member profile Katy 10 minutes ago

Arguments are below.
|
|
\/

 

[aidancheddar]

Well, Google needs to view the members, let alone posts, so we can search for them later when not on the site.

 

[heat33330]

Yeah, and? Unfortunately Google just has to spy on everyone but you can block them in the robots.txt of your site.

 

[aidancheddar]

Yeah, and? Unfortunately Google just has to spy on everyone

Granted, this is a public forum.

 

[Rhys]

Yeah, and? Unfortunately Google just has to spy on everyone but you can block them in the robots.txt of your site.

Uh, what?

Google is not "spying on everyone", it is indexing all pages it can find so that searches can find the information on those pages. Blocking google using robots.txt is quite frankly the worst thing you could ever do with a website.

 

[xezno]

Uh, what?

Google is not "spying on everyone", it is indexing all pages it can find so that searches can find the information on those pages. Blocking google using robots.txt is quite frankly the worst thing you could ever do with a website.

Especially if it's the Project Dollhouse forums.

 

[heat33330]

It was a joke, relax. Lol. And I never demanded everyone to block Google using robots.txt now, I was only stating a fact. And it's not "the worst thing you could ever do with a website".

 

[Rhys]

And it's not "the worst thing you could ever do with a website".

Yes it is. Doing this removes your primary source of hits and if you've set it up to block all robots it blocks the Wayback Machine, which is a huge inconvenience if your site provides any kind of unique information and it shuts down at some point. There aren't any pros.

 

[aidancheddar]

And considering how uncommon, near rare, it is to get user screenshots or gameplay videos of TSO prior to EA-Land, blocking search robots would unwise.

 

[heat33330]

Yes it is. Doing this removes your primary source of hits and if you've set it up to block all robots it blocks the Wayback Machine, which is a huge inconvenience if your site provides any kind of unique information and it shuts down at some point. There aren't any pros.

Eh, no. It's not. There are many cases where websites don't want to be in search results or seen through Wayback Machine.

 

[Rhys]

Like what? If you don't want something to be public, either don't post it to the internet or require authentication for access.

 

[heat33330]

If you deny Google from indexing your site in Robots.txt they can still offer URL-only listings of it. There are cons to Google indexing your site.

 

[aidancheddar]

Again, I repet, this is a public forum. An example of what those cons may be would nice.

 

[Rhys]

If you deny Google from indexing your site in Robots.txt they can still offer URL-only listings of it. There are cons to Google indexing your site.

Like what? I don't see any benefit from doing this except maybe around 100kb saved bandwidth every week or so.

 

[heat33330]

Just because this forum wouldn't have a use for it doesn't mean other sites wouldn't. So, in all of the sites on the internet not even one would see a benefit from this? Yeah right... You can choose to have your least important pages indexed i.e. you're "Screenshots" page so your "Main" page will have a better chance of being one of the first results.

 

[Rhys]

The argument was about blocking the whole site, not parts of it. I understand the need to deter search engines from eg. a "user page 435234's posts" page, not the whole site.

 

[aidancheddar]

Just because this forum wouldn't have a use for it doesn't mean other sites wouldn't. So, in all of the sites on the internet not even one would see a benefit from this? Yeah right... You can choose to have your least important pages indexed i.e. you're "Screenshots" page so your "Main" page will have a better chance of being one of the first results.

It already does.

 

[heat33330]

Again, I repeat, I was not only on about this site. Where did I specifically refer to PD?

I guess you're right, there probably is rare cases but the majority wouldn't block their entire site - only pages like errors etc., if that.

 

[xezno]

Just don't add a robots file.

 

[aidancheddar]

Again, I repeat, I was not only on about this site. Where did I specifically refer to PD?

Still haven’t made a real valid case, imo.