Robot.txt edited?

[LittleSteps]

Hey, when having trouble I'll often use Google to search and see if any of my problems have already occurred and been solved before. I do not like using the searching tool that is supplied in Armagetron Forums because of its limitation.
I do enjoy Googling my compiling errors associated with Armagetron and find it a crucial to be able to index the forums through Google.

I have ignored it for a couple days cause I know the developers are moving things around, will the robot.txt allow Google to search through the forums again?

[aP|Nelg]

You can still search with google, it just wont show the text in the engine...
What'd be most helpful if we get the built in search function to not ignore "common words"
this is a phpbb specific issue, I've seen no problem with other things such as what https://omgclan.cf/ uses

[Light]

You can still search with google, it just wont show the text in the engine...

You can, but it's not nearly as nice to look through. I have also always used Google to look for things I needed here since it was much easier. It still is quick since I know which ones I want, but it's kind'a ruined for actually looking for ones I'm not familiar with.

[Lucifer]

Pretty sure we have no intention of excluding google from indexing these forums. That would be counterproductive when we tell people to search the forums if you can't use a decent search engine to do it, wouldn't it?

[LittleSteps]

Pretty sure we have no intention of excluding google from indexing these forums. That would be counterproductive when we tell people to search the forums if you can't use a decent search engine to do it, wouldn't it?

This problem still occurs, I am fairly sure the forums is already excluding google from indexing the forums. Have you tried to search anything, then include armagetron forums after it, or http://forums3.armagetronad.net/? comes up with nothing.

[Light]

I'm doubting it was accidental.
http://forums3.armagetronad.net/robots.txt

We will probably need to wait for Tank to give some input on it or remove it.

[Lucifer]

I just paged him in irc. I vaguely remember him stopping bots back in the day when there were lots of bad bots and few good ones. So, it's probably a relic from back then. (Probably something I objected to back then, but I didn't exactly have a voice )

[Light]

I just paged him in irc. I vaguely remember him stopping bots back in the day when there were lots of bad bots and few good ones. So, it's probably a relic from back then. (Probably something I objected to back then, but I didn't exactly have a voice )

robots.txt doesn't stop bad bots, sadly. The crawlers that abide by the rules in there are programmed to check for them. So, unless blocking ones such as Google or Bing was the goal, it was kind of pointless.

[Z-Man]

It's usual practice to block good bots from indexing the member list and user profiles. Just registering accounts with spammy links in their profiles to increase pageranks is a common stealth spambot technique, and just not having those enter Google's databases stops that from being useful. Smart spambots would also read robots.txt for such blocks and just not bother. Then again, the resources wasted by useless spam accounts are those of the hijacked zombie PCs, so there's little incentive to do that.

Anyway, blocking the whole site is a bit overkill.

[Light]

It's usual practice to block good bots from indexing the member list and user profiles.

If it were targeted areas, there'd really be nothing to fuss about but ..

Anyway, blocking the whole site is a bit overkill.

Though, from how Luci made it sound, Tanks goal was to block spambots and whatnot which would have nothing to do with that file as you already said. Maybe it was just an in-the-moment thing caused by stress of the attacks without figuring out exactly how it works, or idk. I kind'a wish he was sharing the lead with someone that's around a little more though.

[Lucifer]

He changed it. I just checked.

He told me in irc that during the February attack, there was a non-trivial part of the attack that included the site being crawled by every imaginable bot. So editing robots.txt at that time made at least the good bots leave us alone.

He's added a crawl delay to the good bots and excluded all others, so the ones that actually care about robots.txt will do the right thing.

Edit: Google should be back along shortly, I'd imagine, but if somebody wants to submit the site to attract their bot, go right ahead.

[ConVicT]

I kind'a wish he was sharing the lead with someone that's around a little more though.

I agree and disagree at the same time.
He doesn't have the time, or/and doesn't care.
Hand the forums to someone who gives a shit, seriously (not even a facking april fools...)

Beyond a joke now!

Also like to point out that this still doesn't work either!