Notice On 500 Internal Server Errors

[Maypews]

Recently due to extreme issues with scrapers (which I believe came from us finally ranking on search engines) we have had to make a few tweaks. These bots were having serious performance impact on the site, causing the chat not to send messages and for pages to load very slowly. So as a counter measure some defensive measures were added.

The most prominent is that most of these bots would not advance past the first page of the site while scraping. When you first log on to the site or check the site for the first time in a while a ID will be appended to the end of your URL, and this essentially never went away in most bot sessions. So one of our solutions was to serve error pages when the URL has that identifier in it.

This will however affect normal users as well on their first page load. The solution is simply to refresh the page if you experience it. Once the SID is no longer at the end of your URL all pages will work normally. Alternatively if you want to bypass this all together you can change your browser useragent via extension and set it to Linux. Although it is a minor annoyance, I think it's better than having to deal with Cloudflare or Anubis when you visit the site.

[MetalHaruhi]

I was wondering if I was the only one getting these errors.

Do you have access to the web server's config files? .htaccess on Apache, Nginx would be the config file for the site in question.

If so, you might want to try out this AI scrapper blocklist, also add a robots.txt file too for good measure (though most predatory scrappers will ignore it.)

It might help a lil bit. I added it on my own VPS for my forum and I haven't had problems so far.

Here are some PhpBB extensions that could help mitigate the spam issue as well:

• Stop Forum Spam
• Agreement Delay
• hCaptcha

[Maypews]

I was wondering if I was the only one getting these errors.

Do you have access to the web server's config files? .htaccess on Apache, Nginx would be the config file for the site in question.

If so, you might want to try out this AI scrapper blocklist, also add a robots.txt file too for good measure (though most predatory scrappers will ignore it.)

It might help a lil bit. I added it on my own VPS for my forum and I haven't had problems so far.

Here are some PhpBB extensions that could help mitigate the spam issue as well:

• Stop Forum Spam
• Agreement Delay
• hCaptcha

Appreciate the post. I am aware of the Stop Forum Spam extension, but I got rid of it because it blocked an actual user from registering while lots of bots were on the site unharmed (this was some time ago, back in March-ish). Agreement Delay I am also aware of and plan on using if we start having issues with automated registrations, but as of now our question captcha seems to be handling that pretty well. If i'm not mistaken, HCaptcha also seems to predominantly handle registrations though I haven't looked as much into this one admittedly. Can it be used for other things?

We also have honeypots setup but we started getting so many IPs (averaged like 100k visitors a day!) that these alone were proving to no longer be enough due to the sheer quantity of them. I did not know about that blocklist you sent though, I'll look into setting that up. Thanks!

[dokyun]

If so, you might want to try out this AI scrapper blocklist, also add a robots.txt file too for good measure (though most predatory scrappers will ignore it.)

It might help a lil bit. I added it on my own VPS for my forum and I haven't had problems so far.

Here are some PhpBB extensions that could help mitigate the spam issue as well:

• Stop Forum Spam
• Agreement Delay
• hCaptcha

The problem isn't spam or AI scrapers but a botnet masquerading as normal users pounding the site with requests. This is the actual part of the .htaccess config that handles it:

Code: Select all

<IfModule mod_rewrite.c>

RewriteEngine On

RewriteCond %{REQUEST_METHOD} ^(GET)$

RewriteCond %{REQUEST_URI} !^/(index|ucp|acp|mcp).php

RewriteCond %{HTTP_USER_AGENT} (Windows|Macintosh)

RewriteCond %{QUERY_STRING} (?:^|&)sid=

RewriteRule ^(.*)$ http://127.0.0.1:32000 [P,L]

</IfModule>

Someone has money to burn trying to scrape this site. As you can see, it's actually supposed to forward these requests to a reverse proxy. The server behind the proxy is supposed to serve up gzip bombs, but it's not working so that's why it's been 500 errors.

[Maypews]

Someone has money to burn trying to scrape this site.

Gee.. I wonder who that could be.

committee2.png (621.53 KiB) Viewed 32 times