Wiki taken down for a bit.

Something wrong with it? Got an idea for it? Post here.

Moderator: Lucifer

Post Reply
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Wiki taken down for a bit.

Post by Z-Man »

Bad timing. Sorry. Looks like someone is using the server to farm bitcoins. Should be back up in an hour or so.
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Re: Wiki taken down for a bit.

Post by Z-Man »

Wiki back up. The code injection came over my wife's website, the datestamp says it started just two days ago. The attacker was able to execute arbitrary code as the webserver's user www-data. In principle, they would have been able to peek around and steal the wiki user database with password hashes and email addresses. My educated guess is that they were not interested in that. If you want to sneakily steal passwords and make the most of them, you don't leave behind a process that causes 100% CPU load. Yeah, it was bitcoin mining.

I'll investigate further, obviously, and we will make a more thorough announcement. In the meantime, I'd say that only if your password on the Wiki is the same as your home banking password, you should change both. And reconsider your password reuse policy :)

Sorry this happened again. Does anyone know a good way to prevent one site jeopardizing data of another running on the same server?
User avatar
Light
Reverse Outside Corner Grinder
Posts: 1667
Joined: Thu Oct 20, 2011 2:11 pm

Re: Wiki taken down for a bit.

Post by Light »

Z-Man wrote:Sorry this happened again. Does anyone know a good way to prevent one site jeopardizing data of another running on the same server?
If I make the assumption you're using Apache, you could use mod_privileges. That will let you run the virtual hosts under different users. Also, you shouldn't make the files globally available. You can probably chroot on top of that, locking them to their directory. I believe they could still screw with globally accessible files though. So, guest read permission might have to go away.

I would also suggest helping her realize why her site was vulnerable. It's not too hard to protect a site from injections. Keep the server and database safer that way.

Which brings another thought to mind. Don't use the same database user for each site. Might be fairly obvious, but I've come across a lot of stupid things people do, so I thought it might be worth mentioning. If there is anything they can inject some SQL, it will be open season on everything you've got. May as well limit them a little bit. :P

All that really comes to mind. Maybe limit what bash commands they can use as well. A user made specifically for a website doesn't really need to be able to do a lot with bash, unless you've got something special set up for it. Remember that exec() and shell_exec() is going to be limited with what you edit there, but that's the goal.
User avatar
fman23
On Lightcycle Grid
Posts: 36
Joined: Thu Sep 08, 2011 1:54 am
Location: Yes
Contact:

Re: Wiki taken down for a bit.

Post by fman23 »

For that, I use a different user for each website and don't give them permission to access each other. The only way I can think of that might be better would be experimenting with chroots or running each server in a virtual machine on that main server.
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Re: Wiki taken down for a bit.

Post by Z-Man »

*reading that later*

Apparently, this is now a sport. A different minerbot is active. They are using straight simple apache/php vulnerabilities as far as I read.
User avatar
Light
Reverse Outside Corner Grinder
Posts: 1667
Joined: Thu Oct 20, 2011 2:11 pm

Re: Wiki taken down for a bit.

Post by Light »

Z-Man wrote:*reading that later*

Apparently, this is now a sport. A different minerbot is active. They are using straight simple apache/php vulnerabilities as far as I read.
You could just knock her site offline until you get it taken care of? Also, make sure there's no cronjobs or startup scripts that are going to respawn all of it.
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Re: Wiki taken down for a bit.

Post by Z-Man »

Yes. Steps taken:
1. Shut down web server completely, investigate.
2. Shut down apparently affected site, turn on web server again.
3. Kill bot. Yes, it had a crontab going.
4. Find new bot at new path with new crontab.
5. Cleaned that. www-data can no loger edit its crontab, what appears to be the entry exploit disabled.

The injected code on my wife's site probably was not the initial exploit, rather it was left there as a backdoor for the future.

So, second bot gone, acces path shut with some boards and nails.
Light wrote:
Z-Man wrote:Sorry this happened again. Does anyone know a good way to prevent one site jeopardizing data of another running on the same server?
If I make the assumption you're using Apache, you could use mod_privileges. That will let you run the virtual hosts under different users.
Precisely the kind of thing I was looking for.
Light wrote:Also, you shouldn't make the files globally available.
That's a given once it is actually possible.
Light wrote:You can probably chroot on top of that, locking them to their directory. I believe they could still screw with globally accessible files though. So, guest read permission might have to go away.
Would that be possible inside the apache process? I mean, sure, one can chroot the whole server, but can the server chroot its instances responsible for the different sites?
Light wrote:I would also suggest helping her realize why her site was vulnerable. It's not too hard to protect a site from injections. Keep the server and database safer that way.
I'm the admin, so any techical stuff is entirely my fault. After our last attack, all regular injection paths were eliminated by setting minimal rights: no script execution in data directories, no write permission to www-data in script directories. All of that does not help if one postpones OS distribution upgrades, though (security updates were running in a cronjob, but they only go so far). Problem: It's a VPS, and you can't upgrade the kernel without a full reinstallation. I already upgraded the OS anyway, so its kernel is one distribution version down... seems to work OK, but I should not stretch that further.
Light wrote:Which brings another thought to mind. Don't use the same database user for each site. Might be fairly obvious, but I've come across a lot of stupid things people do, so I thought it might be worth mentioning. If there is anything they can inject some SQL, it will be open season on everything you've got. May as well limit them a little bit. :P
Yeah :) Only the wiki is using the database, so no problem there.
Light wrote:All that really comes to mind. Maybe limit what bash commands they can use as well. A user made specifically for a website doesn't really need to be able to do a lot with bash, unless you've got something special set up for it. Remember that exec() and shell_exec() is going to be limited with what you edit there, but that's the goal.
Not a bad idea, would have stopped the first bot. Maybe not the second one, that seemed to be purely using php to do its work. But the same principle applies: it was exploiting a system that was not in fact needed.

Urgh. Looks like there is no better time to get a bigger server than now, doesn't it?

Ladle recordings will suffer a bit this time, too.
User avatar
Light
Reverse Outside Corner Grinder
Posts: 1667
Joined: Thu Oct 20, 2011 2:11 pm

Re: Wiki taken down for a bit.

Post by Light »

Z-Man wrote:The injected code on my wife's site probably was not the initial exploit, rather it was left there as a backdoor for the future.
It depends on the possible code injections. If you can inject PHP, then it could be the source of just about anything given the right permissions.
Z-Man wrote:Would that be possible inside the apache process? I mean, sure, one can chroot the whole server, but can the server chroot its instances responsible for the different sites?
Yes, but you have to be using a different user for the virtualhost. A quick Google search should be able to help you out with this.
Z-Man wrote:It's a VPS, and you can't upgrade the kernel without a full reinstallation.
That sounds like a huge pain in the ass. Running a normal server you can just reboot.
Z-Man wrote:Urgh. Looks like there is no better time to get a bigger server than now, doesn't it?
I hate moving servers, so that never sounds like fun to me. lol
Z-Man wrote:Ladle recordings will suffer a bit this time, too.
You could always host them on any file sharing site until you're ready to host them yourself. Or maybe even as attachments here? Unless, maybe you were talking about you actually recording them, in which case I have no useful input. :P
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Re: Wiki taken down for a bit.

Post by Z-Man »

Light wrote:
Z-Man wrote:The injected code on my wife's site probably was not the initial exploit, rather it was left there as a backdoor for the future.
It depends on the possible code injections. If you can inject PHP, then it could be the source of just about anything given the right permissions.
That was the case. It was this one: http://www.exploit-db.com/exploits/29290/
Light wrote:
Z-Man wrote:It's a VPS, and you can't upgrade the kernel without a full reinstallation.
That sounds like a huge pain in the ass. Running a normal server you can just reboot.
Yep. I'm pondering getting a real server where I can really do as I please or falling back to shared web hosting where those issues are taken care of for me. Probably loses too much flexibility, though. I bet a shared hoster where you can insulate YOUR sites from one another will be hard to find.
Light wrote:
Z-Man wrote:Ladle recordings will suffer a bit this time, too.
You could always host them on any file sharing site until you're ready to host them yourself. Or maybe even as attachments here? Unless, maybe you were talking about you actually recording them, in which case I have no useful input. :P
Hosting is fine :) They are on my home server. Making them for this ladle, however, collided a bit with the damage control work. Indirectly. Both collided with eating from opposite directions.
bilbo baggins
Round Winner
Posts: 214
Joined: Sat Jan 01, 2011 6:05 pm

Re: Wiki taken down for a bit.

Post by bilbo baggins »

your intelligence blows my mind :o
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Re: Wiki taken down for a bit.

Post by Z-Man »

Don't be overly impressed. Google did most of the hard work (well, and I guess the guys discovering the vulnerability and the other guys writing the forum posts google turned up), I just plugged in the odd POST requests that showed up in the webserver logs. Took me a long time to find out where user specific crontabs are managed :(

Edit: Heh, and sometimes, you find this helpfully documented sploit script on your system:

Code: Select all

#!/bin/sh

# A cleaner implementation of: http://www.exploit-db.com/papers/15311/
# This script uses Perl as a setuid() provider, but any lang will do
...
That one didn't work. Would have been a root right escalation. Phew.

In total, there were at least FOUR completely different miner bots active, each killing the one that came before. Definitely a sport. In addition to the exploit attempt above, the logs show one simple direct su attempt. In addition to CPU time, they used quite a bit of bandwidth, 150 gigs a day. Nothing I have seen so far has been even a tiny bit stealthy. Like, bot 3 brought a full copy of boost with it and compiled itself. Still, if I were to gain parital control over a system and make some noise by neccessity, but wanted to leave a backdoor anyway, I would also leave a second, easy to discover backdoor for the admin to find and feel smug about... I'll definitely compare all php scipts with what is supposed to be there.
Post Reply