Java-Gaming.org Hi !
Featured games (91)
games approved by the League of Dukes
Games in Showcase (804)
Games in Android Showcase (239)
games submitted by our members
Games in WIP (868)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  JGO server folded in on itself  (Read 10614 times)
0 Members and 1 Guest are viewing this topic.
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Posted 2016-01-13 11:23:52 »

In the eternal quest of figuring out why the JGO server keeps pulling the rug from under itself, the VM managed to hide its tracks surprisingly well, in an unforeseen self-inflicted denial of service attack.

10.30 PM: Arrived home from work in Belgium.
11.00 PM: I went to bed.
11.30 PM: The daily backup was made.
11.55 PM: VM instance collapsed, unreachable (no ping, no http, no ssh), constant 100% CPU usage, no disk I/O, no network I/O.
11:56 PM: Pingdom sends me down-notification texts
07:30 AM: alarm clock wakes me from my slumber
07:31 AM: I see 2 texts from Pingdom... I notice JGO is actually down
07:33 AM: VM is in lockdown, cannot even login with KVM
07:40 AM: Ensure we have backups, reboot into rescue mode.
07:42 AM: Trying to mount virtual disk, turns out to be corrupted, repairable though.
07:52 AM: Cloning JGO instance to new VM.
07:53 AM: Cloned instance has mountable disk, at least last-minute data is safe.
07:55 AM: Rolling back backup on original instance.
08:00 AM: Jumped in the shower, f**king cold - boiler gave up due to low water pressure... is there a leak? Who gives a shit! No time!
08:07 AM: (short shower...) JGO backup restored, live.
08:08 AM: java-gaming.org does not work, www.java-gaming.org does work - there seems to be some virtualhost issue
08:25 AM: off to work.
09:25 AM: working... (arrived damn late)
11:55 AM: solved virtualhost issue: despite the configuration being explicit and correct, apache2 decided that the java-gaming.org hostname became the default, and hence was routed to /var/www, which contained only an index.html with: '<h1>It works!</h1>'.

1  
2  
3  
4  
5  
6  
7  
8  
<VirtualHost *:80>
    ServerAdmin webmaster@java-gaming.org
    DocumentRoot "/home/jgo/public_html/"
   ServerName java-gaming.org // ignored
    ServerAlias pastebin.java-gaming.org www.java-gaming.org
    ErrorLog "/home/jgo/log/apache2/java-gaming.org-error_log"
    CustomLog "/home/jgo/log/apache2/java-gaming.org-access_log" common
</VirtualHost>


1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
// added dummy domain to trigger apache2 to use as 000-default
+ ServerAdmin default@default.com
+ DocumentRoot /home/websites/domains/default/
+ ServerName default

<VirtualHost *:80>
    ServerAdmin webmaster@java-gaming.org
    DocumentRoot "/home/jgo/public_html/"
    ServerName java-gaming.org
    ServerAlias pastebin.java-gaming.org www.java-gaming.org
    ErrorLog "/home/jgo/log/apache2/java-gaming.org-error_log"
    CustomLog "/home/jgo/log/apache2/java-gaming.org-access_log" common
</VirtualHost>



Open issues:
 - what happened at: 11.55 PM
 - why did apache2 change its interpretation of defined virtualhosts

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline ags1

JGO Kernel


Medals: 367
Projects: 7


Make code not war!


« Reply #1 - Posted 2016-01-13 11:31:48 »

Much appreciated!

Your first mistake:

1  
11.00 PM: I went to bed.


You should know by now that JGO schedules its outages for your downtimes Smiley

Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #2 - Posted 2016-01-13 18:22:37 »

Is there some (reliable!) app that scans through your text-messages and triggers an alarm upon a pattern match?

This one seems pretty decent...
   https://play.google.com/store/apps/details?id=ax.ha.it.smsalarm

So far it seems to work, survives a reboot, ignores audio-levels/mutes. It's a shame the Pingdom app (which had an alarm set) didn't trigger... yet the text-message from Pingdom was received.
Anybody any experience with the above app (or similar apps)?

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline jonjava
« Reply #3 - Posted 2016-01-13 20:30:36 »



Nothing suspicious in the log files between 11:50-55?

Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #4 - Posted 2016-01-13 21:52:18 »

Heh, that's the first place I looked. Nothing suspicious, just the usual jibber jabber. Emo

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline Cero
« Reply #5 - Posted 2016-01-14 19:46:30 »

Wow that was a lot of info right there, timestamps even.
You didnt have to copy your twitter feed  :D

Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #6 - Posted 2016-01-14 19:55:32 »

#1 complaint after downtime of any website is lack of details Pointing

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline ra4king

JGO Kernel


Medals: 508
Projects: 3
Exp: 5 years


I'm the King!


« Reply #7 - Posted 2016-01-14 20:20:22 »

Your first mistake is using Apache2 Pointing

Offline Herjan
« Reply #8 - Posted 2016-01-14 22:46:26 »

07:52 AM: Cloning JGO instance to new VM.
...
08:08 AM: java-gaming.org does not work, www.java-gaming.org does work - there seems to be some virtualhost issue
...
11:55 AM: solved virtualhost issue: despite the configuration being explicit and correct, apache2 decided that the java-gaming.org hostname became the default, and hence was routed to /var/www, which contained only an index.html with: '<h1>It works!</h1>'.

Open issues:
 - why did apache2 change its interpretation of defined virtualhosts

Just to be sure:
Are you sure you enabled the apache config before you changed it? You moved to a new VM, so you would have to (re-)enable the config before (re)starting apache.
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #9 - Posted 2016-01-15 08:40:46 »

07:52 AM: Cloning JGO instance to new VM.
...
08:08 AM: java-gaming.org does not work, www.java-gaming.org does work - there seems to be some virtualhost issue
...
11:55 AM: solved virtualhost issue: despite the configuration being explicit and correct, apache2 decided that the java-gaming.org hostname became the default, and hence was routed to /var/www, which contained only an index.html with: '<h1>It works!</h1>'.

Open issues:
 - why did apache2 change its interpretation of defined virtualhosts

Just to be sure:
Are you sure you enabled the apache config before you changed it? You moved to a new VM, so you would have to (re-)enable the config before (re)starting apache.

The config was alright. It had survived a handful of reboots already. To be honest, I'm not sure what 're-enable the config' means. Without a config, not a single virtualhost would have been loaded, and none of the websites would have worked. Anyhoo, I went through quite a bit trial and error, and determined that basically every single change worked (even changing the main ServerName to xyz.java-gaming.org), except... java-gaming.org. Then I ran
apachectl -S
and noticed java-gaming.org was picked up as default (whether it was the first or last virtualhost defined in apache2.conf...), and (as said) rerouted to /var/www/.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #10 - Posted 2016-01-20 16:22:30 »

JGO became unreachable for about 2 hours because my ISPs nameserver stopped responding to queries.

I have created a spiffy twitter account, @JavaGamingOrg, which you guys can monitor in case the regular means of communication fail.


Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Pages: [1]
  ignore  |  Print  
 
 

 
Riven (581 views)
2019-09-04 15:33:17

hadezbladez (5510 views)
2018-11-16 13:46:03

hadezbladez (2402 views)
2018-11-16 13:41:33

hadezbladez (5772 views)
2018-11-16 13:35:35

hadezbladez (1223 views)
2018-11-16 13:32:03

EgonOlsen (4661 views)
2018-06-10 19:43:48

EgonOlsen (5682 views)
2018-06-10 19:43:44

EgonOlsen (3198 views)
2018-06-10 19:43:20

DesertCoockie (4095 views)
2018-05-13 18:23:11

nelsongames (5115 views)
2018-04-24 18:15:36
A NON-ideal modular configuration for Eclipse with JavaFX
by philfrei
2019-12-19 19:35:12

Java Gaming Resources
by philfrei
2019-05-14 16:15:13

Deployment and Packaging
by philfrei
2019-05-08 15:15:36

Deployment and Packaging
by philfrei
2019-05-08 15:13:34

Deployment and Packaging
by philfrei
2019-02-17 20:25:53

Deployment and Packaging
by mudlee
2018-08-22 18:09:50

Java Gaming Resources
by gouessej
2018-08-22 08:19:41

Deployment and Packaging
by gouessej
2018-08-22 08:04:08
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!