Today's Outage

Well something caused the web and mail servers CPU load ( load average ) to spike to levels that caused the server to become mostly unresponsive.

I became aware of the situation early this morning while at work but was unable to leave work to correct the issue. In the rare instances in which I need to log into the server to correct an issue I can usually leave work to fix the problem.

Today I was not able to until after I had gotten home from work.

I am investigating the cause of the CPU load spike and will let you know what I find.

Sorry for the inconvenience.

Move to new datacenter is complete...

Ok, the servers have been moved and are running at the new datacenter. It will take some time for all the DNS information to propogate so some domains will not resolve for up to 24 hours.

However, the IP address of the server is 65.197.143.152. You can use this IP address to SSH or FTP to the server until your domain DNS resolves.

I just did an FTP transfer test and the speed *IS AMAZING*.. You'll be satisfied with the connectivity now.

Servers are at new datacenter

The servers are now at the new datacenter. I am now just waiting for the datacenter techs to install the servers in the racks and turn the power on.

All DNS information is updated and will propogate over the next day or so. I expect that by later tonight all service will be restored.

Thanks again for your patience.

Move to datacenter update

The servers are now in shiny new 4U Rackmount cases and on the way to the datacenter to be installed and hooked up to a new multi-homed backbone.

By the end of today I expect everything to be reconfigured as new IP's have been assigned.

It may take a day for all DNS servers to get the updated information.

Thanks again for your patience. With this move the service will be greatly improved.

Network Outage

The server(s) are unreachable at this time due to an upstream network outage. I've contacted the upstream provider and they gave an estimate of about 15 to 20 minutes for the network to be repaired.

So I expect the server(s) to be reachable again by about 2:00PM PST or sooner.

This is a good time for me to help you figure out where the problem is.

If you are running windows you should open a DOS Command window and type:

C:\>tracert midvalleyhosting.com

If you are running some form of Unix you probably have the traceroute command so you'd type:

#>traceroute midvalleyhosting.com

The output you get should look similar to this:

Tracing route to midvalleyhosting.com [65.249.23.250]
over a maximum of 30 hops:

 1   <10 ms   <10 ms   <10 ms  10.122.1.10
 2   <10 ms     1 ms   <10 ms  63.224.205.158
 3    42 ms    42 ms    41 ms  eugn-dsl-gw03-195.eugn.qwest.net [67.42.192.195]

 4    42 ms    42 ms    42 ms  eugn-agw1.inet.qwest.net [67.42.192.93]
 5    42 ms    42 ms    42 ms  egn-core-01.inet.qwest.net [205.171.150.33]
 6    51 ms    55 ms    48 ms  stl-core-01.inet.qwest.net [205.171.5.125]
 7    48 ms    47 ms    48 ms  sea-brdr-01.inet.qwest.net [205.171.26.54]
 8    47 ms    47 ms    47 ms  POS2-3.BR1.SEA1.ALTER.NET [204.255.169.29]
 9    47 ms    47 ms    49 ms  0.so-7-3-0.xl1.sea1.alter.net [152.63.105.18]
10    52 ms    51 ms    51 ms  0.so-6-2-0.xl1.por3.alter.net [152.63.105.233]
11    51 ms    51 ms    52 ms  0.so-7-0-0.xr1.por3.alter.net [152.63.107.105]
12    53 ms    51 ms    52 ms  191.atm4-0.gw4.por3.alter.net [152.63.106.33]
13    60 ms    61 ms    60 ms  alyrica-gw.customer.alter.net [157.130.180.38]
14     *        *        *     Request timed out.
15    92 ms    65 ms    65 ms  router.midvalleyhosting.com [65.197.143.200]
16    68 ms    68 ms    67 ms  hosting.midvalleyhosting.com [65.249.23.250]

Trace complete.

Notice that the last "hops" are router.midvalleyhosting.com and hosting.midvalleyhosting.com. If you see these two lines then you have a good connection all the way to midvalley hostings service. If somewhere along the way you start getting "Destination host unreachable" errors etc then you know where the problem is.

For example, the outage that is happening now produced the following tracert output:

Tracing route to dailysnapshots.org [65.249.23.250]
over a maximum of 30 hops:

 1   <10 ms   <10 ms   <10 ms  10.122.1.254
 2     1 ms     1 ms   <10 ms  63.224.205.158
 3    42 ms    41 ms    43 ms  eugn-dsl-gw03-195.eugn.qwest.net [67.42.192.195]

 4    41 ms    42 ms    43 ms  eugn-agw1.inet.qwest.net [67.42.192.93]
 5    42 ms    42 ms    42 ms  egn-core-01.inet.qwest.net [205.171.150.33]
 6    47 ms    48 ms    67 ms  stl-core-01.inet.qwest.net [205.171.5.125]
 7    48 ms    47 ms    48 ms  sea-brdr-01.inet.qwest.net [205.171.26.54]
 8    47 ms    48 ms    47 ms  POS2-3.BR1.SEA1.ALTER.NET [204.255.169.29]
 9    48 ms    48 ms    48 ms  0.so-7-3-0.xl1.sea1.alter.net [152.63.105.18]
10    52 ms    51 ms    52 ms  0.so-6-2-0.xl1.por3.alter.net [152.63.105.233]
11    52 ms    52 ms    52 ms  0.so-7-0-0.xr1.por3.alter.net [152.63.107.105]
12    53 ms    53 ms    51 ms  191.atm4-0.gw4.por3.alter.net [152.63.106.33]
13    60 ms    60 ms    61 ms  alyrica-gw.customer.alter.net [157.130.180.38]
14     *        *        *     Request timed out.
15  host150.alyrica.com [65.197.143.150]  reports: Destination host unreachable

Here you can see that alyrica.com has a problem and the packets are not even making it to midvalley hostings servers.

DNS Changes - Post Server(s) Move

It has come to my attention since the server move that some of the domains have DNS information that is not being propogated to all DNS servers. This means that in some cases people will not be able to browse to certain websites that are hosted by
midvalleyhosting ( me ).

I am currently in the process of making sure that *EVERY* domain has 3 nameservers to avoid any further DNS failures in the future.

A few domains have only 1 valid master DNS server ( ns.midvalleyhosting.com ) and of course it's IP address has changed but not all DNS server know about the change. This is what is causing some DNS issues currently. I believe only 2 of the 34 domains that I host still have this issue.

Please continue to let me know of any problems that you encounter as a result of the move.

Thanks again for your patience.

Server move is completed!

Ok folks... so the move is completed. The DNS server, mail server, ftp server and all other server software has been reconfigured and as far as I can tell everything is functioning as expected.

Please let me know if you have any problems.

This morning it seems to me that the connection to the server(s) is slow. Let me know if you notice slow connections and slow website page reloads etc. If this period of slow connections continues, I'll be calling my bandwidth provider and ask to get it fixed.

Server(s) are back online...

The server(s) are back online. Some functions may not be operating correctly yet, but as far as I know the webserver, ftp server and the mail server are operational.

Please let me know if you encounter any problems. I will be continuing to test everything as the day progresses.

The move is on!

Well it's now 1:22AM PST and the new temporary DNS server is in place. I am done for tonight.

Expect the main server(s) to be partially if not fully operational sometime tommorrow.

Server(s) being moved tonight!

As mentioned previously via email, the server(s) are being moved this weekend to a new facility which provides double the bandwidth.

During this time expect interruptions of service. No incoming email will lost, they will be queued up for delivery by the sending mail server and delivered when the server(s) are back online.