How I spent my Saturday night

Folks, this is going to be a long story, one that I’m certain explains my eHarmony results. Please, grab a cup of coffee spiked with the liquor of your choosing (you’ll need it), sit down, and dive into my tale of Active Directory chaos. It’s a bit geeky, but I’ll try to explain stuff as I go along.

First, some background: I’m currently living alone right now, as my new roommate is finishing up a summer semester abroad. (He’s back in a couple weeks.) My intent since I learned my old roommate was moving out has been to relocate my base of operations, including my twin bed, to the room in the back; this will permit me to simplify my network topology a lot (i.e. far fewer random wires in main hallways and such). The second goal was to replace nightwind, the Windows 2000 Advanced Server that has been my primary Active Directory server for the last year and a half, as the machine responsible for operating the weather station and, ultimately, responsible for the AD. (Active Directory is a Microsoft technology used to assist in administering large networks; I run it at home as a testing tool that I simply can’t do in a work environment.) Nightwind had recently developed some problems with its disk, too, so I wanted to get it out of any seriously mission-critical apps before the whole topology went crumbling down. Third, I wanted to finish migrating my Active Directory to an all-Windows Server 2003 environment, enabling additional features and security. All are definitely doable things, and a major plus.

The machine that was to replace nightwind in operating the weather station is named healer, a little Dell Dimension 4100 I picked up some time back after doing a side job. It’s a relatively small, quiet machine that is an ideal fit sitting below the station as it’s much more out of the way than Nightwind’s giant freaking case. Some time ago, I loaded Windows Server 2003 on it and managed to configure it as a backup domain controller for Nightwind — this way, I could service user logons if Nightwind ever developed a problem.

Then, the power went haywire on the side of my room where the computers lived. This forced a rearranging of the room, but unfortunately diminished the amount of available power. Thus, healer was shut off, and not brought back up for a while. I kept meaning to reinstate healer as the weather station operator, but that never came to fruition, so the box just stayed off.

I recently acquired another Windows server, known as thearbor, which I commissioned as a domain controller in tandem with Nightwind. (It also does a lot of other really neat stuff.) The goal was to migrate my domain to rally around thearbor, which would permit me to demote nightwind and go to the all-2003 environment I was craving. The final piece of the puzzle, really, was to have a hungover Saturday in which I would not want to venture outside but wouldn’t mind working on computers all day.

That Saturday arrived this past weekend. After lounging around and watching Airport ’79: The Concorde (arguably the hokiest Airport movie but it’s got the Concorde in it, so it has to count for SOMETHING), I got off my ass and started making trouble. The first step was to bring healer up and transfer the weather station to it. It was here where I further confirmed that Dell Dimensions are not built to do much beyond general Web surfing — after plugging in the station, it immediately began giving phantom wind speed readings of 28.4 and 14.1 MPH. WTF, mate? I tweaked serial port settings and upgraded the WUHU software that I use to manage the automated upload to Weather Underground to no avail — so healer failed this test. I left it hooked into the network to service DNS and proceeded to shut down and unhook the other machines for the move to the other room.

After relocating the machines and the network hub that supported them, I fired them back up and left them in their state. Once thearbor and nightwind were up and running, it was time to relocate healer — it was not going to serve the weather station as I had predicted it would. It was nice to be able to finally get rid of that awful long cable that draped over the counter into the back utility/laundry/miscellany room for the last year. (That cable now snakes through the bathroom, a much less intrusive pathway when I get it all taped down and stuff.) However, this also meant some weather station downtime that I wasn’t expecting to happen. (Sorry, folks.)

When I was plugging the machines back in, I ended up plugging in thearbor’s network cable to the wrong NIC (it has three). Thus, it pulled a number from DHCP…and I had one hell of a time to get DNS to forget this number. Once I moved the cable back to the right connection, it was slowing down communications to the box because it was trying on the DHCP’d address that was now invalid, rather than the static address. This turned out to be a really lousy problem. More on this in a bit.

I then connected nightwind to a monitor and a console and began working on demoting it as a domain controller. It then let me know that I needed to make sure I had another “global catalog” server — a server that basically holds all the juicy details of the Active Directory — because if I demoted nightwind, which was one of these, then I would basically toast the whole shebang. Sure enough, I needed to assign another machine as a global catalog — with the help of the Windows 2003 Support Tools, I was able to configure thearbor as one of these machines. Once I confirmed that thearbor was a functioning global catalog (which was hellacious because nightwind’s permissions were actually breaking the directory’s replication!), I ran the demotion stuff on nightwind. It transferred all the roles to thearbor, as I requested, and it just worked. Taps played, and it was no longer a domain controller. The best part was that logons and everything were processing properly — minor victory no. 2.

The next step was to see if healer had picked up all the changes to the domain. Peculiarly, it hadn’t gotten the changes (which were many), so I popped into the Event Viewer to see what was up. I then got slapped in the face with a lovely Active Directory reality — if a domain controller is off for more than 60 days, it’s “tombstoned” and not permitted to replicate. Worse, a tombstoned machine usually necessitates, at the very least, a forced removal of Active Directory and cleanup on the domain controller. It’s not unprecedented to have to reinstall the operating system. E-gad, Brain! However, the forcible removal worked like a champ, and cleanup wasn’t too bad, either. ntdsutil is a neat little toy, if not one of the most dangerous things ever. :P I was able to reassign the domain controller role to healer, and it began replicating properly. Minor victory no. 3.

I then began running some exhaustive diagnostics on nightwind, trying to isolate the disk problems. The disk came up clean (even though it locked up cold when rebooting once), and it just needed a defragment as well as disabling the antivirus protection, which was destroying disk performance. This took a while to realize, though. In the meantime, I was trying to figure out what in God’s name was going on with the DNS. The A record linking thearbor to the DHCP address it had for all of 30 seconds wouldn’t frickin’ die. So, I began reconfiguring healer to use a secondary, read-only copy of the zone. I then realized that you don’t do this to an Active Directory domain controller, as I watched my entire DNS pretty much delete itself before my eyes for some inexplicable reason. This brought a moment of panic that was luckily avoided by just restarting the Net Logon service on both boxes (and telling healer to act as an Active Directory Integrated DNS server again, like it should have always been in the first place). I chalked this awful error up to it being 3:30 in the morning. I continued to fight with the A record until about 4, when I finally was able to eradicate it by digging a bit deeper into the DNS configuration. Ridiculous. But, nonetheless, minor victory no. 4.

It then became time to wheel nightwind back into the other room to once again serve as the weather station server — hey, it did a great job before. I upgraded WUHU on nightwind before disconnecting it from the console, and then started unplugging stuff. Chalk it up to it being about 4:10 AM at this point, but I unplugged thearbor from the wall instead. Frightening. Just what I needed — an instant test of fault tolerance. Luckily, stuff kept operating, and it all bounced back to normal after I restored power to thearbor. I finished off by elevating the domain to a Windows Server 2003-exclusive operational level, which enabled all sorts of new obscure administrative features which may one day make life easier.

4:30 AM arrived — time to sleep, this time with a fully upgraded network, and a weather station that not only reported the proper wind, but also reported it finally with a decimal point of precision. Before, WUHU would just ignore the decimal when uploading to Weather Underground, so it was nice to reclaim that functionality. An increase in accuracy is always a good thing.

A day later, things are still operating like clockwork. The DNS stupidity was resolved, and life is good. But man, oh man, what a trip to get there. Next Saturday I think I’ll spend it doing something a little less geeky. We shall see…

3 thoughts on “How I spent my Saturday night

  1. Chuck Boyd

    Well, it took TWO cups of coffee and I’m not really sure what you said BUT you succeeded so congratulations. The double hum of machinery and the hum of satisfaction.

    Great 16 day vacation (retirement = time off from doing nothing?)and I had a great return to Los Angeles. We stayed in the Westin Bonaventure downtown. When I was last in L.A. the site was a huge hole in the ground as it was being built! They’ve been open more than 30 years.

    Did all the tourist things in San Francisco, in Los Angeles and even in Denver where I visited my other daughter. Saw the Giants lose in the 8th inning (Bonds 0-4) then sat back and watched Memorial Day weekend fireworks go up and disappear in the San Fran fog. Neat.

    Give me a buzz and we’ll have some Boddingtons.

    Chuck

  2. NotMyBest2Day

    That actually sounded pretty fun. I love network upgrades. I run all of my cabling through the walls and attic though. I have some computer work to do, as well. I’m waiting on a 500gb SATA-II drive to come back from being replaced so I can get my 4x500gb raid-5 going for storage in my dual Socket F server. In order to do that though, I have to upgrade my BIOS, but the upgrade requires Windows to be reinstalled, and I need to do that anyway to get rid of the windows updates for 2000.

    I’ve tested this already, and without the updates, I can DL at 1.06MB/sec (10mbit ethernet from modem to router), and as soon as I apply any random or all of the updates, it goes down to 340k/sec and that’s it. It was fine this time last year, and considering June 2006 was the end of the original support cycle, I’m pretty sure it’s a built-in “feature” to try to get people to upgrade to the newer OS that doesn’t have that problem. I need to do some more extensive testing (installation with the CMOS clock set for 2005) to verify my claim, but I’m positive that it’s true.

    Then I’m taking my four IDE drives that are in use right now and putting them in another server which is a dual Socket A server, and I’m getting a TV tuner card to put in there to record and encode. I’m getting a gigabit NIC to slap in and connect with a crossover to my Socket F server. It’s going to be a pretty nice setup.

    I have also successfully figured out how to set my Linux server up to be an ad-hoc access point. Unfortunately, the Wii doesn’t like ad-hoc–it wants a real, genuine access point, and I’m not buying one. They just just let it use ad-hoc networks.

    Kind of off topic, but as I was typing this reply, cable went out, but only TV. Internet is still up..for some unknown and very strange reason. That never happens like that. Not that I’m complaining.

  3. Chris M

    And to think all I have is a Win2K advanced server machine running at the fire dept. Its only real purpose is to authenticate logon and serve as a fileserver. I really don’t know enough to dick with it much more…lol. My most recent victory was getting VPN to work between two VPN firewalls so that the remote station out in bumblefuck can ues the network – sorta. They bought an XP Home machine….yuck….so no user or machine group policy or domain logon.

Comments are closed.