10.06.06
Posted in Coding, Sysadmin, Tech/Geek at 3:20 pm by Craig
I’ve been thinking of using S3 to store backups of various machines (basically all linux/OSX ones), but what’s been holding me back is the inability of S3 to do rsync on the server side. rsync really needs an instance of rsync running “near” where the data is stored in order to do its cleverest compression/do-not-transmit smarts. Rsync is basically a win if you have a high-bandwidth link between the rsync server and the backing store, and a lower bandwidth link beterrn the rsync server and client. With S3, you’d have to run the rsync server side yourself, remote from S3, which kind of defeats the purpose of rsync…
But then I had a brainstorm. Amazon’s ECC service, which parallels S3, allows you to create a virtual machine and turn it on/off as needed in the amazon compute cloud. The ECC instances have high-bandwidth connectivity to S3 storage, and so would be ideal for running an rsync server! You can set up an ECC instance which serves rsync, and then your backup script can turn the instance on, do the rsync, then shut the instance down when it’s done.
Now all I have to do is actually create the ECC instance, then create some kind of wrapper around the whole thing which does the startup-backup-shutdown wrapping around the ECC API, and voila!
Permalink
09.22.05
Posted in Sysadmin at 11:32 pm by Craig
This is just a note to self for later when I’ll need to do this (once my second disk gets plugged into my new hosted machine):
for i in 0 1 2 3;do /sbin/mdadm --grow /dev/md$i --raid-disks=2;done
/sbin/mdadm --manage /dev/md0 --add /dev/sdb1
/sbin/mdadm --manage /dev/md1 --add /dev/sdb2
/sbin/mdadm --manage /dev/md2 --add /dev/sdb3
/sbin/mdadm --manage /dev/md3 --add /dev/sdb4
Permalink
04.22.05
Posted in Sysadmin at 8:56 am by Craig
I loved it when cron sent me this email. Actually, it sent it to me yesterday around noon, but I just received it now, and I’m pleased as punch:
WARNING: Some disks in your RAID arrays seem to have failed!
Below is the content of /proc/mdstat:
Personalities : [linear] [raid0] [raid1] [raid5] [multipath] [raid6]
md1 : active raid1 hdg1[1] hde1[0]
152512 blocks [2/2] [UU]
md2 : active raid1 hdg2[1] hde2[0]
883456 blocks [2/2] [UU]
md0 : active raid1 hdg3[1] hde3[2](F)
159043392 blocks [2/1] [_U]
Permalink
04.24.04
Posted in Sysadmin at 10:09 pm by Craig
Ok, so since last we met our hero, the following has transpired:
- Replacing RAM and moving it around to different slots seems to have resolved the won’t POST issue. But, the hi-lo-hi-lo warble continues. I can actually boot the box, get it all up and everything, and the warble tone just continues. Sounds like maybe a temperature alarm or something, but temps of the CPUs and all other components are just fine. Could be a case-open alarm, but not sure. Hopefully MSI will know and help me stop it
- I’ve decided to make this a 2.6 box – and lo! Gentoo is 2.6 now, so I’m gonna do that
- Crap. ATA Raid is not really there in 2.6 yet. Bugger. Ok, well I guess for now one of my two 160GB disks will just sit idle, and once 2.6.x support ATA RAID, I’ll do the RAID-1 then, mirroring the existing drive.
- Gentoo’s quite cool – they have x86-64 stuff very nicely done, with 2.6 kernel and all. Quite straightforward to get things all up and going in fullblown 64 bit mode
More to follow.
Permalink
04.21.04
Posted in Sysadmin at 10:39 pm by Craig
So, got a new power supply. Now the stupid machine won’t post. Just gives a hi-lo-hi-lo-hi-lo speaker warble. All the BIOS POST code pages I can find which list this symptom claim that it’s due to the CPU overheating, which is clearly a crock in this case. However, after hours of trying all kinds of combinations of different components, I think the problem is with the RAM. I got DDR400 RAM, since the price was the same as the slower stuff, and I figured heck, worst case I’m underclocking and it’ll work fine. Not so. Seems like RAM got all smart all of a sudden since I last bought some. After not having any luck at all on the MSI support pages, nor on the various BIOS POST debugging pages, I finally searched AMD’s website, and lo, under “recommended motherboards”,
this gem:
Motherboard Notes:
- Supports registered DDR400 memory with Opteron™ 246 models and above.
Great. I, of course, sprang only for the 244s, not 246s. So I get to make trip #3 to Fry’s tomorrow, and either *downgrade* my RAM, or upgrade my CPUs. Probably the former, since the latter costs money. Damn, but that’s annoying though. You would have thought that DDR400 RAM could just be smart and run at 333 or 200 or something. But apparently not. Well, at leasty I hope that’s the problem. We’ll see in the morning when Fry’s opens.
Permalink
11.18.03
Posted in Sysadmin at 9:52 am by Craig
Heard back from the mod_torrent guy, who’s a student, and has been pretty busy, which is why it took him a while to get back to me. I’m now added as a project admin on the mod_torrent project on sf.net, and I’ve decided to actually spec this sucker out and start coding. First step in this process was finally getting around to upgrading this box to apache2 (don’t think it makes much sense to start a new dev project based on apache1 right now). The upgrade seems to have been pretty seamless — in any case this blogs seems to be working, as do most of the other bits I’ve looked at on the server here so far… Let me know if any of you notice anything broken. Especially if it looks like it’s due to the upgrade.
Permalink
07.03.03
Posted in Sysadmin at 2:53 am by Craig
So recently, I’ve been doing a fair amount of downloading and uploading (where frequently for extended periods of time, my DSL line upstream and downstream bandwidth is completely saturated). This leads to me not being able to do any kind of foreground internet access, like browsing the web, or opening SSH connections, etc. I figured there’s got to be some neat way of using the linux QoS tools to help out with this, possibly also using iptables for identifying which packets to prioritize. So here’s what I ended up doing (in a nutshell):
###############################################################################
## Special Chain MANGLE_TOS (used by MANGLE_OUTPUT and MANGLE_PREROUTING)
$IPTABLES -t mangle -N MANGLE_TOS
$IPTABLES -t mangle -F MANGLE_TOS
# First, mark any known-bulk ports bulk as bulk
for bulk_port in $BULK_PORTS; do
$IPTABLES -t mangle -A MANGLE_TOS -p tcp --sport $bulk_port -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A MANGLE_TOS -p tcp --dport $bulk_port -j TOS --set-tos Maximize-Throughput
done
# if Normal-Services, then prioritize to Minimum-Delay by default
$IPTABLES -t mangle -A MANGLE_TOS -m tos --tos Normal-Service -j TOS --set-tos Minimize-Delay
# If the packet is anything but Minimize-Delay now, then let it go
$IPTABLES -t mangle -A MANGLE_TOS -m tos --tos ! Minimize-Delay -j RETURN
# Additionally, set TOS to Maximize-Throughput on large packets (for eg rsync over ssh)
# but, allow the occasional large packet to retain its TOS (2 per second) in case someone
# is doing something like cat'ing a mid-size file over ssh, etc.
# So to implement this, drop out of this chain if the packet is small...
$IPTABLES -t mangle -A MANGLE_TOS -p tcp -m length --length 0:512 -j RETURN
# We drop off the chain if the limit of 2/s (burst 10) is not exceeded...
$IPTABLES -t mangle -A MANGLE_TOS -m limit --limit 2/s --limit-burst 10 -j RETURN
# Then if we're still in the chain, make this packet Maximize-Througput
$IPTABLES -t mangle -A MANGLE_TOS -j TOS --set-tos Maximize-Throughput
###############################################################################
I then add that chain onto my OUTPUT and PREROUTING chains in the mangle table. BULK_PORTS is initialized to stuff like ftp-data, kaazaa ports, bittorrent ports, etc. The neat part here is that it’ll automatically catch anything which I hadn’t anticipated was going to be a bulk transfer (ie lots of packets of size >512 bytes), while still allowing small packets to jump up in priority, and even allowing the occasional large packet to still be prioritized high. Those last 5 iptables commands are pure gold!
Then, I set up some priority queues on my DSL pppoe interface, which treat packets specially based on their TOS, and also treat some “special” packets specially:
# clean existing down- and uplink qdiscs, hide errors
$TC qdisc del dev ppp0 root 2> /dev/null > /dev/null
$TC qdisc del dev ppp0 ingress 2> /dev/null > /dev/null
# Use a root HTB division system
$TC qdisc add dev ppp0 root handle 1: htb default 20
# Throttle the total uplink speed to prevent queues in the DSL modem
$TC class add dev ppp0 parent 1: classid 1:1 htb rate ${UPLOAD}kbit burst 6k
# Create two channels going out -- one high priority, the other low priority, but allow them to borrow bandwidth from each other if the other's idle
$TC class add dev ppp0 parent 1:1 classid 1:10 htb rate 64kbit ceil ${UPLOAD}kbit burst 6k prio 1
$TC class add dev ppp0 parent 1:1 classid 1:20 htb rate 32kbit ceil ${UPLOAD}kbit burst 6k prio 2
# Create queues for both of these classes
$TC qdisc add dev ppp0 parent 1:10 handle 10: sfq perturb 10
$TC qdisc add dev ppp0 parent 1:20 handle 20: sfq perturb 10
# Set up filters for these classes
# TOS Minimum Delay gets high priority
$TC filter add dev ppp0 parent 1:0 protocol ip prio 1 u32 match ip tos 0x10 0xff flowid 1:10
# ICMP get high priority
$TC filter add dev ppp0 parent 1:0 protocol ip prio 2 u32 match ip protocol 1 0xff flowid 1:20
# ACK packets get high priority
$TC filter add dev ppp0 parent 1:0 protocol ip prio 1 u32 \
match ip protocol 6 0xff \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xffc0 at 2 \
match u8 0x10 0xff at 33 \
flowid 1:10
# Now slow downloads slightly, to prevent queuing at ISP
$TC qdisc add dev ppp0 handle ffff: ingress
$TC filter add dev ppp0 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate ${DOWNLOAD}kbit burst 10k drop flowid :1
So here is where we do the actual prioritization. Basically, anything which is TOS=Minimize-Delay gets on the high-priority queue, as do ICMP packets (for low apparent ping times), and tcp ACK packets (so the downloads aren’t getting slowed by having their ACKs delayed). The ingress filter at the end drops any packets which are overloading my max download speed. The purpose here is to use TCP to keep the download buffers at the ISP from filling up, which would cause any ACKs coming back to us to get delayed, as would any packets coming back to us as the results of our outbound Minimize-Delay packets.
These few filtering rules have revolutionized my ability to make good use of my bandwidth — I can now keep a high sustained upload/download going more or less all the time, while not impacting my ability to do interactive networking. So I can participate in a number of bittorrents inside virtual screens on my server at home (might take a couple days to get a whole torrent stream downloaded, and meanwhile others will be uploading from my copy), and then still be able to remotely login to my machine. Meanwhile, people visiting my website will also see a much lesser slowdown than would otherwise be the case.
Next week’s episode: Trying this on Windows using the QoS stuff there (hopes not high).
Permalink
06.25.03
Posted in Sysadmin at 10:02 pm by Craig
Ok, the switchover to WordPress from B2 is now completed. The main site is now running off the new software. Please let me know about any problems you hit.
Permalink
