07.13.03
Posted in Life at 11:14 am by Craig
After starting to have contractions at about 7am on Friday, Erica worked and worked through friday and saturday, eventually going in to the hospital to alleviate the pain and get rehydrated when she started to get really tired out. Then, after the contractions picked up again, she popped out a very pretty girl, Evan Alice, who weighed 7lbs 4oz at πam sunday morning. Both mother and baby are now home again, and doing great, gurgling at each other on the bed behind me as I type. The paucity of bandwidth here at home has for now limited me to just one photo uploaded to our
gallery, but more will be joining that one soon.
Permalink
07.03.03
Posted in Sysadmin at 2:53 am by Craig
So recently, I’ve been doing a fair amount of downloading and uploading (where frequently for extended periods of time, my DSL line upstream and downstream bandwidth is completely saturated). This leads to me not being able to do any kind of foreground internet access, like browsing the web, or opening SSH connections, etc. I figured there’s got to be some neat way of using the linux QoS tools to help out with this, possibly also using iptables for identifying which packets to prioritize. So here’s what I ended up doing (in a nutshell):
###############################################################################
## Special Chain MANGLE_TOS (used by MANGLE_OUTPUT and MANGLE_PREROUTING)
$IPTABLES -t mangle -N MANGLE_TOS
$IPTABLES -t mangle -F MANGLE_TOS
# First, mark any known-bulk ports bulk as bulk
for bulk_port in $BULK_PORTS; do
$IPTABLES -t mangle -A MANGLE_TOS -p tcp --sport $bulk_port -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A MANGLE_TOS -p tcp --dport $bulk_port -j TOS --set-tos Maximize-Throughput
done
# if Normal-Services, then prioritize to Minimum-Delay by default
$IPTABLES -t mangle -A MANGLE_TOS -m tos --tos Normal-Service -j TOS --set-tos Minimize-Delay
# If the packet is anything but Minimize-Delay now, then let it go
$IPTABLES -t mangle -A MANGLE_TOS -m tos --tos ! Minimize-Delay -j RETURN
# Additionally, set TOS to Maximize-Throughput on large packets (for eg rsync over ssh)
# but, allow the occasional large packet to retain its TOS (2 per second) in case someone
# is doing something like cat'ing a mid-size file over ssh, etc.
# So to implement this, drop out of this chain if the packet is small...
$IPTABLES -t mangle -A MANGLE_TOS -p tcp -m length --length 0:512 -j RETURN
# We drop off the chain if the limit of 2/s (burst 10) is not exceeded...
$IPTABLES -t mangle -A MANGLE_TOS -m limit --limit 2/s --limit-burst 10 -j RETURN
# Then if we're still in the chain, make this packet Maximize-Througput
$IPTABLES -t mangle -A MANGLE_TOS -j TOS --set-tos Maximize-Throughput
###############################################################################
I then add that chain onto my OUTPUT and PREROUTING chains in the mangle table. BULK_PORTS is initialized to stuff like ftp-data, kaazaa ports, bittorrent ports, etc. The neat part here is that it’ll automatically catch anything which I hadn’t anticipated was going to be a bulk transfer (ie lots of packets of size >512 bytes), while still allowing small packets to jump up in priority, and even allowing the occasional large packet to still be prioritized high. Those last 5 iptables commands are pure gold!
Then, I set up some priority queues on my DSL pppoe interface, which treat packets specially based on their TOS, and also treat some “special” packets specially:
# clean existing down- and uplink qdiscs, hide errors
$TC qdisc del dev ppp0 root 2> /dev/null > /dev/null
$TC qdisc del dev ppp0 ingress 2> /dev/null > /dev/null
# Use a root HTB division system
$TC qdisc add dev ppp0 root handle 1: htb default 20
# Throttle the total uplink speed to prevent queues in the DSL modem
$TC class add dev ppp0 parent 1: classid 1:1 htb rate ${UPLOAD}kbit burst 6k
# Create two channels going out -- one high priority, the other low priority, but allow them to borrow bandwidth from each other if the other's idle
$TC class add dev ppp0 parent 1:1 classid 1:10 htb rate 64kbit ceil ${UPLOAD}kbit burst 6k prio 1
$TC class add dev ppp0 parent 1:1 classid 1:20 htb rate 32kbit ceil ${UPLOAD}kbit burst 6k prio 2
# Create queues for both of these classes
$TC qdisc add dev ppp0 parent 1:10 handle 10: sfq perturb 10
$TC qdisc add dev ppp0 parent 1:20 handle 20: sfq perturb 10
# Set up filters for these classes
# TOS Minimum Delay gets high priority
$TC filter add dev ppp0 parent 1:0 protocol ip prio 1 u32 match ip tos 0x10 0xff flowid 1:10
# ICMP get high priority
$TC filter add dev ppp0 parent 1:0 protocol ip prio 2 u32 match ip protocol 1 0xff flowid 1:20
# ACK packets get high priority
$TC filter add dev ppp0 parent 1:0 protocol ip prio 1 u32 \
match ip protocol 6 0xff \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xffc0 at 2 \
match u8 0x10 0xff at 33 \
flowid 1:10
# Now slow downloads slightly, to prevent queuing at ISP
$TC qdisc add dev ppp0 handle ffff: ingress
$TC filter add dev ppp0 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate ${DOWNLOAD}kbit burst 10k drop flowid :1
So here is where we do the actual prioritization. Basically, anything which is TOS=Minimize-Delay gets on the high-priority queue, as do ICMP packets (for low apparent ping times), and tcp ACK packets (so the downloads aren’t getting slowed by having their ACKs delayed). The ingress filter at the end drops any packets which are overloading my max download speed. The purpose here is to use TCP to keep the download buffers at the ISP from filling up, which would cause any ACKs coming back to us to get delayed, as would any packets coming back to us as the results of our outbound Minimize-Delay packets.
These few filtering rules have revolutionized my ability to make good use of my bandwidth — I can now keep a high sustained upload/download going more or less all the time, while not impacting my ability to do interactive networking. So I can participate in a number of bittorrents inside virtual screens on my server at home (might take a couple days to get a whole torrent stream downloaded, and meanwhile others will be uploading from my copy), and then still be able to remotely login to my machine. Meanwhile, people visiting my website will also see a much lesser slowdown than would otherwise be the case.
Next week’s episode: Trying this on Windows using the QoS stuff there (hopes not high).
Permalink
