Question
This all started when I was trying to download a podcast on my Mac. This happens in both Snow Leopard and Lion when the Ethernet interface is set to use Jumbo Frames.
prompt> curl -v -o x.mpg http://audio.wnyc.org/freakonomics_specials/freakonomics_specials041112.mp3
* About to connect() to audio.wnyc.org port 80 (#0)
* Trying 204.93.180.53... Local Interface en0 is ip 172.16.1.2 using address family 2
* Local port: 0
* connected
* Connected to audio.wnyc.org (204.93.180.53) port 80 (#0)
> GET /freakonomics_specials/freakonomics_specials041112.mp3 HTTP/1.1
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8r zlib/1.2.3
> Host: audio.wnyc.org
> Accept: */*
>
OK
< Server: nginx/0.7.65
< Date: Mon, 16 Apr 2012 23:39:03 GMT
< Content-Type: audio/mpeg
< Content-Length: 42075070
< Last-Modified: Tue, 10 Apr 2012 21:15:08 GMT
< Connection: close
< Content-Disposition: attachment
< Accept-Ranges: bytes
<
0 40.1M 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0^C
The headers come through fine, but the download never gets anywhere. This is the only web server I have this kind of trouble with, but it’s still annoying and I’d like to see if there’s a fix other than forgoing jumbo frames everywhere.
I determined that I can download a partial file as long as the size of the chunk I’m downloading is 1448 bytes or less. I can use range 0-1447 or 10000-11447 so it’s not the position in the file, it’s the size of the chunk. The WAN MTU on my router had been set to 1500, so I tried reducing it in steps until I got to 1400 and it still didn’t make any difference.
I was thinking this was a problem with Path MTU Black Hole discovery because the problem goes away when I stop using jumbo frames on the ethernet interface. But I have everything set up for black hole discovery (as far as I can tell) and ping doesn’t see any problem:
ping -g1435 -G1445 204.93.180.53PING 204.93.180.53 (204.93.180.53):
(1435 ... 1445) data bytes
1443 bytes from 204.93.180.53: icmp_seq=0 ttl=51 time=69.223 ms
1444 bytes from 204.93.180.53: icmp_seq=1 ttl=51 time=75.542 ms
1445 bytes from 204.93.180.53: icmp_seq=2 ttl=51 time=72.136 ms
1446 bytes from 204.93.180.53: icmp_seq=3 ttl=51 time=73.732 ms
1447 bytes from 204.93.180.53: icmp_seq=4 ttl=51 time=72.057 ms
1448 bytes from 204.93.180.53: icmp_seq=5 ttl=51 time=73.377 ms
1449 bytes from 204.93.180.53: icmp_seq=6 ttl=51 time=71.717 ms
1450 bytes from 204.93.180.53: icmp_seq=7 ttl=51 time=73.293 ms
1451 bytes from 204.93.180.53: icmp_seq=8 ttl=51 time=71.874 ms
1452 bytes from 204.93.180.53: icmp_seq=9 ttl=51 time=73.206 ms
1453 bytes from 204.93.180.53: icmp_seq=10 ttl=51 time=71.289 ms
In fact, ping works all the way up to 1494 bytes, though I believe the Mac counts the bytes differently than other *nixes. (I think it counts as data the 8 bytes of the ICMP header, but not the 20 bytes of IP header, unlike most which don’t count either and some which count both.)
I don’t want to give up the performance benefits of jumbo frames on my LAN just for this one broken web site, but of course I want my podcast. So I’m looking for suggestions for things to try.
My Mac has 2 built-in ethernet ports, so one thing I tried was connecting the second one with MTU 1500 and forcing curl to use that port. Curl said it was using that port, but the MTU of that port had no effect on whether the download worked or not. It was the MTU of the first active ethernet port that mattered. I don’t know what that means, either.
Suggestions, anyone?
Answer
Workarounds:
- Lower the interface MTU to 1500. System Preferences -> Network, select interface (e.g. Built-in Ethernet), click Advanced… button, Ethernet tab, MTU drop-down or set Configure to Automatically. This solves the problem, but means you cannot use Jumbo Frames on the LAN (at least from this computer).
- If this were on Linux: use
iptablesto clamp the TCP MSS for just this host.--set-mss 1460
Solution:
- Fix the server configuration to either lower the MTU or make Path MTU Discovery work.
As it happens, I was finally able to find someone to talk to at the CDN company who could implement the solution, and they lowered the MTU and that solved the problem.
Check more discussion of this question.