It's not my design, but from looking from the outside I'd almost definitely "the combination". I don't think the villain here is USB per se (although being limited to USB 1.0 speeds undoubtedly doesn't help), but the fact that he's asking a single-core MCU to multitask pretty hard and it simply doesn't have the resources for it. Here's what he said on the blog:
Note the line about "switching over the DRAM". One thing that may not be particularly obvious about his design until you think about it a bit is that the DRAM isn't really "RAM" so far as the LPC1343 is concerned. It's a random 4-bit wide DRAM device hanging off GPIO pins and thus requires a software loop to "bit-bang" values in and out of it. Without digging into his code I'd be hard-pressed to lay out in detail exactly where the worst problems are, but here's some things to ponder.
1: With the DRAM device being software driven and inadequate framebuffer memory available internally that means that he has to find time multiple times per frame to suck lines of pixels from DRAM into main memory so the USART hardware can clock them out using DMA. (He can't use DMA straight from DRAM.) This takes time, obviously, and it'd be interesting to know if it's something that can be done while the DMA loop is executing. (IE, does the DMA controller cause RAM contention that would prevent the main CPU from filling the buffer that it's reading from while he's reading?)
2: USB has limited granularity with it comes to initiating transactions: You can only start one once per millisecond. (You can transfer a semi- arbitrary amount of data once you've started the transaction, but that's your granularity for starting and stopping one.) A single frame of video at 60hz is only 16.6ms. Remember that even if we're using DMA to push pixels during the 342 active lines of the display you're still going to have be spending some time sending the hsync pulses and setting up the DMA transfers for each line. (Hypothetically maybe the hardware offers a hardware timer able to handle the hsync? Again, I'm too lazy to look at the code.) So somehow you have to find time in all that shuffling data to and from RAM and twiddling hsync pulses to be able to take a transfer block "when you can" and gulp data from USB without breaking the refresh loop. Pure speculation, but I'm guessing what he ended up resorting to is only taking data during the front/back porches. (And being able to do *that* would still basically require being able to either set up timers for vsync and hsync and have them happen automagically, or being able to respond to interrupts to do the needful *during* a USB transfer. It does look like DMA for USB transfers is supported, but perhaps you can't do that at the same time you're using the USART.)
3: If you google for "LPC1343 usb transfer speed" the first few links will point to developers hashing around the various limitations that USB has on that device, including one thread where someone makes a pretty solid case that the best that device can do is around 4mb/s. That's 500Kb/s, which is only about 2/5th of what you'd need for 60 FPS. So even best case that class of device would only give you 24 FPS, USB working non-stop. Divide 24 by 4 and you come up with 6. It's an interesting coincidence that he's getting about 1/6th the theoretical bandwidth the chip is capable of while roughly 1/6th of your average video frame is vsync front/back porch/blanking area. That sort of supports the speculation above that he can't receive data from USB and refresh the screen at the same time, be that because of DMA memory contention, issues with driving his chosen RAM chip, whatever... But, again, it's pure speculation.
I'm sure he's doing about as well as his chosen hardware can handle, I'm totally not criticizing the genius of his design, but... it is an interesting illustration of how a few TTL gates, a hardware shift register, and a few other little bits can make an 8mhz 68000 (Or, heck, a 2Mhz 6502) outperform a 72mhz ARM when it comes to walking and chewing gum at the same time. This is why I'm just generically not that fond of software-generated video solutions. (With the partial exception of the Propeller because it's basically built specifically for the job.)
The eZ80 is cool. The only thing really not to like about it is it only comes in surface mount. (Yeah, yeah, I know, not a problem for these kids today.) I've been thinking about ordering a couple DIP-package Z180s just for kicks; it should be fairly straightforward to substitute for a plain Z-80 in some of those single-board wire-wrap projects I've been looking at and it includes an onboard MMU, DMA controller, and some UARTS that might come in handy and save a chip or two. It's "purely 8 bit" instead of "24 bit" like the ez80, though.