I had that issue, too, using Linux for a router. It was because it only supports tracking 32k simultaneous (or nearby) connections, and using BitTorrent would kill that limit. To fix it, I just limited BitTorrent's outgoing connection count and limited the upload/download speed, and I haven't had a problem since.