Hi ixcoatl!
Now - i thought a while about which answer would be good... It is hard to say what you have there. This is something like "fishing in the fog" and more guessing than knowing what's up... My amp is running without any problems and with a very high and clear sound...
Not boxy - promised! Even with my cheap ole crappy boxes in my "work room" the amp sounds - hmm - good.
The amp can drive a lot of sources from (afaik) 500mV up to 2-3V. So - it can handle nearly all sources but no phono or mic. inputs. That's not your problem. Your pots are very OK. I'm using 50K ALPS (blue velvets) since one week. The old ones (Radiohm) were not symetrical and they've worked like a bunch of crap. ALPS keeps papa smiling! But - that's also not your problem. The transformers shouldn't be a problem too. If they are matched a bit, they both should have the same output power and they should work parallel at the end. Just a little question - to be sure - how did you connect them?? Each to one rectifier bridge, or the outputs in serial to get a centre tap??
OK - what i think now is that you have bad connections somewhere. At the moment i'ld guess that you didn' connect the RCAs seperately to each channel. Each RCA positive and negative lead should go to its own amp channel (the pot ground included). That will say, that each input channel must be completely connected to one amp channel. Even the "ground" should be connected to its own amp channel ground. Maybe you've got a complete crosstalk. To keep each channel away from the other (imo) is very important for this little piece of magic. (I had a problem with the first layout because the RCA grounds were sticked together. That sounded like 4 boxes in a small bathroom - or whatever like).
The next suspicious one is the input cap and the "C2" that is shown here:
Application Circuit Shematic I don't know which kind of input cap you've used. "C2" must have the value shown on the scheme and good quality (It's a frequency setting component). The input cap is one of the mainparts on this little amp. This cap must be able to pass all details from the incoming signal. A high(!) quality cap is recommended for the input section. (Not a "high end - high price" - simply a good one). It might be the case that you use an - hmm - "cheap" electrolytic type here. That wouldn't be a good idea! E-caps have the characteristic that they often don't pass higher frequencies correctly.
OK - please check all connections again and again until you're sure that it's all right. As said - all ground connections must go to each channel seperately. Please refer to the schematic on the project page. Each single wiring (for one channel) is shown on the scheme and should be followed. For the wiring and the parts connections refer to:
HiFi Amp Scheme.
I now must remark also that this little amp is not a "state of the art" super duper "top of the tops" high end device for 30 bucks... It's just simple to build and yummy yummy good!
Sry - i don't have some other ideas at the moment - but i hope that was a little help now

If not - i'll take the time to make it work right - with your kindly help...