The upstream channels are comparatively low (under 80 MHz) and the downstream channels are comparatively high (over 80 MHz to 800-1000 MHz depending on the system). Splitting them out is accomplished with bidirectional high and low pass filters called "diplexers".
The upstream spectrum is (at the moment) is 5-42 MHz in the US, though most people don't use below 20 MHz and often avoid 26-28 MHz because of interference.
Let's say if the CMTS device does not support this, what are the other options for routing layer2 traffic coming out of the CMTS?
I don't recommend PPPoE. :)
PPPoE not supported on any of the DOCSIS 3.0 certified CMTSs except the Cisco UBR and then it must also be the termination point for the PPPoE session, though it can be part of a L2TP (LAC<--LNS) handoff to another device that can handle the PPPoE termination. I will certainly agree that's its not a good technology for DOCSIS systems.
If I would know more about the device I would say that put a linuxbox after it (on the ISP facing nic) and mark the packets going out with arptables/ebtables then send them out of different nics to different dhcp servers.
Any suggestions are welcome.
You might start by sharing a high level overview of what it is that you're trying to accomplish. If it's simply sandboxing people who haven't paid their bills, there are well-known ways to do that. If it's business services over DOCSIS, there are likewise ways to do that.
Nailed it here.