The Basics
Along with the LG VL600 this modem was the launch device for the Verizon 4G LTE network late last year. Despite being quite large (over twice the size of a normal 3G modem) it’s not a bad device and performs quite well in speed tests. Inside is a Qualcomm MDM9600 chipset providing both CDMA 1xRTT and EVDO on the standard North American 850 MHz Cellular and 1900 MHz PCS bands, and LTE on Verizon’s Upper 700 MHz C-block band. This device cannot roam internationally.
Linux Support
The UML290 exposes four USB interfaces: a standard CDC-ACM AT command port which supports PPP, a QCDM port, a WMC port, and a raw IP network port. Of these, only the AT command and the QCDM ports are really usable in Linux. You can connect to the LTE network using standard ETSI 27.007 GSM-style AT commands like AT+CGDCONT and ATD#99* and such. Connections to the 3G EVDO network can be made with the standard ATD#777 command. Unfortunately, the PPP functionality does not support data connection handoff between the EVDO and LTE networks, so you have to break the connection and reconnect with the appropriate ATD command when necessary. Why is that?
To allow seamless operation between the EVDO and LTE networks Verizon upgraded parts of their core network to eHRPD. HRPD (High Rate Packet Data) is the new name for HDR (High Data Rate) which was the old name for the IS-856 standard developed by Qualcomm ten years ago for high speed 3G packet data. EVDO (Evolution Data Only) is just the marketing name for all that. eHRPD stands for “evolved” or “enhanced” HRPD and essentially drops in pieces of the LTE core network modified to work with older EVDO protocols. Normally your device uses the eHRPD protocol when starting a data session since both the network and the modem support it. But when you use traditional CDMA PPP via ATD#777 the session is between pppd on your computer and the packet data gateway in the network, in contrast to GSM/WCDMA/LTE where the PPP session is only between pppd and the modem itself, not over the air. My theory here is that to maintain backwards compatibility or for some other reason, PPP data sessions using ATD#777 only allow HRPD, and thus handoffs between EVDO and LTE don’t work because the LTE side doesn’t like the older HRPD.
This leads to the problem where you, as the user, have to poke values into the NV_HDRSCP_FORCE_AT_CONFIG_I NVRAM item to manually switch between HRPD and eHRPD just to get connected. Why does this matter? Because the only way to connect to the EVDO network on Linux is with a direct PPP data session using ATD#777. That sucks.
All Hail WMC (wait, what?)
Hardware often makes me want to dress all in black, sit at the end of the bar, drink, and cry. Often Matthew Garrett is right there with me so at least I have company on my trip to black, black oblivion. The hope is that talking to the UML290 on the WMC port and using the modem’s native network interface makes this stupid handoff problem just go away because the modem firmware takes care of the data session protocols and handoffs when you’re not using direct PPP. But that means that we need to reverse engineer both the WMC protocol and the network interface. I’ll drink to that.
It turns out the network interface appears to just be passing raw IP packets over USB. At least that’s what the Windows USB traces tell me unless I’ve had to much Jacky D in which case they just look like Care Bears and rainbows. Qualcomm posted some driver patches for the “smd_rmnet” driver for Android devices that describe a “raw IP” mode for RMNET interfaces that lead me to believe I’m on the right track here. We’ll see.
The WMC bits are the best part though. This Pantech-specific (as far as I can tell) protocol that has been around at least since 2005 since I’ve got an Audiovox PC5740 that uses it and a Pantech PX-500 on Sprint that looks similar yet different. WMC is just another binary protocol; essentially encoding structs on the wire but with a bunch of stupid at the front and some idiot at the end. It’s got a frame start marker of 0xC8, except when there’s more shit at the front. It’s got a frame terminator of 0x7E, except when it doesn’t. It gets HDLC escaped, except when even control characters get escaped instead of just the escape characters. It’s got standard command numbers, except when it doesn’t.
The basic WMC frame starts with 0xC8. The PC5740 and the PX-500 both accept plain WMC requests like this. The UML290 on the other hand uses just about the most convoluted format I can think of. I’d really love to know why. I hope there’s a good reason. Instead the Verizon connection manager sends the WMC packet prefixed with “AT*WMC=”, then 0xC8, and then a bunch of binary data. And not only are the HDLC escape characters escaped, all control characters under 0x20 are escaped too. Even better, the request terminates with a 0x0D instead of the standard 0x7E. So you end up with something looking like this:
41542a574d433dc87d2a87b80d
and when all the framing and shit is removed, it comes down to a single byte: 0x0A. That’s it. Really. Why is this so hard? It’s USB for crying out loud. We’re not on serial links anymore where if somebody picks up the telephone downstairs you get a bunch of garbage in your XMODEM transfer.
It gets better. There’s a CRC-16 at the end, which is pretty standard with these sorts of binary modem protocols. Qualcomm writes the original firmware for all these modems anyway and they all include a Qualcomm DIAG port which speaks a protocol using the standard HDLC framing with CRC-16 (polynomial 0x8408 and seed of 0xFFFF) and a frame terminator of 0x7E. So you’d think they’d re-use those bits. THINK AGAIN. Perhaps because they woke up one day and decided to make life hard for everyone on the planet, the Pantech engineers working on the UML290 decided to use a CRC-16 initial seed of 0xAAFE. What the fuck? Even the PC5740 and the PX-500 use a standard HDLC CRC-16 seed of 0xFFFF like just about everything else on the planet.
But it gets better. The responses from the UML290 don’t bother to include a valid CRC-16; instead it’s just 0x3030. Wow, class work guys. I’m sure there’s good reason for that. Or not. At least the PC5740 and PX-500 get points for valid CRCs.
Which begs the question: why do people still use these serial protocols? Every other piece of USB-connected wireless hardware I’ve seen, from WiFi devices to WiMAX cards, don’t bother with this serial framing shit at all. Even for firmware uploads. They just push packed structs up and down the wire. USB already has a 16-bit CRC check for data packets. Let’s re-invent the wheel for no good reason just because it’s fun.
Why do mobile broadband modems have to be different? Why all the framing and escaping and general eye gouging with shards broken glass? Why duplicate what USB already does? If your modem doesn’t use USB, doesn’t that protocol already have integrity protection and error checking? Cause if it doesn’t you’re already in for a world of hurt.
As an embedded engineer you just have to wake up one morning and say “This is fucking stupid.” But I suppose that’s not something a 6-month product cycle allows. Which is why, as open-source engineers that have to talk to hardware, we tend to drink. And then cry a lot.
Your blog post finally made me understand how all of this fits. You see, the people who write the drivers are hardware engineers. They design chips. They poke hardware directly. When stuff doesn’t work, they don’t update the software, they throw circuits at the problem. Software that is more than a thousand lines is scary to them, so they try to avoid it. And I hope you realize that they code in asm, not Python or such nonsense. Keep that in mind for the 1000 line metric. Also, when they use these chips, what they do is they are real man, they trigger the inputs of these chips manually by nudging electrons at the wires, they don’t use no fancy Verilog.
Now unfortunately, from time to time their boss comes in and tells them that their hardware needs to interface with the real world. So they subcontract someone that does this. As always, they are very suspicious of subcontractors, so it’s best to leave them alone and not interface with them too much. Those subcontractors will then come up with a Windows driver and a small chip that talks to the Windows driver – using shenanigans such as USB, who would use that when you could be soldering the beautiful chips directly onto the CPU? Anyway, who cares, let’s glue those things together and out comes a new device.
Now the subcontractors of course got a specification of features on how things should work – because the sales team sold something – and they have their own favorite way of doing things. As the cheapest subcontractor gets the job – and not the sanest, smartest or someone with prior experience – those favorite things might be different from job to job. So one subcontractor likes talking AT, another one talks PPP and the next one prefers raw IP. So off they go and implement it.
Of course, as all these devices work slightly differently, they all have slightly different features, too. And the sales department knows about those features. Because the boss told them. So they sell them. Now what the sales guys don’t know – or care about – is that you need different interfaces to get those different features. One might only work via subcontractor A’s method and some might require subcontractor B’s. But that’s not what they care about. They sell stuff to make money – not make money for the company, but for them. They’re paid by how much revenue they generate, not by how much bullshit they avoid after all.
Now somewhere in this company, there’s an economist. And he figured out something. He realized that if you glue all those different interfaces together on one device, you have to great advantages. You don’t need to build multiple devices and you can really sell all the features in one device. Then he did some computations in Excel and figured out that indeed, it is making things cheaper. So he told this to the boss who looked at the numbers, saw that it made sense (it was cheaper after all) and made someone do it. Now, you’re story isn’t detailed enough to tell me if that was in-house or another subcontractor. But I’m leaning towards subcontractor. Anyway, what they did is take all those different interfaces, glue them together on one chip and write a windows driver that can talk to any of them, depending on some config settings (probably compile time?). And those get set depending on what features they sold to the guys that want the chip.
And this is the device you see. It can do a lot of things, but none of those things can do everything you want. It can probably also do a lot more things that you haven’t realized yet, because you haven’t encountered a driver that had those config settings enabled, but those will come, I assure you. If just because the boss is still busy hiring new subcontractors for writing interfaces with features that don’t yet exist in any interface because the sales team sold something that the boss told them that he picked up from an engineer who did a joke in a meeting. But you’ll learn about that feature soon enough, when T-Mobile or so releases their new device.
Man, I wish I was making that up. But I’m actually dead serious.