I think that the actual calibration screen isn't working properly. I should be able to move my controller all the way into the far corners of the coordinates, but they're staying really close to the origin; iirc, it uses a 256-unit coordinate system. It also seems like the 10% center dead zones I have set up in GremlinEx aren't being recognized by the game, like they'll wiggle around the origin unstably even with the joystick at dead center. I guess I could try setting it to joystick-only. I spent like 5.5 hours today just trying to figure out how to get GremlinEx, vJoy, and HIDHide working together, so it'll have to wait until later.
Update: I changed the in-game settings to joystick-only. That did nothing to fix the jittering calibration. I tried changing the dosbox_rebel.conf joystick settings one at a time. The fix that finally worked was changing the joystick setting timed=true. The in-game calibration setting uses a 33-unit coordinate system (I could be wrong about that, however). It'll vary by joystick and drift, but I set mine to 28,28 for the upper left and 26,26 for the lower right. It doesn't necessarily make the controls any better than the game in its normal operating condition, but at least the controls are potentially usable now.