Custom HID Gamepad works on Windows, has some inverted buttons in Linux - usb

Question

I've designed a custom gamepad using an STM32. It has four analog axes, and 22 buttons. I have written firmware for it, including the USB Descriptor (that took a few tries).
It works exactly as expected on Windows, but on Linux (Raspbian specifically) the last 5 buttons are inverted ("on" by default, "off" when pressed)
It is critical to my product that it work on both platforms (Mac too, but I don't have one to test right now)
Any ideas why this would be, and what to do about it?
Here's the output of lsusb -v for this device:
Bus 001 Device 003: ID 0483:5750 STMicroelectronics
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 64
idVendor 0x0483 STMicroelectronics
idProduct 0x5750
bcdDevice 2.00
iManufacturer 1 STMicroelectronics
iProduct 2 STM32 Custom Human interface
iSerial 3 74E79C303734
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0029
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xc0
Self Powered
MaxPower 100mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 2
bInterfaceClass 3 Human Interface Device
bInterfaceSubClass 0
bInterfaceProtocol 0
iInterface 0
HID Device Descriptor:
bLength 9
bDescriptorType 33
bcdHID 1.11
bCountryCode 0 Not supported
bNumDescriptors 1
bDescriptorType 34 Report
wDescriptorLength 53
Report Descriptor: (length is 53)
Item(Global): Usage Page, data= [ 0x01 ] 1
Generic Desktop Controls
Item(Local ): Usage, data= [ 0x05 ] 5
Gamepad
Item(Main ): Collection, data= [ 0x01 ] 1
Application
Item(Local ): Usage, data= [ 0x01 ] 1
Pointer
Item(Main ): Collection, data= [ 0x00 ] 0
Physical
Item(Local ): Usage, data= [ 0x30 ] 48
Direction-X
Item(Local ): Usage, data= [ 0x31 ] 49
Direction-Y
Item(Local ): Usage, data= [ 0x32 ] 50
Direction-Z
Item(Local ): Usage, data= [ 0x33 ] 51
Rotate-X
Item(Global): Logical Minimum, data= [ 0x00 ] 0
Item(Global): Logical Maximum, data= [ 0xff 0x0f ] 4095
Item(Global): Report Size, data= [ 0x10 ] 16
Item(Global): Report Count, data= [ 0x04 ] 4
Item(Main ): Input, data= [ 0x02 ] 2
Data Variable Absolute No_Wrap Linear
Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Main ): End Collection, data=none
Item(Global): Usage Page, data= [ 0x09 ] 9
Buttons
Item(Local ): Usage Minimum, data= [ 0x01 ] 1
Button 1 (Primary)
Item(Local ): Usage Maximum, data= [ 0x16 ] 22
(null)
Item(Global): Logical Minimum, data= [ 0x00 ] 0
Item(Global): Logical Maximum, data= [ 0x01 ] 1
Item(Global): Report Size, data= [ 0x01 ] 1
Item(Global): Report Count, data= [ 0x16 ] 22
Item(Main ): Input, data= [ 0x02 ] 2
Data Variable Absolute No_Wrap Linear
Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Global): Report Size, data= [ 0x01 ] 1
Item(Global): Report Count, data= [ 0x02 ] 2
Item(Main ): Input, data= [ 0x03 ] 3
Constant Variable Absolute No_Wrap Linear
Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Main ): End Collection, data=none
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0002 1x 2 bytes
bInterval 10
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x01 EP 1 OUT
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0002 1x 2 bytes
bInterval 10
Device Status: 0x0001
Self Powered
I also just noticed that the stick positions are different (again, correct on Windows, and wrong on Linux.)
I did read something about Linux having a kernel driver that does something to these values? jstest-gtk lets me control some aspects of the axes (scaling, dead zones, inversion) maybe that's a clue?