LPC2378 No USB_EVT_OUT

"However, it doesn't seem possible to read more than 64 bytes at once under Windows XP."

On the device side, the report of greater than wMaxPacketSize is split into packets.
On the host app, however, the report is handled at a time using single ReadFile or WriteFile call. How do you call ReadFile or WriteFile?


"We have seen instances where Windows XP appears to throw away the oldest reports, buffering only the last 64 bytes received...Another alternative is obviously to read from Windows more frequently."

The built-in HID device driver has a cyclic buffer for input reports. It holds up to 32 reports as the default for WinXP. (See HidD_SetNumInputBuffers reference)

Windows starts interrupt-IN transactions to HID devices just after the device is plugged-in and enumerated. Please note, these transactions occur regardless of the host application. Until the host application starts, the reports sent by your device are discarded. When your host application starts and open the HID device using CreateFile, the input-report buffer is also allocated and starts to be filled with the reports from the device. If it takes for a while to issue ReadFile after CreateFile, the input-report buffer may have been already filled with the old reports. If your application don't have any interest to these old reports, flush the buffer using HidD_FlushQueue().

If your device firmware puts the input reports every time when the interrupt occurs at the interrupt-IN endpoint (as the KEIL example does), the input reports are generated at the rate defined in the bInterval field (ms) of the endpoint descriptor (actually, at the nearest number of power of 2, less than or equal to bInterval, ie. 1, 2, 4, 8, 16, 32, ...).

As you have experienced buffer overflow on the input-report buffer, there are three solutions for this problem.
a) Send reports just when your device desires.
b) Read multiple reports on the buffer at a time.
c) Increase the bInterval to reduce the rate of the input report.
d) Read out the buffer more frequently.

c) and d) are obvious, and trivial solution.

To implement a) solution to your device, you have to handle the interrupt-IN endpoint out of the interrupt handler of the endpoint. The interrupt of this endpoint is triggered just when the host reads out the data on the endpoint. That is, this interrupt is initiated by the host side, host event. In most cases, however, you are interested in the event caused on the device side as the trigger of the report. For example, USB mice and keyboards send their reports, just when the devices detect user's action. To send a report just when the device wants, the reports are filled to the endpoint at the main loop task, or the interrupt task other than the endpoint, such as timer interrupt. See this topic for details.

"HID Mouse keeps sending unneeded data!"
http://www.keil.com/forum/docs/thread11531.asp

b) You can read out multiple reports on buffer using single ReadFile call.
Call ReadFile with greater buffer and greater nNumberOfBytesToRead.

nNumberOfBytesToRead = (report size plus one (for report ID)) x n;
n: number of reports you want to read out.

ReadFile returns the number of bytes read actually, in lpNumberOfBytesRead.
Using HidD_SetNumInputBuffers(), you can tune the numbers of reports held in the buffer.


"do you have any comments on the proposal I made (in my previous post) to define a Report Descriptor including all possible report sizes ?"

It's possible, and some devices actually implements this scheme (for example, SiLabs USB debug adapter).
But I don't think it makes so much difference from following scheme,
- Declare single report with the greatest size required (plus one).
- Place the actual number of bytes of the report at the first byte
- Fill variable-length body after the length byte (like Pascal style string)
- Padding the rest.

[References]
HidD_SetNumInputBuffers()
msdn2.microsoft.com/.../ms790946.aspx

HidD_FlushQueue()
msdn2.microsoft.com/.../ms790918.aspx

Tsuneo

"However, it doesn't seem possible to read more than 64 bytes at once under Windows XP."

On the device side, the report of greater than wMaxPacketSize is split into packets.
On the host app, however, the report is handled at a time using single ReadFile or WriteFile call. How do you call ReadFile or WriteFile?


"We have seen instances where Windows XP appears to throw away the oldest reports, buffering only the last 64 bytes received...Another alternative is obviously to read from Windows more frequently."

The built-in HID device driver has a cyclic buffer for input reports. It holds up to 32 reports as the default for WinXP. (See HidD_SetNumInputBuffers reference)

Windows starts interrupt-IN transactions to HID devices just after the device is plugged-in and enumerated. Please note, these transactions occur regardless of the host application. Until the host application starts, the reports sent by your device are discarded. When your host application starts and open the HID device using CreateFile, the input-report buffer is also allocated and starts to be filled with the reports from the device. If it takes for a while to issue ReadFile after CreateFile, the input-report buffer may have been already filled with the old reports. If your application don't have any interest to these old reports, flush the buffer using HidD_FlushQueue().

If your device firmware puts the input reports every time when the interrupt occurs at the interrupt-IN endpoint (as the KEIL example does), the input reports are generated at the rate defined in the bInterval field (ms) of the endpoint descriptor (actually, at the nearest number of power of 2, less than or equal to bInterval, ie. 1, 2, 4, 8, 16, 32, ...).

As you have experienced buffer overflow on the input-report buffer, there are three solutions for this problem.
a) Send reports just when your device desires.
b) Read multiple reports on the buffer at a time.
c) Increase the bInterval to reduce the rate of the input report.
d) Read out the buffer more frequently.

c) and d) are obvious, and trivial solution.

To implement a) solution to your device, you have to handle the interrupt-IN endpoint out of the interrupt handler of the endpoint. The interrupt of this endpoint is triggered just when the host reads out the data on the endpoint. That is, this interrupt is initiated by the host side, host event. In most cases, however, you are interested in the event caused on the device side as the trigger of the report. For example, USB mice and keyboards send their reports, just when the devices detect user's action. To send a report just when the device wants, the reports are filled to the endpoint at the main loop task, or the interrupt task other than the endpoint, such as timer interrupt. See this topic for details.

"HID Mouse keeps sending unneeded data!"
http://www.keil.com/forum/docs/thread11531.asp

b) You can read out multiple reports on buffer using single ReadFile call.
Call ReadFile with greater buffer and greater nNumberOfBytesToRead.

nNumberOfBytesToRead = (report size plus one (for report ID)) x n;
n: number of reports you want to read out.

ReadFile returns the number of bytes read actually, in lpNumberOfBytesRead.
Using HidD_SetNumInputBuffers(), you can tune the numbers of reports held in the buffer.


"do you have any comments on the proposal I made (in my previous post) to define a Report Descriptor including all possible report sizes ?"

It's possible, and some devices actually implements this scheme (for example, SiLabs USB debug adapter).
But I don't think it makes so much difference from following scheme,
- Declare single report with the greatest size required (plus one).
- Place the actual number of bytes of the report at the first byte
- Fill variable-length body after the length byte (like Pascal style string)
- Padding the rest.

[References]
HidD_SetNumInputBuffers()
msdn2.microsoft.com/.../ms790946.aspx

HidD_FlushQueue()
msdn2.microsoft.com/.../ms790918.aspx

Tsuneo

Thank-you again Tuneo for your time and detailed reply - it is very much appreciated.

Let me try to address some of the points you make :-

Our wMaxPacketSize is typically only 8 or 16, the same size as our maximum HID report, so we have no need to split reports into multiple frames.

Our host app calls ReadFile() something like the following :-

ReadFile(handle, buffer, sizeof(buffer), &numBytes, pHIDOverlapped);
Result = WaitForSingleObject(pHIDOverlapped, 100);
if (Result == WAIT_OBJECT_O)
{
  GetOverlappedResult(hande, pHIDOverlapped,, &numBytes, FALSE);
}
else
{
  CancelIo(handle);
}

buffer size is currently (65 * 8).

Our firmware only sends reports when it actually has something new to report (unlike, as you say, the Keil example). However, the nature of those reports is asynchronous and sometimes it has to send multiple reports in a short period of time.

To determine if reports were being lost, we modified the firmware so that, on demand, it sent 20 x 8-byte reports in a short time, each containing an incrementing value. When the host app reads these reports it only ever gets a maximum of 64 bytes (the last 8 frames) with older ones having been thrown away.

Regarding your four options :-

We already do (a) and (b). We could do (c) but this would obviously increase the latency of the device, which is not desirable. (d) is the option I referred to in my last post, but Windows is notoriously unpredictable when it come to trying to do something very quickly very regularly - it's not exactly designed for real-time.

Your suggestion to pad reports to the defined maximum size is exactly what we currently do. However, because of the problem with Windows apparently dropping reports under load, we are interested in being able to send smaller reports where possible.

...breaking news...

Our host app programmer has just reported that your suggestion of increasing the number of buffers using HidD_SetNumInputBuffers() is working. Thank-you very much for this.
Without this call however, we still see a limit of 64 bytes, not 32 reports as per the documentation link that you provided !

Are you aware of any difference (other than the maximum number of buffers) under Windows 2000 ?

Thnaks again for all your assistance.

A quick follow-up...

It was my mistake regarding Windows XP; my colleague was actually using Windows 2000.

A quick check with HidD_GetNumInputBuffers() shows that under Windows 2000, the default is 8 reports while under Windows XP, the default is 32.

The microsoft documentation at msdn2.microsoft.com/.../ms790946.aspx is wrong (or at least inaccurate) as it appears to suggest that the default is 32 on both.

Thanks Tsuneo for all your help.

"Our wMaxPacketSize is typically only 8 or 16, the same size as our maximum HID report, so we have no need to split reports into multiple frames."

I don't recommend you to set wMaxPacketSize to other number than 64 for full-speed interrupt and bulk EPs. Surely, the USB spec allows any number (<= 64) for interrupt EPs, and 8, 16, 32, 64 for bulk ones. However, 64 is the de-fact standard; most of devices apply this number, and the host USB stack may not test fully for other numbers.


"Our host app calls ReadFile() something like the following"

Check the return value of ReadFile, first.
OVERLAPPED ReadFile returns TRUE, when it immediately completes the request. This occurs when the input-report buffer already holds enough number of reports for the request.
When it returns FALSE, confirm that GetLastError returns ERROR_IO_PENDING. Now you can safely apply WaitForSingleObject.


"Are you aware of any difference (other than the maximum number of buffers) under Windows 2000 ?"

I've once seen that the MS KB refers briefly to Win2k bug on reading out multiple reports using ReadFile. It was just a comment on an example file of the MS KB. But I cannot find it again...

Tsuneo

See this MS KB article for the details of OVERLAPPED ReadFile handling.

"Asynchronous Disk I/O Appears as Synchronous on Windows NT, Windows 2000, and Windows XP"
support.microsoft.com/.../en-us

When OVERLAPPED ReadFile completes synchronously and returns TRUE, the event object in the OVERLAPPED structure is not signaled. WaitForSingleObject results timeout in this case, though the readout actually succeeds. I think this is the cause of the data drop.

Tsuneo