Lets start out with some assertions where the bug is:

1. your code
2. the kernel

Given the low frequency of the error - my guess is your code.
You should try using strace or carefully walking thru your code
for race conditions and invalid assumptions.

Try and simplify what you are doing; try and slow down the code to try
and induce the error more frequently.

If its the kernel, same as above: produce as small a working test case
and either report the bug, or look at the source to see where the race condition is.