Terminate test on miscompare when hashCheck is true?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Terminate test on miscompare when hashCheck is true?

Brian
Hi-

Is there a flag or parameter to direct COSBench to immediately terminate a test if a file checksum miscompare is found when hashCheck is true?
I modified processBytes() in driver/random/RandomInputStream to inject an invalid checksum for a file during a test. I noticed that the error was logged, and the job continued on, concluding with cleanup and dispose. All artifacts were then gone. Since this could indicate a serious error with my Swift cluster if a true miscompare was found, I would like the job to immediately terminate with no cleanup either. We could then debug the cluster in the exact environment and state to aid in the investigation.

I modified driver/operator/Reader to throw AbortedException if validateChecksum() failed. But that still allowed the other threads to continue working, and with cleanup. I then tried a System.exit(), but that aborted the Driver before the logs could be flushed to disk, and was even worse than my first attempt.

Is there a method or parameter to accomplish this?

Thanks,
Brian
Reply | Threaded
Open this post in threaded view
|

Re: Terminate test on miscompare when hashCheck is true?

ywang19
Administrator
Brian,

So far there is no flag to forcily terminate the process when mismatch occurs, could you fill one issue on github (https://github.com/intel-cloud/cosbench/issues) for tracking?

if you want to have a quick try for terminating at mismatch, you could modify validateChecksum() method in Reader.java to throw Exception at mismatch as following:

if (!calculatedHash.equals(storedHash)) {
                if (storedHash.startsWith(HashUtil.GUARD)) {
                    String err =
                            "Inconsistent Hashes for " + conName + "\\" + objName + ": calculated=" + calculatedHash
                                    + ", stored=" + storedHash;
                    doLogErr(session.getLogger(), err);
                    throw new STorageInterruptedException();
                } else {
                    String warn = "No checksum embedded in " + conName + "\\" + objName;
                    doLogWarn(session.getLogger(), warn);
                }
            }


-yaguang