At Q2BSTUDIO, a leading company in technological development and services, we understand that automated testing is a fundamental tool for detecting errors in systems. However, unlike formal verification methodologies, these tests can overlook certain errors. To address this issue, a study was conducted on BaseX's historical error reports to evaluate the effectiveness of the XPress tool in detecting faults.
We analyzed 1597 error reports in BaseX, excluding those already reported by researchers. The focus was on errors caused by XPath expressions, and we evaluated whether XPress could have detected them by considering three key aspects: the implementation of language functions and features, the structure of test patterns, and the comparison of results across different XML processors.
The analysis revealed that, out of 78 errors studied, 20 could have been detected by XPress. Among the limitations found, four main categories of errors that the tool could not identify were detected: lack of implemented functionalities (51 cases), invalid inputs that should have generated errors but did not (6 cases), discrepancies in results between processors (2 cases), and other miscellaneous issues (6 cases). Despite these limitations, the differential testing approach employed by XPress proved to be highly effective in detecting errors in XPath.
The undetected errors were mainly due to the lack of implementation of certain XML and XPath language functions and features, such as specific constructors, map and array functions, as well as control structures like loops and variables. It was concluded that with improvements in these functionalities, up to 38 additional errors could have been detected. Additionally, it was identified that validation and error handling in invalid expressions is an area for improvement in the tool.
At Q2BSTUDIO, we specialize in developing robust and efficient software, relying on agile methodologies and advanced testing tools to ensure the quality of our products. This study reflects the importance of using innovative approaches, such as differential testing, in validating complex systems, ensuring optimal performance in high-impact technological solutions.
