Skip to content

Fixed a flaky test in XMLTest.java#1

Open
harshith2000 wants to merge 1 commit into
masterfrom
Fix-flaky-tests
Open

Fixed a flaky test in XMLTest.java#1
harshith2000 wants to merge 1 commit into
masterfrom
Fix-flaky-tests

Conversation

@harshith2000

@harshith2000 harshith2000 commented Nov 13, 2023

Copy link
Copy Markdown
Owner

Issue:
The shouldCreateExplicitEndTagWithEmptyValueWhenConfigured test and the shouldNotCreateExplicitEndTagWithEmptyValueWhenNotConfigured test in XMLTest.java fails due to an assertCheck between two XML Strings.

assertEquals(expectedXmlString, xmlForm);

assertEquals(expectedXmlString, xmlForm);

The toString function of the Object class makes no guarantees as to the iteration order of the attributes in the object. This makes the test outcome non-deterministic and the test fails whenever the toString changes the order of the properties. To fix this, the expected and actual values should be checked in a more deterministic way so that the assertions do not fail.

Fix
Expected and Actual values can be converted into JSONObject and the similar can be used to compare these objects. As this function compares the values inside the JSONObjects without needing order, the test becomes deterministic and ensures that the flakiness from the test is removed.

To reproduce:

I utilized the open-source tool NonDex to detect this assumption by altering the order of returned exception types.

To replicate:

Clone the Repository:

git clone https://github.com/stleary/JSON-java.git

Integrate NonDex:

Add the following snippet to the top of the build.gradle file:

plugins {
    id 'edu.illinois.nondex' version '2.1.1-1'
}

Add to the end of the build.gradle file:

apply plugin: 'edu.illinois.nondex'

Execute Test with Gradle:

./gradlew --info test --tests path.to.the.TestClass.testMethod

Run NonDex:

./gradlew --info nondexTest --tests=path.to.the.TestClass.testMethod --nondexRuns=X

The PR does not introduce a breaking change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant