Swapping vertices of triangle if it is scaled inside out (#1879)

Fixed an issue where collide/cast shape against a triangle would return a hit result with mShape2Face in incorrect winding order. This caused an incorrect normal in the enhanced internal edge removal algorithm. This in turn resulted in objects not settling properly on dense triangle grids.

See: https://github.com/godotengine/godot/issues/115009

Docs/ReleaseNotes.md

@@ -21,6 +21,7 @@ For breaking API changes see [this document](https://github.com/jrouwe/JoltPhysi
 * Made it possible to make a class outside the JPH namespace serializable.
 * `VehicleConstraint`s are automatically disabled when the vehicle body is not in the `PhysicsSystem`.
 * Fixed an issue where a character could get stuck. If the character was teleported inside an area surrounded by slopes that are steeper than mMaxSlopeAngle, the code to stop the constraint solver from ping ponging between two planes didn't work properly.
+* Fixed an issue where collide/cast shape against a triangle would return a hit result with mShape2Face in incorrect winding order. This caused an incorrect normal in the enhanced internal edge removal algorithm. This in turn resulted in objects not settling properly on dense triangle grids.
 
 ## v5.5.0
 

Jolt/Physics/Collision/CastConvexVsTriangles.cpp

@@ -92,11 +92,14 @@ void CastConvexVsTriangles::Cast(Vec3Arg inV0, Vec3Arg inV1, Vec3Arg inV2, uint8
 			static_cast<const ConvexShape *>(mShapeCast.mShape)->GetSupportingFace(SubShapeID(), transform_1_to_2.Multiply3x3Transposed(-contact_normal), mShapeCast.mScale, mCenterOfMassTransform2 * transform_1_to_2, result.mShape1Face);
 
 			// Get face of the triangle
-			triangle.GetSupportingFace(contact_normal, result.mShape2Face);
-
-			// Convert to world space
-			for (Vec3 &p : result.mShape2Face)
-				p = mCenterOfMassTransform2 * p;
+			result.mShape2Face.resize(3);
+			result.mShape2Face[0] = mCenterOfMassTransform2 * v0;
+			result.mShape2Face[1] = mCenterOfMassTransform2 * v1;
+			result.mShape2Face[2] = mCenterOfMassTransform2 * v2;
+
+			// When inside out, we need to swap the triangle winding
+			if (mScaleSign < 0.0f)
+				std::swap(result.mShape2Face[1], result.mShape2Face[2]);
 		}
 
 		JPH_IF_TRACK_NARROWPHASE_STATS(TrackNarrowPhaseCollector track;)

Jolt/Physics/Collision/CollideConvexVsTriangles.cpp

@@ -145,6 +145,10 @@ void CollideConvexVsTriangles::Collide(Vec3Arg inV0, Vec3Arg inV1, Vec3Arg inV2,
 		result.mShape2Face[0] = mTransform1 * v0;
 		result.mShape2Face[1] = mTransform1 * v1;
 		result.mShape2Face[2] = mTransform1 * v2;
+
+		// When inside out, we need to swap the triangle winding
+		if (mScaleSign2 < 0.0f)
+			std::swap(result.mShape2Face[1], result.mShape2Face[2]);
 	}
 
 	// Notify the collector

Jolt/Physics/Collision/CollideSphereVsTriangles.cpp

@@ -111,6 +111,10 @@ void CollideSphereVsTriangles::Collide(Vec3Arg inV0, Vec3Arg inV1, Vec3Arg inV2,
 		result.mShape2Face[0] = mTransform2 * (mSphereCenterIn2 + v0);
 		result.mShape2Face[1] = mTransform2 * (mSphereCenterIn2 + v1);
 		result.mShape2Face[2] = mTransform2 * (mSphereCenterIn2 + v2);
+
+		// When inside out, we need to swap the triangle winding
+		if (mScaleSign2 < 0.0f)
+			std::swap(result.mShape2Face[1], result.mShape2Face[2]);
 	}
 
 	// Notify the collector