feat: set {oversampling=3, rescore=1} as default knn search mode; added syntax for omitting k in knn queries; query limit is implicitly set to k when k is used

* feat: set {oversampling=3, rescore=1} as default knn search mode
* feat: added syntax for omitting k in knn queries; query limit is implicitly set to k when k is used

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sergey Nikolaev <[email protected]>

manual/chinese/Searching/KNN.md

@@ -240,7 +240,7 @@ POST /search
 <!-- example manual_vector -->
 或者，您可以手动插入预计算的向量数据，确保其与创建表时指定的维度匹配。您也可以插入空向量；这意味着文档将从向量搜索结果中排除。
 
-**重要：** 当使用 `hnsw_similarity='cosine'` 时，向量在插入时会自动归一化为单位向量（数学长度/大小为 1.0 的向量）。这种归一化保留了向量的方向，同时标准化了其长度，这是进行高效余弦相似度计算所必需的。这意味着存储的值将与您原始输入的值不同。
+**重要提示：** 当使用 `hnsw_similarity='cosine'` 时，插入向量时会自动归一化为单位向量（数学长度/模为 1.0 的向量）。这种归一化保留了向量的方向，同时标准化其长度，这是高效计算余弦相似度所必需的。这意味着存储的值将与您原始输入的值不同。
 
 <!-- intro -->
 ##### SQL:
@@ -303,9 +303,9 @@ POST /insert
 
 ### KNN 向量搜索
 
-现在，您可以使用 SQL 或 JSON 格式中的 `knn` 子句执行 KNN 搜索。两个接口支持相同的基本参数，确保无论您选择哪种格式，都能获得一致的体验：
+现在，您可以使用 SQL 或 JSON 格式中的 `knn` 子句执行 KNN 搜索。两种接口支持相同的本质参数，无论您选择哪种格式，都能确保一致的体验：
 
-- SQL: `select ... from <table name> where knn ( <field>, <k>, <query vector> [,<options>] )`
+- SQL: `select ... from <table name> where knn ( <field>, <query vector> [,<options>] )`
 - JSON:
   ```
   POST /search
@@ -315,7 +315,6 @@ POST /insert
       {
           "field": "<field>",
           "query": "<text or vector>",
-          "k": <k>,
           "ef": <ef>,
 		  "rescore": <rescore>,
 		  "oversampling": <oversampling>
@@ -323,19 +322,19 @@ POST /insert
   }
   ```
 
-参数如下：
-* `field`：这是包含向量数据的浮点向量属性的名称。
-* `k`：这表示要返回的文档数量，是层次可导航小世界（HNSW）索引的关键参数。它指定单个 HNSW 索引应返回的文档数量。然而，最终结果中包含的实际文档数量可能会有所不同。例如，如果系统处理的是分为磁盘块的实时表，每个块可能返回 `k` 个文档，从而导致总数超过指定的 `k`（因为累积计数将是 `num_chunks * k`）。另一方面，如果在请求 `k` 个文档后，基于特定属性过滤掉了一些文档，则最终文档计数可能少于 `k`。需要注意的是，参数 `k` 不适用于 ramchunks。在 ramchunks 的上下文中，检索过程的操作方式不同，因此，`k` 参数对返回文档数量的影响不适用。
-* `query`：（推荐参数）搜索查询，可以是：
-  - 文本字符串：如果字段配置了自动嵌入，将自动转换为嵌入。如果字段没有自动嵌入，将返回错误。
-  - 向量数组：与 `query_vector` 的工作方式相同。
-* `query_vector`：（遗留参数）作为数字数组的搜索向量。为了向后兼容，仍然支持。
-  **注意：** 使用 `query` 或 `query_vector`，而不是在同一请求中同时使用两者。
-* `ef`：搜索过程中使用的动态列表的可选大小。更高的 `ef` 会导致更准确但更慢的搜索。默认值为 10。
-* `rescore`：启用 KNN 重新评分（默认禁用）。在 SQL 中设置为 `1` 或在 JSON 中设置为 `true` 以启用重新评分。在使用量化向量（可能存在过采样）完成 KNN 搜索后，使用原始（全精度）向量重新计算距离，并重新排序结果以提高排名准确性。
-* `oversampling`：设置一个因子（浮点值），在执行 KNN 搜索时将 `k` 乘以该因子，导致检索到的候选者数量超过所需的量，使用量化向量。默认情况下不应用过采样。如果启用重新评分，这些候选者可以稍后重新评估。过采样也适用于非量化向量。由于它增加了 `k`，这会影响 HNSW 索引的工作方式，因此可能会导致结果准确性的小幅变化。
-
-文档始终按与搜索向量的距离进行排序。您指定的任何其他排序标准将在此主要排序条件之后应用。要检索距离，有一个内置函数 [knn_dist()](../Functions/Other_functions.md#KNN_DIST%28%29)。
+参数包括：
+* `field`: 这是包含向量数据的浮点向量属性的名称。
+* `k`: 已弃用的选项。请改用查询 `limit`。它曾用于指定单个 HNSW 索引应返回的文档数量。然而，最终结果中包含的文档数量可能会有所不同。例如，如果系统处理的是划分为磁盘块的实时表，每个块可能返回 `k` 个文档，导致总数超过指定的 `k`（因为累计数量将是 `num_chunks * k`）。另一方面，如果在请求 `k` 个文档后，根据特定属性过滤掉一些文档，最终文档数量可能少于 `k`。需要注意的是，参数 `k` 不适用于 ramchunks。在 ramchunks 的上下文中，检索过程运作方式不同，因此 `k` 参数对返回文档数量的影响不适用。
+* `query`:（推荐参数）搜索查询，可以是：
+  - 文本字符串：如果字段配置了自动嵌入，则会自动转换为嵌入。如果字段未配置自动嵌入，将返回错误。
+  - 向量数组：与 `query_vector` 的作用相同。
+* `query_vector`:（旧版参数）作为数字数组的搜索向量。仍为向后兼容性提供支持。
+  **注意：** 在同一请求中使用 `query` 或 `query_vector`，不要同时使用两者。
+* `ef`: 搜索期间使用的动态列表的大小。较高的 `ef` 会导致更准确但更慢的搜索。默认值为 10。
+* `rescore`: 启用 KNN 重评分（默认启用）。在 SQL 中设置为 `0` 或在 JSON 中设置为 `false` 以禁用重评分。在使用量化向量完成 KNN 搜索（可能有过采样）后，距离将使用原始（全精度）向量重新计算，结果将重新排序以提高排名准确性。
+* `oversampling`: 设置一个因子（浮点值），在执行 KNN 搜索时乘以 `k`，导致使用量化向量检索的候选对象数量超过所需。默认应用 `oversampling=3.0`。如果启用了重评分，这些候选对象可以稍后重新评估。过采样也适用于非量化向量。由于它增加了 `k`，这会影响 HNSW 索引的工作方式，可能会导致结果准确性的小幅变化。
+
+文档始终按其与搜索向量的距离排序。您指定的任何附加排序条件将在此主要排序条件之后应用。要获取距离，有一个内置函数称为 [knn_dist()](../Functions/Other_functions.md#KNN_DIST%28%29)。
 
 <!-- intro -->
 ##### SQL:
@@ -343,7 +342,7 @@ POST /insert
 <!-- request SQL -->
 
 ```sql
-select id, knn_dist() from test where knn ( image_vector, 5, (0.286569,-0.031816,0.066684,0.032926), { ef=2000, oversampling=3.0, rescore=1 } );
+select id, knn_dist() from test where knn ( image_vector, (0.286569,-0.031816,0.066684,0.032926), { ef=2000, oversampling=3.0, rescore=1 } );
 ```
 <!-- response SQL -->
 
@@ -370,7 +369,6 @@ POST /search
 	{
 		"field": "image_vector",
 		"query": [0.286569,-0.031816,0.066684,0.032926],
-		"k": 5,
 		"ef": 2000, 
 		"rescore": true,
 		"oversampling": 3.0
@@ -421,12 +419,12 @@ POST /search
 
 ### 向量量化
 
-HNSW 索引需要完全加载到内存中才能执行 KNN 搜索，这可能导致显著的内存消耗。为了减少内存使用，可以应用标量量化——一种通过用有限数量的离散值表示每个组件（维度）来压缩高维向量的技术。Manticore 支持 8 位和 1 位量化，这意味着每个向量组件从 32 位浮点压缩到 8 位甚至 1 位，分别减少内存使用 4 倍或 32 倍。这些压缩表示还允许更快的距离计算，因为可以在单个 SIMD 指令中处理更多的向量组件。尽管标量量化引入了一些近似误差，但在搜索准确性和资源效率之间通常是值得的权衡。为了获得更好的准确性，量化可以与重新评分和过采样结合使用：检索到的候选者数量超过请求的数量，并且这些候选者的距离使用原始 32 位浮点向量重新计算。
+HNSW 索引需要完全加载到内存中才能执行 KNN 搜索，这可能导致显著的内存消耗。为了减少内存使用，可以应用标量量化——一种通过将每个组件（维度）表示为有限数量的离散值来压缩高维向量的技术。Manticore 支持 8 位和 1 位量化，这意味着每个向量组件从 32 位浮点数压缩为 8 位甚至 1 位，分别减少内存使用量 4 倍或 32 倍。这些压缩表示还允许更快的距离计算，因为可以在单个 SIMD 指令中处理更多向量组件。尽管标量量化会引入一些近似误差，但通常这是在搜索精度和资源效率之间值得权衡的。为了获得更好的精度，量化可以与重评分和过采样结合使用：检索的候选对象数量超过请求的数量，并使用原始 32 位浮点向量重新计算这些候选对象的距离。
 
 支持的量化类型包括：
-* `8bit`：每个向量组件量化为 8 位。
-* `1bit`：每个向量组件量化为 1 位。使用不对称量化，查询向量量化为 4 位，存储向量量化为 1 位。这种方法提供了比更简单的方法更高的精度，但在性能上有一些权衡。
-* `1bitsimple`：每个向量组件量化为 1 位。此方法比 `1bit` 更快，但通常准确性较低。
+* `8bit`: 每个向量组件量化为 8 位。
+* `1bit`: 每个向量组件量化为 1 位。使用非对称量化，查询向量量化为 4 位，存储向量量化为 1 位。这种方法比简单方法提供更高的精度，但有一些性能权衡。
+* `1bitsimple`: 每个向量组件量化为 1 位。此方法比 `1bit` 快，但通常精度较低。
 
 <!-- intro -->
 ##### SQL:
@@ -447,9 +445,9 @@ Query OK, 0 rows affected (0.01 sec)
 
 ### 通过 ID 查找相似文档
 
-> 注意：通过 ID 查找相似文档需要 [Manticore Buddy](../Installation/Manticore_Buddy.md)。如果它不起作用，请确保 Buddy 已安装。
+> 注意：通过 ID 查找相似文档需要 [Manticore Buddy](../Installation/Manticore_Buddy.md)。如果不起作用，请确保已安装 Buddy。
 
-根据特定文档的唯一 ID 查找相似文档是一项常见任务。例如，当用户查看特定项目时，Manticore Search 可以有效地识别并显示与其在向量空间中最相似的项目列表。以下是您可以执行此操作的方法：
+根据特定文档的唯一 ID 查找相似文档是一项常见任务。例如，当用户查看某个特定项目时，Manticore Search 可以高效地识别并显示在向量空间中与该项目最相似的项目列表。以下是实现方法：
 
 - SQL: `select ... from <table name> where knn ( <field>, <k>, <document id> )`
 - JSON:

manual/english/Searching/KNN.md

@@ -305,7 +305,7 @@ POST /insert
 
 Now, you can perform a KNN search using the `knn` clause in either SQL or JSON format. Both interfaces support the same essential parameters, ensuring a consistent experience regardless of the format you choose:
 
-- SQL: `select ... from <table name> where knn ( <field>, <k>, <query vector> [,<options>] )`
+- SQL: `select ... from <table name> where knn ( <field>, <query vector> [,<options>] )`
 - JSON:
   ```
   POST /search
@@ -315,7 +315,6 @@ Now, you can perform a KNN search using the `knn` clause in either SQL or JSON f
       {
           "field": "<field>",
           "query": "<text or vector>",
-          "k": <k>,
           "ef": <ef>,
 		  "rescore": <rescore>,
 		  "oversampling": <oversampling>
@@ -325,15 +324,15 @@ Now, you can perform a KNN search using the `knn` clause in either SQL or JSON f
 
 The parameters are:
 * `field`: This is the name of the float vector attribute containing vector data.
-* `k`: This represents the number of documents to return and is a key parameter for Hierarchical Navigable Small World (HNSW) indexes. It specifies the quantity of documents that a single HNSW index should return. However, the actual number of documents included in the final results may vary. For instance, if the system is dealing with real-time tables divided into disk chunks, each chunk could return `k` documents, leading to a total that exceeds the specified `k` (as the cumulative count would be `num_chunks * k`). On the other hand, the final document count might be less than `k` if, after requesting `k` documents, some are filtered out based on specific attributes. It's important to note that the parameter `k` does not apply to ramchunks. In the context of ramchunks, the retrieval process operates differently, and thus, the `k` parameter's effect on the number of documents returned is not applicable.
+* `k`: Deprecated option. Use query `limit` instead. It was used to specify the quantity of documents that a single HNSW index should return. However, the actual number of documents included in the final results may vary. For instance, if the system is dealing with real-time tables divided into disk chunks, each chunk could return `k` documents, leading to a total that exceeds the specified `k` (as the cumulative count would be `num_chunks * k`). On the other hand, the final document count might be less than `k` if, after requesting `k` documents, some are filtered out based on specific attributes. It's important to note that the parameter `k` does not apply to ramchunks. In the context of ramchunks, the retrieval process operates differently, and thus, the `k` parameter's effect on the number of documents returned is not applicable.
 * `query`: (Recommended parameter) The search query, which can be either:
   - Text string: Automatically converted to embeddings if the field has auto-embeddings configured. Will return an error if the field doesn't have auto-embeddings.
   - Vector array: Works the same as `query_vector`.
 * `query_vector`: (Legacy parameter) The search vector as an array of numbers. Still supported for backward compatibility.
   **Note:** Use either `query` or `query_vector`, not both in the same request.
 * `ef`: optional size of the dynamic list used during the search. A higher `ef` leads to more accurate but slower search. The default is 10.
-* `rescore`: Enables KNN rescoring (disabled by default). Set to `1` in SQL or `true` in JSON to enable rescoring. After the KNN search is completed using quantized vectors (with possible oversampling), distances are recalculated with the original (full-precision) vectors and results are re-sorted to improve ranking accuracy.
-* `oversampling`: Sets a factor (float value) by which `k` is multiplied when executing the KNN search, causing more candidates to be retrieved than needed using quantized vectors. No oversampling is applied by default. These candidates can be re-evaluated later if rescoring is enabled. Oversampling also works with non-quantized vectors. Since it increases `k`, which affects how the HNSW index works, it may cause a small change in result accuracy.
+* `rescore`: Enables KNN rescoring (enabled by default). Set to `0` in SQL or `false` in JSON to disable rescoring. After the KNN search is completed using quantized vectors (with possible oversampling), distances are recalculated with the original (full-precision) vectors and results are re-sorted to improve ranking accuracy.
+* `oversampling`: Sets a factor (float value) by which `k` is multiplied when executing the KNN search, causing more candidates to be retrieved than needed using quantized vectors. `oversampling=3.0` is applied by default. These candidates can be re-evaluated later if rescoring is enabled. Oversampling also works with non-quantized vectors. Since it increases `k`, which affects how the HNSW index works, it may cause a small change in result accuracy.
 
 Documents are always sorted by their distance to the search vector. Any additional sorting criteria you specify will be applied after this primary sort condition. For retrieving the distance, there is a built-in function called [knn_dist()](../Functions/Other_functions.md#KNN_DIST%28%29).
 
@@ -343,7 +342,7 @@ Documents are always sorted by their distance to the search vector. Any addition
 <!-- request SQL -->
 
 ```sql
-select id, knn_dist() from test where knn ( image_vector, 5, (0.286569,-0.031816,0.066684,0.032926), { ef=2000, oversampling=3.0, rescore=1 } );
+select id, knn_dist() from test where knn ( image_vector, (0.286569,-0.031816,0.066684,0.032926), { ef=2000, oversampling=3.0, rescore=1 } );
 ```
 <!-- response SQL -->
 
@@ -370,7 +369,6 @@ POST /search
 	{
 		"field": "image_vector",
 		"query": [0.286569,-0.031816,0.066684,0.032926],
-		"k": 5,
 		"ef": 2000, 
 		"rescore": true,
 		"oversampling": 3.0

src/knnmisc.cpp

@@ -77,6 +77,15 @@ bool IsKnnDist ( const CSphString & sExpr )
 	return sExpr==GetKnnDistAttrName() || sExpr=="knn_dist()";
 }
 
+
+void SetupKNNLimit ( CSphQuery & tQuery )
+{
+	auto & tKNN = tQuery.m_tKnnSettings;
+
+	if ( tKNN.m_iK < 0 )
+		tKNN.m_iK = tQuery.m_iLimit;
+}
+
 ///////////////////////////////////////////////////////////////////////////////
 
 void NormalizeVec ( VecTraits_T<float> & dData )
@@ -659,8 +668,7 @@ std::pair<RowidIterator_i *, bool> CreateKNNIterator ( knn::KNN_i * pKNN, const
 		NormalizeVec(dPoint);
 
 	std::string sErrorSTL;
-	int64_t iRequested = tKNN.m_iK * tKNN.m_fOversampling;
-	knn::Iterator_i * pIterator = pKNN->CreateIterator ( pKNNAttr->m_sName.cstr(), { dPoint.Begin(), (size_t)dPoint.GetLength() }, iRequested, tKNN.m_iEf, sErrorSTL );
+	knn::Iterator_i * pIterator = pKNN->CreateIterator ( pKNNAttr->m_sName.cstr(), { dPoint.Begin(), (size_t)dPoint.GetLength() }, tKNN.GetRequestedDocs(), tKNN.m_iEf, sErrorSTL );
 	if ( !pIterator )
 	{
 		sError = sErrorSTL.c_str();
@@ -687,7 +695,7 @@ RowIteratorsWithEstimates_t	CreateKNNIterators ( knn::KNN_i * pKNN, const CSphQu
 	if ( !tRes.first )
 		return dIterators;
 
-	dIterators.Add ( { tRes.first, int64_t ( tQuery.m_tKnnSettings.m_iK*tQuery.m_tKnnSettings.m_fOversampling ) } );
+	dIterators.Add ( { tRes.first, tQuery.m_tKnnSettings.GetRequestedDocs() } );
 	return dIterators;
 }
 

src/knnmisc.h

@@ -42,6 +42,7 @@ private:
 bool							IsKnnDist ( const CSphString & sExpr );
 const char *					GetKnnDistAttrName();
 const char *					GetKnnDistRescoreAttrName();
+void							SetupKNNLimit ( CSphQuery & tQuery );
 
 ISphExpr *						CreateExpr_KNNDist ( const CSphVector<float> & dAnchor, const CSphColumnInfo & tAttr );
 ISphExpr *						CreateExpr_KNNDistRescore ( const CSphVector<float> & dAnchor, const CSphColumnInfo & tAttr );

src/queuecreator.cpp

@@ -2333,8 +2333,9 @@ int QueueCreator_c::ReduceOrIncreaseMaxMatches() const
 	const auto & tKNN = m_tQuery.m_tKnnSettings;
 	if ( !tKNN.m_sAttr.IsEmpty() && tKNN.m_fOversampling > 1.0f )
 	{
-		int64_t iRequested = tKNN.m_iK * tKNN.m_fOversampling;
-		return Max ( Max ( m_tSettings.m_iMaxMatches, iRequested ), 1 );
+		int64_t iRequested = tKNN.GetRequestedDocs();
+		if ( !tKNN.m_sAttr.IsEmpty() && iRequested > tKNN.m_iK )
+			return Max ( Max ( m_tSettings.m_iMaxMatches, iRequested ), 1 );
 	}
 
 	if ( m_tQuery.m_bExplicitMaxMatches || m_tQuery.m_bHasOuter || !m_tSettings.m_bComputeItems )

src/searchdsql.cpp

@@ -381,6 +381,7 @@ public:
 	bool			SetMatch ( const SqlNode_t & tValue );
 	bool			AddMatch ( const SqlNode_t & tValue, const SqlNode_t & tIndex );
 	bool			SetKNN ( const SqlNode_t & tAttr, const SqlNode_t & tK, const SqlNode_t & tValues, const CSphVector<CSphNamedVariant> * pOpts, bool bAutoEmb );
+	bool			SetKNN ( const SqlNode_t & tAttr, const SqlNode_t & tValues, const CSphVector<CSphNamedVariant> * pOpts, bool bAutoEmb );
 	void			AddConst ( int iList, const SqlNode_t& tValue );
 	void			SetLocalStatement ( const SqlNode_t & tName );
 	bool			AddFloatRangeFilter ( const SqlNode_t & tAttr, float fMin, float fMax, bool bHasEqual, bool bExclude=false );
@@ -458,6 +459,7 @@ private:
 	void			AutoAlias ( CSphQueryItem & tItem, SqlNode_t * pStart, SqlNode_t * pEnd );
 	bool			CheckOption ( Option_e eOption ) const override;
 	SqlStmt_e		GetSecondaryStmt () const;
+	bool			SetKNN ( const SqlNode_t & tAttr, int iK, const SqlNode_t & tValues, const CSphVector<CSphNamedVariant> * pOpts, bool bAutoEmb );
 	void			NegateFilterTree ( int iNode );
 	void			FixupLastBacktick ( SqlNode_t * pExpr ) const noexcept;
 };
@@ -1443,6 +1445,14 @@ static bool ParseKNNOption ( const CSphNamedVariant & tOpt, KnnSearchSettings_t
 		tKNN.m_bRescore = !!tOpt.m_iValue;
 		return true;
 	}
+	else if ( sName=="k" )
+	{
+		if ( tOpt.m_eType!=VariantType_e::BIGINT )
+			return false;
+
+		tKNN.m_iK = tOpt.m_iValue;
+		return true;
+	}
 
 	return false;
 }
@@ -1450,15 +1460,31 @@ static bool ParseKNNOption ( const CSphNamedVariant & tOpt, KnnSearchSettings_t
 
 bool SqlParser_c::SetKNN ( const SqlNode_t & tAttr, const SqlNode_t & tK, const SqlNode_t & tValues, const CSphVector<CSphNamedVariant> * pOpts, bool bAutoEmb )
 {
-	auto & tKNN = m_pQuery->m_tKnnSettings;
-
-	ToString ( tKNN.m_sAttr, tAttr );
-	tKNN.m_iK = tK.GetValueInt();
-	if ( tKNN.m_iK <= 0 )
+	int iK = tK.GetValueInt();
+	if ( iK <= 0 )
 	{
 		yyerror ( this, "k parameter must be positive" );
 		return false;
 	}
+
+	return SetKNN ( tAttr, iK, tValues, pOpts, bAutoEmb );
+}
+
+
+bool SqlParser_c::SetKNN ( const SqlNode_t & tAttr, const SqlNode_t & tValues, const CSphVector<CSphNamedVariant> * pOpts, bool bAutoEmb )
+{
+	return SetKNN ( tAttr, -1, tValues, pOpts, bAutoEmb );
+}
+
+
+bool SqlParser_c::SetKNN ( const SqlNode_t & tAttr, int iK, const SqlNode_t & tValues, const CSphVector<CSphNamedVariant> * pOpts, bool bAutoEmb )
+{
+	auto & tKNN = m_pQuery->m_tKnnSettings;
+
+	tKNN.m_iK = iK;
+
+	ToString ( tKNN.m_sAttr, tAttr );
+
 	if ( pOpts )
 		for ( auto & i : *pOpts )
 			if ( !ParseKNNOption ( i, tKNN ) )
@@ -2242,7 +2268,6 @@ static bool SetupFacetDistinct ( CSphVector<SqlStmt_t> & dStmt, CSphString & sEr
 }
 
 
-
 bool sphParseSqlQuery ( Str_t sQuery, CSphVector<SqlStmt_t> & dStmt, CSphString & sError, ESphCollation eCollation )
 {
 	if ( !IsFilled ( sQuery ) )
@@ -2308,6 +2333,8 @@ bool sphParseSqlQuery ( Str_t sQuery, CSphVector<SqlStmt_t> & dStmt, CSphString
 				tQuery.m_iSQLSelectEnd - tQuery.m_iSQLSelectStart );
 		}
 
+		SetupKNNLimit(tQuery);
+
 		// validate tablefuncs
 		// tablefuncs are searchd-level builtins rather than common expression-level functions
 		// so validation happens here, expression parser does not know tablefuncs (ignorance is bliss)

src/sphinx.cpp

@@ -1608,6 +1608,13 @@ uint64_t FilterTreeItem_t::GetHash() const
 
 //////////////////////////////////////////////////////////////////////////
 
+int64_t KnnSearchSettings_t::GetRequestedDocs() const
+{
+	assert ( m_iK>=0 );
+	return m_bRescore ? int64_t ( m_fOversampling * m_iK ) : m_iK;
+}
+
+
 struct SelectBounds_t
 {
 	int		m_iStart;
@@ -8085,7 +8092,7 @@ bool CSphIndex_VLN::ChooseIterators ( CSphVector<SecondaryIndexInfo_t> & dSIInfo
 		SelectIteratorCtx_t tSelectIteratorCtx ( tQuery, dFilters, m_tSchema, tMaxSorterSchema, m_pHistograms, m_pColumnar.get(), m_tSI, iCutoff, m_iDocinfo, 1 );
 		tSelectIteratorCtx.m_bFromIterator = true;
 
-		int iRequestedKNNDocs = Min ( int64_t(tQuery.m_tKnnSettings.m_iK * tQuery.m_tKnnSettings.m_fOversampling), m_iDocinfo );
+		int iRequestedKNNDocs = Min ( tQuery.m_tKnnSettings.GetRequestedDocs(), m_iDocinfo );
 		tSelectIteratorCtx.m_fDocsLeft = float(iRequestedKNNDocs)/m_iDocinfo;
 		dSIInfo = SelectIterators ( tSelectIteratorCtx, fBestCost, dWarnings );
 	}

src/sphinx.h

@@ -512,12 +512,14 @@ struct ScrollSettings_t
 struct KnnSearchSettings_t
 {
 	CSphString		m_sAttr;				///< which attr to use for KNN search (enables KNN if not empty)
-	int				m_iK = 0;				///< KNN K
+	int				m_iK = 0;				///< KNN K (-1 means auto)
 	int				m_iEf = 0;				///< KNN ef
-	bool			m_bRescore = false;		///< KNN rescoring
-	float			m_fOversampling = 1.0f;	///< KNN oversampling
+	bool			m_bRescore = true;		///< KNN rescoring
+	float			m_fOversampling = 3.0f;	///< KNN oversampling
 	CSphVector<float> m_dVec;				///< KNN anchor vector
 	std::optional<CSphString> m_sEmbStr;	///< string to generate embeddings from
+
+	int64_t			GetRequestedDocs() const;
 };
 
 /// search query. Pure struct, no member functions

src/sphinxjsonquery.cpp

@@ -1243,12 +1243,23 @@ static bool ParseKNNQuery ( const JsonObj_c & tJson, CSphQuery & tQuery, CSphStr
 
 	auto & tKNN = tQuery.m_tKnnSettings;
 	if ( !tJson.FetchStrItem ( tKNN.m_sAttr, "field", sError ) )	return false;
-	if ( !tJson.FetchIntItem ( tKNN.m_iK, "k", sError ) )			return false;
-	if ( tKNN.m_iK <= 0 )
-	{
-		sError = "k parameter must be positive";
+
+	JsonObj_c tK = tJson.GetIntItem ( "k", sError, true );
+	if ( !sError.IsEmpty() )
 		return false;
+
+	if ( tK )
+	{
+		tKNN.m_iK = (int)tK.IntVal();
+		if ( tKNN.m_iK <= 0 )
+		{
+			sError = "k parameter must be positive";
+			return false;
+		}
 	}
+	else
+		tKNN.m_iK = -1;
+
 	if ( !tJson.FetchIntItem ( tKNN.m_iEf, "ef", sError, true ) )	return false;
 	if ( tKNN.m_iEf < 0 )
 	{
@@ -1551,6 +1562,8 @@ bool sphParseJsonQuery ( const JsonObj_c & tRoot, ParsedJsonQuery_t & tPJQuery )
 	if ( !SetupScroll ( tQuery, sError ) )
 		return false;
 
+	SetupKNNLimit(tQuery);
+
 	return true;
 }
 

src/sphinxql.y

@@ -707,21 +707,41 @@ knn_item:
 			if ( !pParser->SetKNN ( $3, $5, $8, nullptr, false ) )
 				YYERROR;
 		}
+	| TOK_KNN '(' ident ',' '(' const_list ')' ')'
+		{
+			if ( !pParser->SetKNN ( $3, $6, nullptr, false ) )
+				YYERROR;
+		}
 	| TOK_KNN '(' ident ',' const_int ',' TOK_QUOTED_STRING ')'
 		{
 			if ( !pParser->SetKNN ( $3, $5, $7, nullptr, true ) )
 				YYERROR;
 		}
+	| TOK_KNN '(' ident ',' TOK_QUOTED_STRING ')'
+		{
+			if ( !pParser->SetKNN ( $3, $5, nullptr, true ) )
+				YYERROR;
+		}
 	| TOK_KNN '(' ident ',' const_int ',' '(' const_list ')' ',' '{' named_const_list '}' ')'
 		{
 			if ( !pParser->SetKNN ( $3, $5, $8, &( pParser->GetNamedVec ( $12.GetValueInt() ) ), false ) )
 				YYERROR;
 		}
+	| TOK_KNN '(' ident ',' '(' const_list ')' ',' '{' named_const_list '}' ')'
+		{
+			if ( !pParser->SetKNN ( $3, $6, &( pParser->GetNamedVec ( $10.GetValueInt() ) ), false ) )
+				YYERROR;
+		}
 	| TOK_KNN '(' ident ',' const_int ',' TOK_QUOTED_STRING ',' '{' named_const_list '}' ')'
 		{
 			if ( !pParser->SetKNN ( $3, $5, $7, &( pParser->GetNamedVec ( $10.GetValueInt() ) ), true ) )
 				YYERROR;
 		}
+	| TOK_KNN '(' ident ',' TOK_QUOTED_STRING ',' '{' named_const_list '}' ')'
+		{
+			if ( !pParser->SetKNN ( $3, $5, &( pParser->GetNamedVec ( $8.GetValueInt() ) ), true ) )
+				YYERROR;
+		}
 	;
 
 opt_join_clause:

test/clt-tests/core/test-alter-rebuild-knn.rec

@@ -1,6 +1,6 @@
 ––– block: ../base/start-searchd-with-buddy –––
 ––– input –––
-mysql -P9306 -v -h0 -e "create table t(f text); insert into t values(1, 'abc'); select * from t;"
+mysql -P9306 -v -h0 -E -e "create table t(f text); insert into t values(1, 'abc'); select * from t;"
 ––– output –––
 --------------
 create table t(f text)
@@ -11,13 +11,11 @@ insert into t values(1, 'abc')
 --------------
 select * from t
 --------------
-+------+------+
-| id   | f    |
-+------+------+
-|    1 | abc  |
-+------+------+
+*************************** 1. row ***************************
+id: 1
+ f: abc
 ––– input –––
-mysql -P9306 -v -h0 -e "alter table t add column v1 FLOAT_VECTOR knn_type='hnsw' knn_dims='3' hnsw_similarity='COSINE'; insert into t(id,v1) values(2, (0.2,0.2,0.2)); select * from t;"
+mysql -P9306 -v -h0 -E -e "alter table t add column v1 FLOAT_VECTOR knn_type='hnsw' knn_dims='3' hnsw_similarity='COSINE'; insert into t(id,v1) values(2, (0.2,0.2,0.2)); select * from t;"
 ––– output –––
 --------------
 alter table t add column v1 FLOAT_VECTOR knn_type='hnsw' knn_dims='3' hnsw_similarity='COSINE'
@@ -28,14 +26,16 @@ insert into t(id,v1) values(2, (0.2,0.2,0.2))
 --------------
 select * from t
 --------------
-+------+------+----------------------------------+
-| id   | f    | v1                               |
-+------+------+----------------------------------+
-|    2 |      | 0.57735032,0.57735032,0.57735032 |
-|    1 | abc  | 0.000000,0.000000,0.000000       |
-+------+------+----------------------------------+
+*************************** 1. row ***************************
+id: 2
+ f: 
+v1: 0.57735032,0.57735032,0.57735032
+*************************** 2. row ***************************
+id: 1
+ f: abc
+v1: 0.000000,0.000000,0.000000
 ––– input –––
-mysql -P9306 -v -h0 -e "flush ramchunk t; select * from t;"
+mysql -P9306 -v -h0 -E -e "flush ramchunk t; select * from t;"
 ––– output –––
 --------------
 flush ramchunk t
@@ -43,14 +43,16 @@ flush ramchunk t
 --------------
 select * from t
 --------------
-+------+------+----------------------------------+
-| id   | f    | v1                               |
-+------+------+----------------------------------+
-|    1 | abc  | 0.000000,0.000000,0.000000       |
-|    2 |      | 0.57735032,0.57735032,0.57735032 |
-+------+------+----------------------------------+
+*************************** 1. row ***************************
+id: 1
+ f: abc
+v1: 0.000000,0.000000,0.000000
+*************************** 2. row ***************************
+id: 2
+ f: 
+v1: 0.57735032,0.57735032,0.57735032
 ––– input –––
-mysql -P9306 -v -h0 -e "alter table t add column v2 FLOAT_VECTOR knn_type='hnsw' knn_dims='3' hnsw_similarity='L2'; insert into t(id,v2) values(3,(1,1,1)); flush ramchunk t; select * from t"
+mysql -P9306 -v -h0 -E -e "alter table t add column v2 FLOAT_VECTOR knn_type='hnsw' knn_dims='3' hnsw_similarity='L2'; insert into t(id,v2) values(3,(1,1,1)); flush ramchunk t; select * from t"
 ––– output –––
 --------------
 alter table t add column v2 FLOAT_VECTOR knn_type='hnsw' knn_dims='3' hnsw_similarity='L2'
@@ -64,13 +66,21 @@ flush ramchunk t
 --------------
 select * from t
 --------------
-+------+------+----------------------------------+----------------------------+
-| id   | f    | v1                               | v2                         |
-+------+------+----------------------------------+----------------------------+
-|    1 | abc  | 0.000000,0.000000,0.000000       | 0.000000,0.000000,0.000000 |
-|    3 |      |                                  | 1.000000,1.000000,1.000000 |
-|    2 |      | 0.57735032,0.57735032,0.57735032 | 0.000000,0.000000,0.000000 |
-+------+------+----------------------------------+----------------------------+
+*************************** 1. row ***************************
+id: 1
+ f: abc
+v1: 0.000000,0.000000,0.000000
+v2: 0.000000,0.000000,0.000000
+*************************** 2. row ***************************
+id: 3
+ f: 
+v1: 
+v2: 1.000000,1.000000,1.000000
+*************************** 3. row ***************************
+id: 2
+ f: 
+v1: 0.57735032,0.57735032,0.57735032
+v2: 0.000000,0.000000,0.000000
 ––– input –––
 stdbuf -oL searchd --stopwait > /dev/null
 ––– output –––
@@ -79,25 +89,39 @@ rm -fr /var/lib/manticore/t/*.spknn
 ––– output –––
 ––– block: ../base/start-searchd-with-buddy –––
 ––– input –––
-mysql -P9306 -h0 -v -e "select * from t where knn(v2, 10, (0.5,0.5,0.5))"
+mysql -P9306 -h0 -v -E -e "select * from t where knn(v2, 10, (0.5,0.5,0.5))"
 ––– output –––
 --------------
 select * from t where knn(v2, 10, (0.5,0.5,0.5))
 --------------
 ERROR 1064 (42000) at line 1: table t: KNN index not loaded
 ––– input –––
-mysql -P9306 -h0 -v -e "alter table t rebuild knn; select * from t where knn(v2, 10, (0.5,0.5,0.5))"
+mysql -P9306 -h0 -v -E -e "alter table t rebuild knn; select *, knn_dist() dist from t where knn(v2, 10, (0.5,0.5,0.5)) order by dist asc, id asc"
 ––– output –––
 --------------
 alter table t rebuild knn
 --------------
 --------------
-select * from t where knn(v2, 10, (0.5,0.5,0.5))
---------------
-+------+------+----------------------------------+----------------------------+-----------+
-| id   | f    | v1                               | v2                         | @knn_dist |
-+------+------+----------------------------------+----------------------------+-----------+
-|    1 | abc  | 0.000000,0.000000,0.000000       | 0.000000,0.000000,0.000000 |  0.750000 |
-|    2 |      | 0.57735032,0.57735032,0.57735032 | 0.000000,0.000000,0.000000 |  0.750000 |
-|    3 |      |                                  | 1.000000,1.000000,1.000000 |  0.750000 |
-+------+------+----------------------------------+----------------------------+-----------+
+select *, knn_dist() dist from t where knn(v2, 10, (0.5,0.5,0.5)) order by dist asc, id asc
+--------------
+*************************** 1. row ***************************
+       id: 3
+        f: 
+       v1: 
+       v2: 1.000000,1.000000,1.000000
+@knn_dist: 0.750000
+     dist: 0.750000
+*************************** 2. row ***************************
+       id: 2
+        f: 
+       v1: 0.57735032,0.57735032,0.57735032
+       v2: 0.000000,0.000000,0.000000
+@knn_dist: 0.750000
+     dist: 0.750000
+*************************** 3. row ***************************
+       id: 1
+        f: abc
+       v1: 0.000000,0.000000,0.000000
+       v2: 0.000000,0.000000,0.000000
+@knn_dist: 0.750000
+     dist: 0.750000

test/clt-tests/mcl/auto-embeddings-backup-restore.rec

@@ -39,19 +39,28 @@ mysql -h0 -P9306 -e "SELECT id FROM test_backup WHERE KNN(vec, 2, 'artificial in
 +------+
 |    1 |
 |    3 |
+|    2 |
 +------+
 ––– comment –––
 Verify KNN returns semantically closest documents (distances confirm ranking)
 ––– input –––
-mysql -h0 -P9306 -e "SELECT id, title, content, KNN_DIST() as distance FROM test_backup WHERE KNN(vec, 3, 'artificial intelligence') ORDER BY distance"
-––– output –––
-+------+------------------+------------------+------------+
-| id   | title            | content          | distance   |
-+------+------------------+------------------+------------+
-|    1 | machine learning | neural networks  | 0.851733#!/[0-9]{0,3}/!# |
-|    3 | computer vision  | image processing | 1.130702#!/[0-9]{0,3}/!# |
-|    2 | deep learning    | transformers     | 1.311729#!/[0-9]{0,3}/!# |
-+------+------------------+------------------+------------+
+mysql -h0 -P9306 -E -e "SELECT id, title, content, KNN_DIST() as distance FROM test_backup WHERE KNN(vec, 3, 'artificial intelligence') ORDER BY distance"
+––– output –––
+*************************** 1. row ***************************
+      id: 1
+   title: machine learning
+ content: neural networks
+distance: 0.851733#!/[0-9]{0,5}/!#
+*************************** 2. row ***************************
+      id: 3
+   title: computer vision
+ content: image processing
+distance: 1.13070#!/[0-9]{0,5}/!#
+*************************** 3. row ***************************
+      id: 2
+   title: deep learning
+ content: transformers
+distance: 1.3117#!/[0-9]{0,5}/!#
 ––– comment –––
 Check manticore-backup is available
 ––– input –––
@@ -198,6 +207,7 @@ mysql -h0 -P9306 -e "SELECT id FROM test_backup WHERE KNN(vec, 2, 'artificial in
 +------+
 |    1 |
 |    3 |
+|    2 |
 +------+
 ––– comment –––
 Test ALTER TABLE after restore
@@ -260,7 +270,7 @@ mysql -h0 -P9306 -e "FLUSH RAMCHUNK test_copy; OPTIMIZE TABLE test_copy OPTION s
 ––– output –––
 0
 ––– comment –––
-Verify k parameter works for disk chunks (returns top-2 as expected)
+Verify k parameter works for disk chunks (oversampling returns more than k)
 ––– input –––
 mysql -h0 -P9306 -e "SELECT id FROM test_copy WHERE KNN(vec, 2, 'artificial intelligence')"
 ––– output –––
@@ -269,4 +279,6 @@ mysql -h0 -P9306 -e "SELECT id FROM test_copy WHERE KNN(vec, 2, 'artificial inte
 +------+
 |    1 |
 |    3 |
+|    2 |
+|    4 |
 +------+

test/clt-tests/mcl/auto-embeddings-dml-test.rec

@@ -93,6 +93,7 @@ mysql -h0 -P9306 -e "SELECT id FROM test_vector_regen WHERE KNN(vec, 2, 'artific
 +------+
 |    1 |
 |    2 |
+|    3 |
 +------+
 ––– input –––
 mysql -h0 -P9306 -e "REPLACE INTO test_vector_regen (id, content) VALUES (1, 'Cooking recipes')"; echo $?
@@ -112,6 +113,7 @@ mysql -h0 -P9306 -e "SELECT id FROM test_vector_regen WHERE KNN(vec, 2, 'artific
 +------+
 |    2 |
 |    3 |
+|    1 |
 +------+
 ––– comment –––
 TEST 5: TRUNCATE with vector tables

test/clt-tests/mcl/auto-embeddings-endpoints.rec

@@ -33,21 +33,21 @@ mysql -h0 -P9306 -e "INSERT INTO emb_test (id, title, content) VALUES
 ––– output –––
 0
 ––– comment –––
-IMPORTANT: Optimize table to make k parameter work correctly
+IMPORTANT: Optimize table so k parameter is applied (oversampling still applies)
 Data is currently in ramchunk where k parameter is not applied
 ––– input –––
 mysql -h0 -P9306 -e "FLUSH RAMCHUNK emb_test; OPTIMIZE TABLE emb_test OPTION sync=1, cutoff=1"; echo $?
 ––– output –––
 0
 ––– comment –––
-Verify k=2 returns exactly 2 documents after optimization
+Verify k=2 returns oversampled results after optimization
 ––– input –––
 mysql -h0 -P9306 -e "SELECT COUNT(*) FROM emb_test WHERE KNN(vec, 2, 'artificial intelligence')"
 ––– output –––
 +----------+
 | count(*) |
 +----------+
-|        2 |
+|        3 |
 +----------+
 ––– comment –––
 === PART 2: ENDPOINT TESTS WITH OPTIMIZED TABLE ===
@@ -61,6 +61,7 @@ curl -s "http://localhost:9308/cli?select%20id,%20title%20from%20emb_test%20wher
 +----+------------------+
 | 1  | machine learning |
 | 2  | computer vision  |
+| 3  | natural language |
 +----+------------------+
 ––– comment –––
 CLI_JSON endpoint with distance
@@ -75,7 +76,7 @@ SQL mode=raw endpoint
 ––– input –––
 curl -s -X POST "http://localhost:9308/sql?mode=raw" -d "select count(*) from emb_test where knn(vec, 2, 'neural networks')" | jq -r '.[0].data[0]."count(*)"'
 ––– output –––
-2
+3
 ––– comment –––
 JSON API insert with auto-embedding
 ––– input –––
@@ -126,7 +127,7 @@ mysql -h0 -P9306 -e "SELECT COUNT(*) FROM chunk_test WHERE KNN(vec, 1, 'learning
 |        3 |
 +----------+
 ––– comment –––
-AFTER optimization: k parameter works correctly
+AFTER optimization: k parameter still oversamples by default
 ––– input –––
 mysql -h0 -P9306 -e "FLUSH RAMCHUNK chunk_test; OPTIMIZE TABLE chunk_test OPTION sync=1, cutoff=1"; echo $?
 ––– output –––
@@ -137,5 +138,5 @@ mysql -h0 -P9306 -e "SELECT COUNT(*) FROM chunk_test WHERE KNN(vec, 1, 'learning
 +----------+
 | count(*) |
 +----------+
-|        1 |
+|        3 |
 +----------+

test/clt-tests/mcl/auto-embeddings-error-handling.rec

@@ -180,6 +180,7 @@ mysql -h0 -P9306 -e "SELECT id FROM test_rowwise WHERE KNN(vec, 1, 'artificial i
 | id   |
 +------+
 |    1 |
+|    2 |
 +------+
 ––– comment –––
 Test 16: Columnar vec attribute only
@@ -207,6 +208,7 @@ mysql -h0 -P9306 -e "SELECT id FROM test_vec_columnar WHERE KNN(vec, 1, 'artific
 | id   |
 +------+
 |    1 |
+|    2 |
 +------+
 ––– comment –––
 Test 17: Full columnar table (engine='columnar')
@@ -234,6 +236,7 @@ mysql -h0 -P9306 -e "SELECT id FROM test_full_columnar WHERE KNN(vec, 1, 'artifi
 | id   |
 +------+
 |    1 |
+|    2 |
 +------+
 ––– comment –––
 Test 18: Verify SHOW CREATE TABLE differences

test/clt-tests/mcl/auto-embeddings-hnsw-configs.rec

@@ -59,70 +59,85 @@ mysql -h0 -P9306 -e "FLUSH RAMCHUNK test_ip; OPTIMIZE TABLE test_ip OPTION sync=
 ––– comment –––
 Test 4: Verify KNN search returns correct documents
 ––– input –––
-mysql -h0 -P9306 -e "SELECT id FROM test_l2 WHERE KNN(vec, 2, 'artificial intelligence')"
-––– output –––
-+------+
-| id   |
-+------+
-|    1 |
-|    2 |
-+------+
-––– input –––
-mysql -h0 -P9306 -e "SELECT id FROM test_cosine WHERE KNN(vec, 2, 'artificial intelligence')"
-––– output –––
-+------+
-| id   |
-+------+
-|    1 |
-|    2 |
-+------+
-––– input –––
-mysql -h0 -P9306 -e "SELECT id FROM test_ip WHERE KNN(vec, 2, 'artificial intelligence')"
-––– output –––
-+------+
-| id   |
-+------+
-|    1 |
-|    2 |
-+------+
+mysql -h0 -P9306 -E -e "SELECT id FROM test_l2 WHERE KNN(vec, 2, 'artificial intelligence')"
+––– output –––
+*************************** 1. row ***************************
+id: 1
+*************************** 2. row ***************************
+id: 2
+*************************** 3. row ***************************
+id: 3
+––– input –––
+mysql -h0 -P9306 -E -e "SELECT id FROM test_cosine WHERE KNN(vec, 2, 'artificial intelligence')"
+––– output –––
+*************************** 1. row ***************************
+id: 1
+*************************** 2. row ***************************
+id: 2
+*************************** 3. row ***************************
+id: 3
+––– input –––
+mysql -h0 -P9306 -E -e "SELECT id FROM test_ip WHERE KNN(vec, 2, 'artificial intelligence')"
+––– output –––
+*************************** 1. row ***************************
+id: 1
+*************************** 2. row ***************************
+id: 2
+*************************** 3. row ***************************
+id: 3
 ––– comment –––
 Test 5: Compare distance values between different metrics
 ––– input –––
 echo "L2 (Euclidean) distances:"
-mysql -h0 -P9306 -e "SELECT id, content, KNN_DIST() as distance FROM test_l2 WHERE KNN(vec, 3, 'neural networks') ORDER BY distance"
+mysql -h0 -P9306 -E -e "SELECT id, content, KNN_DIST() as distance FROM test_l2 WHERE KNN(vec, 3, 'neural networks') ORDER BY distance"
 ––– output –––
 L2 (Euclidean) distances:
-+------+------------------+------------+
-| id   | content          | distance   |
-+------+------------------+------------+
-|    2 | deep learning    |   0.491045 |
-|    1 | machine learning | 0.61700082 |
-|    3 | cooking recipes  | 1.42384136 |
-+------+------------------+------------+
+*************************** 1. row ***************************
+      id: 2
+ content: deep learning
+distance: 0.4910#!/[0-9]{2,4}/!#
+*************************** 2. row ***************************
+      id: 1
+ content: machine learning
+distance: 0.6170#!/[0-9]{2,5}/!#
+*************************** 3. row ***************************
+      id: 3
+ content: cooking recipes
+distance: 1.42384#!/[0-9]{2,4}/!#
 ––– input –––
 echo "Cosine similarity distances (smaller = more similar):"
-mysql -h0 -P9306 -e "SELECT id, content, KNN_DIST() as distance FROM test_cosine WHERE KNN(vec, 3, 'neural networks') ORDER BY distance"
+mysql -h0 -P9306 -E -e "SELECT id, content, KNN_DIST() as distance FROM test_cosine WHERE KNN(vec, 3, 'neural networks') ORDER BY distance"
 ––– output –––
 Cosine similarity distances (smaller = more similar):
-+------+------------------+------------+
-| id   | content          | distance   |
-+------+------------------+------------+
-|    2 | deep learning    | 0.24552250 |
-|    1 | machine learning | 0.30850041 |
-|    3 | cooking recipes  | 0.71192080 |
-+------+------------------+------------+
+*************************** 1. row ***************************
+      id: 2
+ content: deep learning
+distance: 0.245522#!/[0-9]{1,3}/!#
+*************************** 2. row ***************************
+      id: 1
+ content: machine learning
+distance: 0.308500#!/[0-9]{1,3}/!#
+*************************** 3. row ***************************
+      id: 3
+ content: cooking recipes
+distance: 0.71192#!/[0-9]{0,3}/!#
 ––– input –––
 echo "Inner product distances (smaller = more similar):"
-mysql -h0 -P9306 -e "SELECT id, content, KNN_DIST() as distance FROM test_ip WHERE KNN(vec, 3, 'neural networks') ORDER BY distance"
+mysql -h0 -P9306 -E -e "SELECT id, content, KNN_DIST() as distance FROM test_ip WHERE KNN(vec, 3, 'neural networks') ORDER BY distance"
 ––– output –––
 Inner product distances (smaller = more similar):
-+------+------------------+------------+
-| id   | content          | distance   |
-+------+------------------+------------+
-|    2 | deep learning    | 0.24552250 |
-|    1 | machine learning | 0.30850047 |
-|    3 | cooking recipes  | 0.71192086 |
-+------+------------------+------------+
+*************************** 1. row ***************************
+      id: 2
+ content: deep learning
+distance: 0.245522#!/[0-9]{1,3}/!#
+*************************** 2. row ***************************
+      id: 1
+ content: machine learning
+distance: 0.308500#!/[0-9]{1,3}/!#
+*************************** 3. row ***************************
+      id: 3
+ content: cooking recipes
+distance: 0.71192#!/[0-9]{0,3}/!#
 ––– comment –––
 === HNSW INDEX PARAMETERS ===
 ––– comment –––
@@ -173,29 +188,20 @@ echo "HNSW configurations test completed"
 ––– output –––
 HNSW configurations test completed
 ––– input –––
-mysql -h0 -P9306 -e "SELECT COUNT(*) FROM test_hnsw_m4"
+mysql -h0 -P9306 -E -e "SELECT COUNT(*) FROM test_hnsw_m4"
 ––– output –––
-+----------+
-| count(*) |
-+----------+
-|        1 |
-+----------+
+*************************** 1. row ***************************
+count(*): 1
 ––– input –––
-mysql -h0 -P9306 -e "SELECT COUNT(*) FROM test_hnsw_m32"
+mysql -h0 -P9306 -E -e "SELECT COUNT(*) FROM test_hnsw_m32"
 ––– output –––
-+----------+
-| count(*) |
-+----------+
-|        1 |
-+----------+
+*************************** 1. row ***************************
+count(*): 1
 ––– input –––
-mysql -h0 -P9306 -e "SELECT COUNT(*) FROM test_hnsw_ef"
+mysql -h0 -P9306 -E -e "SELECT COUNT(*) FROM test_hnsw_ef"
 ––– output –––
-+----------+
-| count(*) |
-+----------+
-|        1 |
-+----------+
+*************************** 1. row ***************************
+count(*): 1
 ––– comment –––
 Test 9: Performance comparison with different M values
 ––– input –––
@@ -210,7 +216,7 @@ Additional documents inserted for performance comparison
 Test 10: Compare search results between different M values
 ––– input –––
 echo "Search with M=4 (faster, less connections):"
-mysql -h0 -P9306 -e "SELECT id FROM test_hnsw_m4 WHERE KNN(vec, 3, 'document content') LIMIT 3\G"
+mysql -h0 -P9306 -E -e "SELECT id FROM test_hnsw_m4 WHERE KNN(vec, 3, 'document content') LIMIT 3"
 ––– output –––
 Search with M=4 (faster, less connections):
 *************************** 1. row ***************************
@@ -221,7 +227,7 @@ id: %{NUMBER}
 id: %{NUMBER}
 ––– input –––
 echo "Search with M=32 (slower, more connections, potentially more accurate):"
-mysql -h0 -P9306 -e "SELECT id FROM test_hnsw_m32 WHERE KNN(vec, 3, 'document content') LIMIT 3\G"
+mysql -h0 -P9306 -E -e "SELECT id FROM test_hnsw_m32 WHERE KNN(vec, 3, 'document content') LIMIT 3"
 ––– output –––
 Search with M=32 (slower, more connections, potentially more accurate):
 *************************** 1. row ***************************

test/clt-tests/mcl/auto-embeddings-json-api.rec

@@ -105,7 +105,7 @@ mysql -h0 -P9306 -e "SELECT COUNT(*) FROM test_json_columnar"
 |        5 |
 +----------+
 ––– comment –––
-Optimize table for proper k parameter behavior in disk chunks
+Optimize table for oversampling behavior in disk chunks
 ––– input –––
 mysql -h0 -P9306 -e "FLUSH RAMCHUNK test_json_columnar; OPTIMIZE TABLE test_json_columnar OPTION sync=1, cutoff=1"; echo $?
 ––– output –––
@@ -116,7 +116,7 @@ Test KNN search via JSON API with explicit query_vector
 VECTOR=$(python3 -c "print(','.join(['0.01']*384))")
 curl -s -X POST http://localhost:9308/search -d "{\"index\":\"test_json_columnar\",\"knn\":{\"field\":\"embedding\",\"query_vector\":[$VECTOR],\"k\":2}}" | jq -r '.hits.total // "0"'
 ––– output –––
-2
+5
 ––– comment –––
 Create table without auto-embeddings configuration
 ––– input –––
@@ -232,7 +232,7 @@ mysql -h0 -P9306 -e "SELECT COUNT(*) FROM test_json_rowwise"
 |        5 |
 +----------+
 ––– comment –––
-Optimize table for proper k parameter behavior in disk chunks
+Optimize table for oversampling behavior in disk chunks
 ––– input –––
 mysql -h0 -P9306 -e "FLUSH RAMCHUNK test_json_rowwise; OPTIMIZE TABLE test_json_rowwise OPTION sync=1, cutoff=1"; echo $?
 ––– output –––
@@ -243,4 +243,4 @@ Test KNN search via JSON API with explicit query_vector
 VECTOR=$(python3 -c "print(','.join(['0.01']*384))")
 curl -s -X POST http://localhost:9308/search -d "{\"index\":\"test_json_rowwise\",\"knn\":{\"field\":\"embedding\",\"query_vector\":[$VECTOR],\"k\":2}}" | jq -r '.hits.total // "0"'
 ––– output –––
-2
+5

test/test_275/test.xml

@@ -43,6 +43,10 @@ searchd
 	select id, tag from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303)) facet tag order by @knn_dist asc;;
 	select id, tag, knn_dist() d from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303)) facet tag order by d asc;;
 
+	select id, knn_dist() from t where knn(emp,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
+	select id, knn_dist() from t where knn(emp,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303)) limit 3;
+	select id, knn_dist() from t where knn(emp,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303),{k=3});
+
 	create table d type='distributed' agent='<agent0_address/>:t';
 	select id, knn_dist() from d where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
 	

test/test_276/model.bin

@@ -1,4 +1,4 @@
-a:1:{i:0;a:14:{i:0;a:6:{s:13:"http_endpoint";s:11:"json/insert";s:11:"http_method";s:4:"POST";s:12:"http_request";s:99:"{
+a:1:{i:0;a:16:{i:0;a:6:{s:13:"http_endpoint";s:11:"json/insert";s:11:"http_method";s:4:"POST";s:12:"http_request";s:99:"{
 	"index":"test_vec",
 	"id":1,
 	"doc": 	{ "title" : "doc one", "vec" : [-0.0738,0.1067,0.0680] }
@@ -99,4 +99,19 @@ a:1:{i:0;a:14:{i:0;a:6:{s:13:"http_endpoint";s:11:"json/insert";s:11:"http_metho
 	"ef": 2000,
 	"rescore": true,
 	"oversampling": 3.0
-}";s:4:"rows";s:589:"{"timed_out":false,"hits":{"total":5,"total_relation":"eq","hits":[{"_id":1,"_score":1,"_knn_dist":0.1010012,"_source":{"title":"doc one","vec":[-0.0738,0.1067,0.068]}},{"_id":3,"_score":1,"_knn_dist":0.37101465,"_source":{"title":"doc three","vec":[-0.3379,0.3237,0.4578]}},{"_id":4,"_score":1,"_knn_dist":0.4464018,"_source":{"title":"doc four","vec":[-0.0292,-0.2448,0.1817]}},{"_id":5,"_score":1,"_knn_dist":0.76171523,"_source":{"title":"doc five","vec":[0.1816,-0.1221,0.682]}},{"_id":2,"_score":1,"_knn_dist":0.80648041,"_source":{"title":"doc two","vec":[0.3968,0.1118,0.7669]}}]}}";s:9:"http_code";i:200;s:4:"http";i:1;}}}
+}";s:4:"rows";s:589:"{"timed_out":false,"hits":{"total":5,"total_relation":"eq","hits":[{"_id":1,"_score":1,"_knn_dist":0.1010012,"_source":{"title":"doc one","vec":[-0.0738,0.1067,0.068]}},{"_id":3,"_score":1,"_knn_dist":0.37101465,"_source":{"title":"doc three","vec":[-0.3379,0.3237,0.4578]}},{"_id":4,"_score":1,"_knn_dist":0.4464018,"_source":{"title":"doc four","vec":[-0.0292,-0.2448,0.1817]}},{"_id":5,"_score":1,"_knn_dist":0.76171523,"_source":{"title":"doc five","vec":[0.1816,-0.1221,0.682]}},{"_id":2,"_score":1,"_knn_dist":0.80648041,"_source":{"title":"doc two","vec":[0.3968,0.1118,0.7669]}}]}}";s:9:"http_code";i:200;s:4:"http";i:1;}i:14;a:6:{s:13:"http_endpoint";s:11:"json/search";s:11:"http_method";s:4:"POST";s:12:"http_request";s:107:"{
+	"index": "test_vec",
+	"knn":
+	{
+	    "field": "vec",
+	    "query_vector": [0.0452, 0.3906, -0.0110]
+	}
+}";s:4:"rows";s:589:"{"timed_out":false,"hits":{"total":5,"total_relation":"eq","hits":[{"_id":1,"_score":1,"_knn_dist":0.1010012,"_source":{"title":"doc one","vec":[-0.0738,0.1067,0.068]}},{"_id":3,"_score":1,"_knn_dist":0.37101465,"_source":{"title":"doc three","vec":[-0.3379,0.3237,0.4578]}},{"_id":4,"_score":1,"_knn_dist":0.4464018,"_source":{"title":"doc four","vec":[-0.0292,-0.2448,0.1817]}},{"_id":5,"_score":1,"_knn_dist":0.76171523,"_source":{"title":"doc five","vec":[0.1816,-0.1221,0.682]}},{"_id":2,"_score":1,"_knn_dist":0.80648041,"_source":{"title":"doc two","vec":[0.3968,0.1118,0.7669]}}]}}";s:9:"http_code";i:200;s:4:"http";i:1;}i:15;a:6:{s:13:"http_endpoint";s:11:"json/search";s:11:"http_method";s:4:"POST";s:12:"http_request";s:120:"{
+	"index": "test_vec",
+	"knn":
+	{
+	    "field": "vec",
+	    "query_vector": [0.0452, 0.3906, -0.0110]
+	},
+	"limit": 3
+}";s:4:"rows";s:382:"{"timed_out":false,"hits":{"total":5,"total_relation":"eq","hits":[{"_id":1,"_score":1,"_knn_dist":0.1010012,"_source":{"title":"doc one","vec":[-0.0738,0.1067,0.068]}},{"_id":3,"_score":1,"_knn_dist":0.37101465,"_source":{"title":"doc three","vec":[-0.3379,0.3237,0.4578]}},{"_id":4,"_score":1,"_knn_dist":0.4464018,"_source":{"title":"doc four","vec":[-0.0292,-0.2448,0.1817]}}]}}";s:9:"http_code";i:200;s:4:"http";i:1;}}}

test/test_276/test.xml

@@ -184,6 +184,29 @@ index test_vec
 }
 </query>
 
+<query endpoint="json/search">
+{
+	"index": "test_vec",
+	"knn":
+	{
+	    "field": "vec",
+	    "query_vector": [0.0452, 0.3906, -0.0110]
+	}
+}
+</query>
+
+<query endpoint="json/search">
+{
+	"index": "test_vec",
+	"knn":
+	{
+	    "field": "vec",
+	    "query_vector": [0.0452, 0.3906, -0.0110]
+	},
+	"limit": 3
+}
+</query>
+
 </httpqueries>
 
 </test>

test/test_298/test.xml

@@ -34,7 +34,7 @@ searchd
 	insert into t values (10, 'title10', 5, (0.4089451730251312,0.0306655615568161,-0.15218715369701385,-0.6105571389198303,-0.29428744316101074,-0.07567915320396423,0.3454975187778473,-0.49481201171875));
 	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
 	flush ramchunk t;
-	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
+	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303), {rescore=0} );
     drop table t;
 
     create table t (title text, tag integer, emp float_vector knn_type='hnsw' knn_dims='8' hnsw_similarity='l2' quantization='8bit');
@@ -51,7 +51,7 @@ searchd
 	insert into t values (10, 'title10', 5, (0.4089451730251312,0.0306655615568161,-0.15218715369701385,-0.6105571389198303,-0.29428744316101074,-0.07567915320396423,0.3454975187778473,-0.49481201171875));
 	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
 	flush ramchunk t;
-	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
+	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303), {rescore=0} );
     drop table t;
 
     create table t (title text, tag integer, emp float_vector knn_type='hnsw' knn_dims='8' hnsw_similarity='l2' quantization='4bit');
@@ -70,7 +70,7 @@ searchd
 	insert into t values (10, 'title10', 5, (0.4089451730251312,0.0306655615568161,-0.15218715369701385,-0.6105571389198303,-0.29428744316101074,-0.07567915320396423,0.3454975187778473,-0.49481201171875));
 	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
 	flush ramchunk t;
-	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
+	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303), {rescore=0} );
     drop table t;
 
     create table t (title text, tag integer, emp float_vector knn_type='hnsw' knn_dims='8' hnsw_similarity='COSINE');
@@ -87,7 +87,7 @@ searchd
 	insert into t values (10, 'title10', 5, (0.4089451730251312,0.0306655615568161,-0.15218715369701385,-0.6105571389198303,-0.29428744316101074,-0.07567915320396423,0.3454975187778473,-0.49481201171875));
 	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
 	flush ramchunk t;
-	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
+	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303), {rescore=0} );
     drop table t;
 
     create table t (title text, tag integer, emp float_vector knn_type='hnsw' knn_dims='8' hnsw_similarity='COSINE' quantization='8bit');
@@ -104,7 +104,7 @@ searchd
 	insert into t values (10, 'title10', 5, (0.4089451730251312,0.0306655615568161,-0.15218715369701385,-0.6105571389198303,-0.29428744316101074,-0.07567915320396423,0.3454975187778473,-0.49481201171875));
 	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
 	flush ramchunk t;
-	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
+	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303), {rescore=0} );
     drop table t;
 
     create table t (title text, tag integer, emp float_vector knn_type='hnsw' knn_dims='8' hnsw_similarity='COSINE' quantization='4bit');
@@ -143,8 +143,8 @@ searchd
 	select id, knn_dist() from t where knn(emp1,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
 	select id, knn_dist() from t where knn(emp2,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
 	flush ramchunk t;
-	select id, knn_dist() from t where knn(emp1,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
-	select id, knn_dist() from t where knn(emp2,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
+	select id, knn_dist() from t where knn(emp1,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303), {rescore=0} );
+	select id, knn_dist() from t where knn(emp2,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303), {rescore=0} );
 
     drop table t;
 
@@ -196,7 +196,7 @@ searchd
 	insert into t values (10, 'title10', 5, (0.4089451730251312,0.0306655615568161,-0.15218715369701385,-0.6105571389198303,-0.29428744316101074,-0.07567915320396423,0.3454975187778473,-0.49481201171875));
 	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
 	flush ramchunk t;
-	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303));
+	select id, knn_dist() from t where knn(emp,3,(-0.11468425393104553,0.3873162567615509,-0.26252424716949463,0.007096240296959877,0.24166066944599152,-0.24653761088848114,0.060873936861753464,0.23045268654823303), {rescore=0} );
 
     drop table t;
 
@@ -211,7 +211,7 @@ searchd
 	create table t (vec float_vector knn_type='hnsw' knn_dims='2' hnsw_similarity='l2' quantization='1bit');
 	insert into t values(1, ());
 	flush ramchunk t;
-	select * from t where knn(vec, 5, (1,2));
+	select * from t where knn(vec, 5, (1,2), {rescore=0} );
 	drop table t;
 
 	create table t (vec float_vector knn_type='hnsw' knn_dims='2' hnsw_similarity='l2' quantization='1bit');
@@ -219,7 +219,7 @@ searchd
 	insert into t values(2, ());
 	insert into t values(3, (3.0, 4.0));
 	flush ramchunk t;
-	select id, knn_dist() from t where knn(vec, 5, (1,2));
+	select id, knn_dist() from t where knn(vec, 5, (1,2), {rescore=0} );
 	drop table t;
 
 	create table t (vec float_vector knn_type='hnsw' knn_dims='2' hnsw_similarity='COSINE' quantization='1bit');
@@ -231,7 +231,7 @@ searchd
 	create table t (vec float_vector knn_type='hnsw' knn_dims='8' hnsw_similarity='l2' quantization='1bit');
 	insert into t values(1, ());
 	flush ramchunk t;
-	select * from t where knn(vec, 5, (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8));
+	select * from t where knn(vec, 5, (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8), {rescore=0} );
 	drop table t;
 
 </sphinxql>