Merge remote-tracking branch 'gitlab/master' into kill_stmt

# Conflicts:
#	src/searchd.cpp
#	src/searchdsql.h
#	src/sphinxql.y

manual/Changelog.md

@@ -4,6 +4,10 @@
 
 ### Major Changes
 * Improved [cost-based optimizer](../Searching/Cost_based_optimizer.md#Cost-based-optimizer) which may increase query response time in many cases.
+* `ALTER TABLE <table name> REBUILD SECONDARY`
+* New tool `manticore-backup` for [backing up and restoring Manticore instance](../Securing_and_compacting_an_index/Backup_and_restore.md)
+* `KILL`
+* Added [FREEZE/UNFREEZE](../Securing_and_compacting_an_index/Freezing_a_table.md) to prepare a real-time/plain table for a backup
 
 ### Minor changes
 * Queries with stateful UDFs are now forced to be executed in a single thread
@@ -13,7 +17,7 @@
     * upgrade Manticore version
     * remove `.spidx` index files
     * start Manticore back
-    * use `ALTER TABLE <table name> REBUILD SECONDARY` (not yet implemented❗) to recover secondary indexes
+    * use `ALTER TABLE <table name> REBUILD SECONDARY` to recover secondary indexes
   - If you are running a replication cluster, full cluster restart should be performed with removal of `.spidx` files and `ALTER TABLE <table name> REBUILD SECONDARY` on all the nodes. Read about [restarting a cluster](../Creating_a_cluster/Setting_up_replication/Restarting_a_cluster.md#Restarting-a-cluster) for more details.
 * `SHOW SETTINGS`
 
@@ -363,7 +367,7 @@ sys	0m0.047s
 
   </details>
 
-- **[ALTER](Updating_index_schema.md) can add/remove a full-text field** (in RT mode). Previously it could only add/remove an attribute.
+- **[ALTER](Updating_table_schema_and_settings.md) can add/remove a full-text field** (in RT mode). Previously it could only add/remove an attribute.
 - 🔬 **Experimental: pseudo-sharding for full-scan queries** - allows to parallelize any non-full-text search query. Instead of preparing shards manually you can now just enable new option [searchd.pseudo_sharding](Server_settings/Searchd.md#pseudo_sharding) and expect up to `CPU cores` lower response time for non-full-text search queries. Note it can easily occupy all existing CPU cores, so if you care not only about latency, but throughput too - use it with caution.
 
 ### Minor changes

manual/Creating_an_index/Local_indexes/Real-time_index.md

@@ -96,7 +96,7 @@ index products {
 * [Update](../../Quick_start_guide.md#Update) attributes and full-text fields
 * [Delete](../../Quick_start_guide.md#Delete) documents
 * [Truncate](../../Emptying_an_index.md) index
-* [Change schema online](../../Updating_index_schema.md#Updating-index-schema-in-RT-mode) with help of command `ALTER`
+* [Change schema online](../../Updating_table_schema_and_settings.md#Updating-index-schema-in-RT-mode) with help of the command `ALTER`
 * [Define index](../../Creating_an_index/Local_indexes/Real-time_index.md) in a configuration file
 * Use [UUID](../../Adding_documents_to_an_index/Adding_documents_to_a_real-time_index.md#Auto-ID) for automatic ID provisioning
 

manual/Creating_an_index/NLP_and_tokenization/Low-level_tokenization.md

@@ -22,7 +22,7 @@ text `RED TUBE 5" LONG` would be indexed as `COLOR TUBE 5 INCH LONG`, and `PLANK
 
 Read more about [regexp_filter here](../../Creating_an_index/NLP_and_tokenization/Low-level_tokenization.md#regexp_filter).
 
-## Index configuration options 
+## Index configuration options
 
 ### charset_table
 
@@ -138,10 +138,10 @@ index products {
 <!-- end -->
 
 <!-- example charset_table 2 -->
-Besides definitions of characters and mappings, there are several built-in aliases that can be used. Current aliases are: 
+Besides definitions of characters and mappings, there are several built-in aliases that can be used. Current aliases are:
 * `english`
 * `russian`
-* `non_cjk` 
+* `non_cjk`
 * `cjk`
 
 <!-- request SQL -->
@@ -324,7 +324,7 @@ Blended characters are indexed both as separators and valid characters. For inst
 
 Blended characters should be used carefully:
 * since as soon as a character is defined as blended it is not a separator any more which can affect search. For example if you put a comma to the `blend_chars` and then search for `dog,cat` it will treat that as a single token `dog,cat` and if during indexation you **didn't** index `dog,cat` as `dog,cat`, but left only `dog cat` then it won't be matched.
-* therefore you need to make sure that this behaviour is desired and control it with help of another setting [blend_mode](../../Creating_an_index/NLP_and_tokenization/Low-level_tokenization.md#blend_mode)
+* therefore you need to make sure that this behaviour is desired and control it with help of the other setting [blend_mode](../../Creating_an_index/NLP_and_tokenization/Low-level_tokenization.md#blend_mode)
 
 Positions for tokens obtained by replacing blended characters with whitespace are assigned as usual, so regular keywords will be indexed just as if there was no `blend_chars` specified at all. An additional token that mixes blended and non-blended characters will be put at the starting position. For instance, if `AT&T company` occurs in the very beginning of the text field, `at` will be given position 1, `t` position 2, `company` position 3, and `AT&T` will also be given position 1 ("blending" with the opening regular keyword). Thus, querying for either `AT&T` or just `AT` will match that document, and querying for `"AT T"` as a phrase will also match it. Last but not least, phrase query for `"AT&T company"` will *also* match it, despite the position.
 
@@ -408,7 +408,7 @@ option = trim_none | trim_head | trim_tail | trim_both | trim_all | skip_pure
 <!-- example blend_mode -->
 Blended tokens indexing mode. Optional, default is `trim_none`.
 
-By default, tokens that mix blended and non-blended characters get indexed in there entirety. For instance, when both at-sign and an exclamation are in `blend_chars`, `@dude!` will get result in two tokens indexed: `@dude!` (with all the blended characters) and `dude` (without any). Therefore `@dude` query will *not* match it. 
+By default, tokens that mix blended and non-blended characters get indexed in there entirety. For instance, when both at-sign and an exclamation are in `blend_chars`, `@dude!` will get result in two tokens indexed: `@dude!` (with all the blended characters) and `dude` (without any). Therefore `@dude` query will *not* match it.
 
 `blend_mode` directive adds flexibility to this indexing behavior. It takes a comma-separated list of options.
 
@@ -427,9 +427,9 @@ Default behavior is to index the entire token, equivalent to `blend_mode = trim_
 
 Make sure you undestand that either of the blend modes limits your search, even the default one `trim_none` as with it and assuming `.` is a blended char:
 * `.dog.` will become `.dog. dog` during indexation
-* and you won't be able to find it by `dog.`. 
+* and you won't be able to find it by `dog.`.
 
-The more modes you use, the higher the chance your keyword will match something. 
+The more modes you use, the higher the chance your keyword will match something.
 
 <!-- request SQL -->
 
@@ -1071,7 +1071,7 @@ CRC dictionaries never store the original keyword text in the index. Instead, ke
 
 That approach has two drawbacks. First, there is a chance of control sum collision between several pairs of different keywords, growing quadratically with the number of unique keywords in the index. However, it is not a big concern as a chance of a single FNV64 collision in a dictionary of 1 billion entries is approximately 1:16, or 6.25 percent. And most dictionaries will be much more compact that a billion keywords, as a typical spoken human language has in the region of 1 to 10 million word forms.) Second, and more importantly, substring searches are not directly possible with control sums. Manticore alleviated that by pre-indexing all the possible substrings as separate keywords (see [min_prefix_len](../../Creating_an_index/NLP_and_tokenization/Wildcard_searching_settings.md#min_prefix_len), [min_infix_len](../../Creating_an_index/NLP_and_tokenization/Wildcard_searching_settings.md#min_infix_len) directives). That actually has an added benefit of matching substrings in the quickest way possible. But at the same time pre-indexing all substrings grows the index size a lot (factors of 3-10x and even more would not be unusual) and impacts the indexing time respectively, rendering substring searches on big indexes rather impractical.
 
-Keywords dictionary fixes both these drawbacks. It stores the keywords in the index and performs search-time wildcard expansion. For example, a search for a 'test\*'prefix could internally expand to 'test|tests|testing' query based on the dictionary contents. That expansion is fully transparent to the application, except that the separate per-keyword statistics for all the actually matched keywords would now also be reported. 
+Keywords dictionary fixes both these drawbacks. It stores the keywords in the index and performs search-time wildcard expansion. For example, a search for a 'test\*'prefix could internally expand to 'test|tests|testing' query based on the dictionary contents. That expansion is fully transparent to the application, except that the separate per-keyword statistics for all the actually matched keywords would now also be reported.
 
 For substring (infix) search extended wildcards may be used. Special symbols like '?' and '%' are supported along with substring (infix) search (e.g. "t?st\*","run%","\*abc\*"). Note, however, these wildcards work only with dict=keywords, and not elsewhere.
 
@@ -1268,11 +1268,11 @@ By default, Manticore full-text index stores not only a list of matching documen
 
 Hitless index will generally use less space than the respective regular index (about 1.5x can be expected). Both indexing and searching should be faster, at a cost of missing positional query and ranking support.  
 
-If used in positional queries (e.g. phrase queries) the hitless words are taken out from them and used as operand without a position.  For example if "hello" and "world" are hitless and "simon" and "says" are not hitless, the phrase query  `"simon says hello world"` will be converted to `("simon says" & hello & world)`, matching "hello" and "world" anywhere in the document and "simon says" as an exact phrase. 
+If used in positional queries (e.g. phrase queries) the hitless words are taken out from them and used as operand without a position.  For example if "hello" and "world" are hitless and "simon" and "says" are not hitless, the phrase query  `"simon says hello world"` will be converted to `("simon says" & hello & world)`, matching "hello" and "world" anywhere in the document and "simon says" as an exact phrase.
 
 A positional query than contains only hitless words will result in an empty phrase node, therefore the entire query will return an empty result and a warning. If the whole dictionary is hitless (using `all`) only boolean matching can be used on the respective index.
 
- 
+
 
 <!-- request SQL -->
 
@@ -1580,7 +1580,7 @@ This list controls what characters will be treated as phrase boundaries, in orde
 
 On phrase boundary, additional word position increment (specified by [phrase_boundary_step](../../Creating_an_index/NLP_and_tokenization/Low-level_tokenization.md#phrase_boundary_step)) will be added to current word position. This enables phrase-level searching through proximity queries: words in different phrases will be guaranteed to be more than phrase_boundary_step distance away from each other; so proximity search within that distance will be equivalent to phrase-level search.
 
-Phrase boundary condition will be raised if and only if such character is followed by a separator; this is to avoid abbreviations such as S.T.A.L.K.E.R or URLs being treated as several phrases. 
+Phrase boundary condition will be raised if and only if such character is followed by a separator; this is to avoid abbreviations such as S.T.A.L.K.E.R or URLs being treated as several phrases.
 
 <!-- request SQL -->
 
@@ -1803,7 +1803,7 @@ utilsApi.sql("CREATE TABLE products(title text, price float) regexp_filter = '(b
 index products {
   # index '13"' as '13inch'
   regexp_filter = \b(\d+)\" => \1inch
-  
+
   # index 'blue' or 'red' as 'color'
   regexp_filter = (blue|red) => color
 

manual/Creating_an_index/NLP_and_tokenization/Morphology.md

@@ -24,12 +24,11 @@ The morphology processors that come with our own built-in Manticore implementati
 * English, Russian, Arabic, and Czech stemmers
 * SoundEx and MetaPhone phonetic algorithms
 * Chinese word breaking algorithm
+* Snowball (libstemmer) stemmers for more than [15 other languages](../../Creating_an_index/NLP_and_tokenization/Supported_languages.md).
 
-You can also link with **libstemmer** library for even more stemmers (see details below). With libstemmer, Manticore also supports morphological processing for more than [15 other languages](../../Creating_an_index/NLP_and_tokenization/Supported_languages.md). Binary packages should come prebuilt with libstemmer support, too.
+Lemmatizers require dictionary `.pak` files that you can [download from the website](https://manticoresearch.com/install/#other-downloads). The dictionaries needs to be put in the directory specified by [lemmatizer_base](../../Server_settings/Common.md#lemmatizer_base). Also, there is the [lemmatizer_cache](../../Adding_data_from_external_storages/Plain_indexes_creation.md#lemmatizer_cache) setting which lets you speed up lemmatizing (and therefore indexing) by spending more RAM for, basically, an uncompressed dictionary cache.
 
-Lemmatizers require a dictionary that needs to be additionally downloaded from the Manticore website. That dictionary needs to be installed in a directory specified by [lemmatizer_base](../../Server_settings/Common.md#lemmatizer_base) directive. Also, there is a [lemmatizer_cache](../../Adding_data_from_external_storages/Plain_indexes_creation.md#lemmatizer_cache) directive that lets you speed up lemmatizing (and therefore indexing) by spending more RAM for, basically, an uncompressed cache of a dictionary.
-
-Chinese segmentation using [ICU](http://site.icu-project.org/) is also available. It is a much more precise, but a little bit slower way (compared to n-grams) to segment Chinese documents. [charset_table](../../Creating_an_index/NLP_and_tokenization/Low-level_tokenization.md#charset_table) must contain all Chinese characters (you can use alias "cjk"). In case of "morphology=icu_chinese" documents are first pre-processed by ICU, then the result is processed by the tokenizer (according to your charset_table) and then other morphology processors specified in the  "morphology" option are applied. When the documents are processed by ICU, only those parts of texts that contain Chinese are passed to ICU for segmentation, others can be modified by other means (different morphologies, charset_table etc.)
+The Chinese language segmentation using [ICU](http://site.icu-project.org/) is also available. It is a much more precise, but a little bit slower way (compared to n-grams) to segment Chinese documents. [charset_table](../../Creating_an_index/NLP_and_tokenization/Low-level_tokenization.md#charset_table) must contain all Chinese characters (you can use alias "cjk"). In case of "morphology=icu_chinese" documents are first pre-processed by ICU, then the result is processed by the tokenizer (according to your charset_table) and then other morphology processors specified in the  "morphology" option are applied. When the documents are processed by ICU, only those parts of texts that contain Chinese are passed to ICU for segmentation, others can be modified by other means (different morphologies, charset_table etc.)
 
 Built-in English and Russian stemmers should be faster than their libstemmer counterparts, but can produce slightly different results, because they are based on an older version.
 

manual/Listing_indexes.md

@@ -15,7 +15,7 @@ General syntax:
 SHOW TABLES [ LIKE pattern ]
 ```
 
-`SHOW TABLES` statement enumerates all currently active indexes along with their types. Existing index types are `local`, `distributed`, `rt`, `percolate` and `template`. 
+`SHOW TABLES` statement enumerates all currently active indexes along with their types. Existing index types are `local`, `distributed`, `rt`, `percolate` and `template`.
 
 
 <!-- intro -->
@@ -203,10 +203,10 @@ utilsApi.sql("SHOW TABLES LIKE 'pro%'")
 ## DESCRIBE
 
 ```sql
-{DESC | DESCRIBE} index [ LIKE pattern ]
+{DESC | DESCRIBE} table [ LIKE pattern ]
 ```
 
-`DESCRIBE` statement lists index columns and their associated types. Columns are document ID, full-text fields, and attributes. The order matches that in which fields and attributes are expected by `INSERT` and `REPLACE` statements. Column types are `field`, `integer`, `timestamp`, `ordinal`, `bool`, `float`, `bigint`, `string`, and `mva`. ID column will be typed as `bigint`. Example:
+`DESCRIBE` statement lists table columns and their associated types. Columns are document ID, full-text fields, and attributes. The order matches that in which fields and attributes are expected by `INSERT` and `REPLACE` statements. Column types are `field`, `integer`, `timestamp`, `ordinal`, `bool`, `float`, `bigint`, `string`, and `mva`. ID column will be typed as `bigint`. Example:
 
 ```sql
 mysql> DESC rt;
@@ -224,6 +224,42 @@ mysql> DESC rt;
 An optional LIKE clause is supported. Refer to
 [SHOW META](Profiling_and_monitoring/SHOW_META.md) for its syntax details.
 
+### SELECT FROM name.table
+
+<!-- example name_table -->
+You can also see table schema by executing the query `select * from <table_name>.table`. The benefit of this method is that you can use `WHERE` for filtering:
+
+<!-- request SQL -->
+```sql
+select * from idx.table where type='text';
+```
+
+<!-- response SQL -->
+```sql
++------+-------+------+----------------+
+| id   | field | type | properties     |
++------+-------+------+----------------+
+|    2 | title | text | indexed stored |
++------+-------+------+----------------+
+1 row in set (0.00 sec)
+```
+
+<!-- end -->
+
+<!-- example name_table2 -->
+
+You can also do many other things, consider `<your_table_name>.table` just a regular Manticore table where the columns are integer and string attributes.
+
+<!-- request SQL -->
+
+```sql
+select field from idx.table;
+select field, properties from idx.table where type in ('text', 'uint');
+select * from idx.table where properties any ('stored');
+```
+
+<!-- end -->
+
 ## SHOW CREATE TABLE
 
 <!-- example show_create -->
@@ -295,4 +331,3 @@ mysql> desc pq table like '%title%';
 +-------+------+----------------+
 1 row in set (0.00 sec)
 ```
-

manual/README.md

@@ -113,7 +113,7 @@
     * [Query cache](Searching/Query_cache.md)
     * [Collations](Searching/Collations.md)
     * [Cost-based optimizer](Searching/Cost_based_optimizer.md)
-* [▪️ Updating index schema](Updating_index_schema.md)    
+* [▪️ Updating table schema and settings](Updating_table_schema_and_settings.md)    
 * [▪️ Functions](Functions.md)
     * [Mathematical functions](Functions/Mathematical_functions.md)
     * [Searching and ranking functions](Functions/Searching_and_ranking_functions.md)
@@ -124,10 +124,13 @@
     * [String functions](Functions/String_functions.md)
     * [Other functions](Functions/Other_functions.md)
 * [▪️ Securing and compacting an index]
+    * [Backup and restore](Securing_and_compacting_an_index/Backup_and_restore.md)
     * [Few words about RT index structure](Securing_and_compacting_an_index/RT_index_structure.md)
     * [Flushing RAM chunk to a new disk chunk](Securing_and_compacting_an_index/Flushing_RAM_chunk_to_a_new_disk_chunk.md)
     * [Flushing RT index to disk](Securing_and_compacting_an_index/Flushing_RAM_chunk_to_disk.md)
     * [Compacting an index](Securing_and_compacting_an_index/Compacting_an_index.md)
+    * [Isolation during flushing and merging](Securing_and_compacting_an_index/Isolation_during_flushing_and_merging.md)
+    * [Freezing a table](Securing_and_compacting_an_index/Freezing_a_table.md)
     * [Flushing attributes](Securing_and_compacting_an_index/Flushing_attributes.md)
     * [Flushing hostnames](Securing_and_compacting_an_index/Flushing_hostnames.md)
 * [▪️ Security]

manual/References.md

@@ -5,7 +5,8 @@
 * [CREATE TABLE IF NOT EXISTS](Creating_an_index/Local_indexes/Plain_and_real-time_index_settings.md#General-syntax-of-CREATE-TABLE) - Creates new table
 * [CREATE TABLE LIKE](Creating_an_index/Local_indexes/Plain_and_real-time_index_settings.md#Creating-a-real-time-index-online-via-CREATE-TABLE) - Creates table using another one as a template
 * [DESCRIBE](Listing_indexes.md#DESCRIBE) - Prints out table's field list and their types
-* [ALTER TABLE](Updating_index_schema.md) - Changes table schema / settings
+* [ALTER TABLE](Updating_table_schema_and_settings.md) - Changes table schema / settings
+* [ALTER TABLE REBUILD SECONDARY](Updating_table_schema_and_settings.md#Rebuild-secondary-index) - Updates/recovers secondary indexes
 * [DROP TABLE IF EXISTS](Deleting_an_index.md#Deleting-an-index) - Deletes table [if it exists]
 * [SHOW TABLES](Listing_indexes.md#SHOW-TABLES) - Shows tables list
 * [SHOW CREATE TABLE](Listing_indexes.md#SHOW-CREATE-TABLE) - Shows SQL command how to create the table
@@ -413,6 +414,7 @@ To be put to section `searchd {}` in configuration file:
   * [pid_file](Server_settings/Searchd.md#pid_file) - Path to Manticore server pid file
   * [predicted_time_costs](Server_settings/Searchd.md#predicted_time_costs) - Costs for the query time prediction model
   * [preopen_indexes](Server_settings/Searchd.md#preopen_indexes) - Whether to forcibly preopen all indexes on startup
+  * [pseudo_sharding](Server_settings/Searchd.md#pseudo_sharding) - Enables pseudo-sharding for search queries to plain and real-time indexes
   * [qcache_max_bytes](Server_settings/Searchd.md#qcache_max_bytes) - Maximum RAM allocated for cached result sets
   * [qcache_thresh_msec](Server_settings/Searchd.md#qcache_thresh_msec) - Minimum wall time threshold for a query result to be cached
   * [qcache_ttl_sec](Server_settings/Searchd.md#qcache_ttl_sec) - Expiration period for a cached result set
@@ -427,6 +429,7 @@ To be put to section `searchd {}` in configuration file:
   * [rt_merge_iops](Server_settings/Searchd.md#rt_merge_iops) - Maximum number of I/O operations (per second) that real-time chunks merging thread is allowed to do
   * [rt_merge_maxiosize](Server_settings/Searchd.md#rt_merge_maxiosize) - Maximum size of an I/O operation that real-time chunks merging thread is allowed to do
   * [seamless_rotate](Server_settings/Searchd.md#seamless_rotate) - Prevents searchd stalls while rotating indexes with huge amounts of data to precache
+  * [secondary_indexes](Server_settings/Searchd.md#secondary_indexes) - Enables using secondary indexes for search queries
   * [server_id](Server_settings/Searchd.md#server_id) - Server identifier used as a seed to generate a unique document ID
   * [shutdown_timeout](Server_settings/Searchd.md#shutdown_timeout) - Searchd `--stopwait` timeout
   * [shutdown_token](Server_settings/Searchd.md#shutdown_token) - SHA1 hash of the password required to invoke `shutdown` command from VIP SQL connection

manual/Searching/Grouping.md

@@ -9,7 +9,7 @@ Manticore supports grouping of search results by one or multiple columns and com
 * have more than one row returned per group
 * have groups filtered
 * have groups sorted
-* be aggregated with help of [](../Searching/Grouping.md#Aggregation-functions)
+* be aggregated with help of the [aggregation functions](../Searching/Grouping.md#Aggregation-functions)
 
 <!-- intro -->
 The general syntax is:

manual/Searching/Highlighting.md

@@ -965,6 +965,9 @@ In addition to common highlighting options, several synonyms are available for J
 
 #### fields
 `fields` object contains attribute names with options. It can also be an array of field names (without any options).
+
+Note, by default the highlighting works the way it tries to highlight the results following the full-text query. I.e. in a general case when you don't specify fields to highlight the highlight is based on your full-text query, but if you specify the fields to highlight it highlights only if the full-text query matches the selected fields.
+
 #### encoder
 `encoder` can be set to `default` or `html`. When set to `html`, retains html markup when highlighting. Works similar to `html_strip_mode=retain` option.
 
@@ -997,7 +1000,7 @@ POST /search
 $index->setName('books');
 $bool = new \Manticoresearch\Query\BoolQuery();
 $bool->must(new \Manticoresearch\Query\Match(['query' => 'one|robots'], 'content'));
- 
+
 $results = $index->search($bool)->highlight(['content'],['highlight_query'=>['match'=>['*'=>'polite distance']]])->get();
 foreach($results as $doc)
 {
@@ -1016,7 +1019,7 @@ foreach($results as $doc)
     }
 }
 ```
- 
+
 <!-- request Python -->
 ``` python
 res = searchApi.search({"index":"books","query":{"match":{"content":"one|robots"}},"highlight":{"fields":["content"],"highlight_query":{"match":{"*":"polite distance"}}}})
@@ -1061,7 +1064,7 @@ query.put("match",new HashMap<String,Object>(){{
 searchRequest.setQuery(query);
 highlight = new HashMap<String,Object>(){{
 put("fields",new String[] {"content","title"});
-put("highlight_query", 
+put("highlight_query",
     new HashMap<String,Object>(){{
         put("match", new HashMap<String,Object>(){{
             put("*","polite distance");
@@ -1104,7 +1107,7 @@ POST /search
 $index->setName('books');
 $bool = new \Manticoresearch\Query\BoolQuery();
 $bool->must(new \Manticoresearch\Query\Match(['query' => 'one|robots'], '*'));
- 
+
 $results = $index->search($bool)->highlight(['content','title'],['pre_tags'=>'before_','post_tags'=>'_after'])->get();
 foreach($results as $doc)
 {
@@ -1220,7 +1223,7 @@ POST /search
 $index->setName('books');
 $bool = new \Manticoresearch\Query\BoolQuery();
 $bool->must(new \Manticoresearch\Query\Match(['query' => 'one|robots'], '*'));
- 
+
 $results = $index->search($bool)->highlight(['content','title'],['no_match_size'=>0])->get();
 foreach($results as $doc)
 {
@@ -1332,7 +1335,7 @@ POST /search
 $index->setName('books');
 $bool = new \Manticoresearch\Query\BoolQuery();
 $bool->must(new \Manticoresearch\Query\Match(['query' => 'one|robots'], '*'));
- 
+
 $results = $index->search($bool)->highlight(['content','title'],['order'=>"score"])->get();
 foreach($results as $doc)
 {
@@ -1443,7 +1446,7 @@ POST /search
 $index->setName('books');
 $bool = new \Manticoresearch\Query\BoolQuery();
 $bool->must(new \Manticoresearch\Query\Match(['query' => 'one|robots'], '*'));
- 
+
 $results = $index->search($bool)->highlight(['content','title'],['fragment_size'=>100])->get();
 foreach($results as $doc)
 {
@@ -1550,7 +1553,7 @@ POST /search
 $index->setName('books');
 $bool = new \Manticoresearch\Query\BoolQuery();
 $bool->must(new \Manticoresearch\Query\Match(['query' => 'one|robots'], '*'));
- 
+
 $results = $index->search($bool)->highlight(['content','title'],['number_of_fragments'=>10])->get();
 foreach($results as $doc)
 {
@@ -1666,7 +1669,7 @@ POST /search
 $index->setName('books');
 $bool = new \Manticoresearch\Query\BoolQuery();
 $bool->must(new \Manticoresearch\Query\Match(['query' => 'one|robots'], '*'));
- 
+
 $results = $index->search($bool)->highlight(['content'=>['limit'=>50],'title'=>new \stdClass])->get();
 foreach($results as $doc)
 {
@@ -1780,7 +1783,7 @@ POST /search
 $index->setName('books');
 $bool = new \Manticoresearch\Query\BoolQuery();
 $bool->must(new \Manticoresearch\Query\Match(['query' => 'and first'], 'content'));
- 
+
 $results = $index->search($bool)->highlight(['content'=>['limit'=>50]],['limits_per_field'=>false])->get();
 foreach($results as $doc)
 {
@@ -1904,7 +1907,7 @@ CALL SNIPPETS(('this is my document text','this is my another text'), 'forum', '
 
 <!-- end -->
 
-Most options are the same as in the [HIGHLIGHT() function](../Searching/Highlighting.md). There are, however, several options that can only be used with `CALL SNIPPETS`. 
+Most options are the same as in the [HIGHLIGHT() function](../Searching/Highlighting.md). There are, however, several options that can only be used with `CALL SNIPPETS`.
 
 <!-- example CALL SNIPPETS load files -->
 The following options can be used to highlight text stored in separate files:

manual/Securing_and_compacting_an_index/Backup_and_restore.md

@@ -0,0 +1,178 @@
+# Backup and restore
+
+It is vital to back up your tables to recover them later and be up and running again in case problems occur, such as system crashes, hardware failure, or data corruption / loss by any reason. Backups are also essential as a safeguard before upgrading Manticore Search to a new version which introduces index format change and can also be used to transfer your data to another system when you want to migrate to a new server.
+
+`manticore-backup` which is a part of Manticore Search [official packages](https://manticoresearch.com/install/) can help you automate backing up tables of a Manticore instance running in [RT mode](../Read_this_first.md#Real-time-mode-vs-plain-mode) and feel safe in case of any outages or crashes.
+
+### Installation
+
+**If you followed [the official installation instructions](https://manticoresearch.com/install/) you should have already everything installed and don't need to worry.** Otherwise, [manticore-backup](https://github.com/manticoresoftware/manticoresearch-backup) requires PHP 8.1.10 or [manticore-executor](https://github.com/manticoresoftware/executor) which is a part of package `manticore-extra` and you need to make sure either of them is available.
+
+⚠️ Note, support of Windows for `manticore-backup` is still in progress. You can use the tool on any `Linux` distribution or `macOS`.
+
+
+### How to use
+
+First, make sure you're running `manticore-backup` on the same server where the Manticore instance you are about to back up is running.
+
+Second, we recommend running the tool under the `root` user so the tool can transfer ownership of the files you are backing up. Otherwise, a backup will be also made but with no ownership transfer. In either case, you should make sure that `manticore-backup` has access to the data dir of the Manticore instance.
+
+<!-- example backup1 -->
+
+The only mandatory argument is `--backup-dir` - the path to put the backup in. In this case if you omit all the other arguments `manticore-backup` will:
+* find Manticore instance running with a default config
+* create a subdirectory in the `--backup-dir` directory with a timestamp in the name.
+* back up all tables found in the instance
+
+<!-- request Example -->
+```bash
+manticore-backup --config=path/to/manticore.conf --backup-dir=backupdir
+```
+
+<!-- response Example -->
+```bash
+Copyright (c) 2022, Manticore Software LTD (https://manticoresearch.com)
+
+Manticore config file: /etc/manticoresearch/manticore.conf
+Tables to backup: all tables
+Target dir: /mnt/backup/
+
+Manticore config
+  endpoint =  127.0.0.1:9308
+
+Manticore versions:
+  manticore: 5.0.2
+  columnar: 1.15.4
+  secondary: 1.15.4
+2022-10-04 17:18:39 [Info] Starting the backup...
+2022-10-04 17:18:39 [Info] Backing up config files...
+2022-10-04 17:18:39 [Info]   config files - OK
+2022-10-04 17:18:39 [Info] Backing up tables...
+2022-10-04 17:18:39 [Info]   pq (percolate) [425B]...
+2022-10-04 17:18:39 [Info]    OK
+2022-10-04 17:18:39 [Info]   products (rt) [512B]...
+2022-10-04 17:18:39 [Info]    OK
+2022-10-04 17:18:39 [Info] Running sync
+2022-10-04 17:18:42 [Info]  OK
+2022-10-04 17:18:42 [Info] You can find backup here: /mnt/backup/backup-20221004171839
+2022-10-04 17:18:42 [Info] Elapsed time: 2.76s
+2022-10-04 17:18:42 [Info] Done
+```
+<!-- end -->
+
+<!-- example backup2 -->
+If you want to select tables to backup feel free to use flag `--tables=tbl1,tbl2` that will back up only required tables and skip all others.
+
+<!-- request Example -->
+```bash
+manticore-backup --backup-dir=/mnt/backup/ --tables=products
+```
+
+<!-- response Example -->
+```bash
+Copyright (c) 2022, Manticore Software LTD (https://manticoresearch.com)
+
+Manticore config file: /etc/manticoresearch/manticore.conf
+Tables to backup: products
+Target dir: /mnt/backup/
+
+Manticore config
+  endpoint =  127.0.0.1:9308
+
+Manticore versions:
+  manticore: 5.0.3
+  columnar: 1.16.1
+  secondary: 0.0.0
+2022-10-04 17:25:02 [Info] Starting the backup...
+2022-10-04 17:25:02 [Info] Backing up config files...
+2022-10-04 17:25:02 [Info]   config files - OK
+2022-10-04 17:25:02 [Info] Backing up tables...
+2022-10-04 17:25:02 [Info]   products (rt) [512B]...
+2022-10-04 17:25:02 [Info]    OK
+2022-10-04 17:25:02 [Info] Running sync
+2022-10-04 17:25:06 [Info]  OK
+2022-10-04 17:25:06 [Info] You can find backup here: /mnt/backup/backup-20221004172502
+2022-10-04 17:25:06 [Info] Elapsed time: 4.82s
+2022-10-04 17:25:06 [Info] Done
+```
+<!-- end -->
+
+## Arguments
+
+| Argument | Description |
+|-|-|
+| `--backup-dir=path` | This is a path to the backup directory where a backup is stored.  The directory must exist. This argument is required and has no default value. On each backup run, it will create directory `backup-[datetime]` in the provided directory and will copy all required tables to it. So the backup-dir is a container of all your backups, and it's safe to run the script multiple times.|
+| `--restore[=backup]` | Restore from `--backup-dir`. Just --restore lists available backups. `--restore=backup` will restore from `<--backup-dir>/backup`. |
+| `--config=/path/to/manticore.conf` | Path to Manticore config. This is optional and in case it's not passed we use a default one for your operating system. It's used to get the host and port to talk with the Manticore daemon. |
+| `--tables=tbl1,tbl2, ...` | Semicolon-separated list of tables that you want to backup. To back up all the tables, just skip this argument. All the provided tables are supposed to exist in the Manticore instance you are backing up from, otherwise the backup will fail. |
+| `--compress` | Whether the backed up files should be compressed. Not by default. | optional |
+| `--unlock` | In rare cases when something goes wrong the tables can be left in locked state. Using this argument you can unlock them. |
+| `--version` | Show the current version. |
+| `--help` | Show this help. |
+
+
+## Restore
+
+<!-- example restore_list -->
+To restore Manticore instance from backup you need to do `manticore-backup --backup-dir=/path/to/backups --restore`. When you don't provide any argument for `--restore` it will just list all backups in the `--backup-dir`.
+
+<!-- request Example -->
+```bash
+manticore-backup --backup-dir=/mnt/backup/ --restore
+```
+
+<!-- response Example -->
+
+```bash
+Copyright (c) 2022, Manticore Software LTD (https://manticoresearch.com)
+
+Manticore config file:
+Backup dir: /tmp/
+
+Available backups: 3
+  backup-20221006144635 (Oct 06 2022 14:46:35)
+  backup-20221006145233 (Oct 06 2022 14:52:33)
+  backup-20221007104044 (Oct 07 2022 10:40:44)
+```
+
+<!-- end -->
+
+<!-- example restore -->
+To start a restore job you need to do `--restore=backup name`, where `backup name` is the name of the backup directory inside the `--backup-dir`.
+
+Note also, that:
+1. There should be no Manticore instance running on the same host and port as the ones you are restoring
+2. Old `manticore.json` should not exist
+3. Old configuration file should not exist
+4. Old directory with tables should exist and should be empty
+
+Only if all the conditions are met, will the restore happen. Don't worry the tool will give you hints so you don't need to remember it. What's important is to not overwrite your existing files, that's why there's the requirement to remove them beforehand if they still exist.
+
+<!-- request Example -->
+```bash
+manticore-backup --backup-dir=/mnt/backup/ --restore=backup-20221007104044
+```
+
+<!-- response Example -->
+
+```bash
+Copyright (c) 2022, Manticore Software LTD (https://manticoresearch.com)
+
+Manticore config file:
+Backup dir: /tmp/
+2022-10-07 11:17:25 [Info] Starting to restore...
+
+Manticore config
+  endpoint =  127.0.0.1:9308
+2022-10-07 11:17:25 [Info] Restoring config files...
+2022-10-07 11:17:25 [Info]   config files - OK
+2022-10-07 11:17:25 [Info] Restoring state files...
+2022-10-07 11:17:25 [Info]   config files - OK
+2022-10-07 11:17:25 [Info] Restoring data files...
+2022-10-07 11:17:25 [Info]   config files - OK
+2022-10-07 11:17:25 [Info] The backup '/tmp/backup-20221007104044' was successfully restored.
+2022-10-07 11:17:25 [Info] Elapsed time: 0.02s
+2022-10-07 11:17:25 [Info] Done
+```
+
+<!-- end -->

manual/Securing_and_compacting_an_index/Freezing_a_table.md

@@ -0,0 +1,66 @@
+# Freezing a table
+
+<!-- example freeze -->
+
+`FREEZE` prepares a real-time/plain table for a safe [backup](../Securing_and_compacting_an_index/Backup_and_restore.md). In particular it:
+1. Disables table compaction. If the table is being compacted right now `FREEZE` will wait for it to finish.
+2. Flushes current RAM chunk into a disk chunk.
+3. Flushes attributes.
+4. Disables implicit operations that may change the files on disk.
+5. Displays actual list of the files belonging to the table.
+
+Built-in tool [manticore-backup](../Securing_and_compacting_an_index/Backup_and_restore.md) uses `FREEZE` to guarantee data consistency. So can you if you want to make your own backup solution or need to freeze tables for whatever else reason. All you need to do is:
+1. `FREEZE` a table.
+2. Grab output of the `FREEZE` command and backup the provided files.
+3. `UNFREEZE` the table once you are done.
+
+<!-- request Example -->
+```sql
+FREEZE t;
+```
+
+<!-- response Example -->
+```sql
++-------------------+---------------------------------+
+| file              | normalized                      |
++-------------------+---------------------------------+
+| data/t/t.0.spa    | /work/anytest/data/t/t.0.spa    |
+| data/t/t.0.spd    | /work/anytest/data/t/t.0.spd    |
+| data/t/t.0.spds   | /work/anytest/data/t/t.0.spds   |
+| data/t/t.0.spe    | /work/anytest/data/t/t.0.spe    |
+| data/t/t.0.sph    | /work/anytest/data/t/t.0.sph    |
+| data/t/t.0.sphi   | /work/anytest/data/t/t.0.sphi   |
+| data/t/t.0.spi    | /work/anytest/data/t/t.0.spi    |
+| data/t/t.0.spm    | /work/anytest/data/t/t.0.spm    |
+| data/t/t.0.spp    | /work/anytest/data/t/t.0.spp    |
+| data/t/t.0.spt    | /work/anytest/data/t/t.0.spt    |
+| data/t/t.meta     | /work/anytest/data/t/t.meta     |
+| data/t/t.ram      | /work/anytest/data/t/t.ram      |
+| data/t/t.settings | /work/anytest/data/t/t.settings |
++-------------------+---------------------------------+
+13 rows in set (0.01 sec)
+```
+
+<!-- end -->
+
+The column `file` provides paths to the table's files inside [data_dir](../Server_settings/Searchd.md#data_dir) of the running instance. The column `normalized` shows absolute paths of the same files. If you want to back up a table it's safe to just copy the provided files with no other preparations.
+
+When a table is frozen, you can't perform `UPDATE` queries on it; they will fail with the error message `index is locked now,
+try again later`.
+
+Also, `DELETE` and `REPLACE` queries have some limitations while the table is frozen:
+* If `DELETE` affects a document stored in a current RAM chunk - it is allowed.
+* If `DELETE` affects a document in a disk chunk, but it was already deleted before - it is allowed.
+* If `DELETE` is going to change an actual disk chunk - it will wait until the table is unfrozen.
+
+Manual `FLUSH` of a RAM chunk of a frozen index will report 'success', however no actual save will happen.
+
+`DROP`/`TRUNCATE` of a frozen table **is** allowed, since such operation is not implicit. We assume that if you truncate or drop a table - you don't need it backed up anyway, therefore it should not have been frozen in the first place.
+
+`INSERT` into a frozen table is supported, but also limited: new data will be stored in RAM (as usual), until `rt_mem_limit` is reached; then new insertions will wait until the table is unfrozen.
+
+If you shut down the daemon with a frozen table, it will behave as in case of a dirty shutdown (e.g. `kill -9`): new inserted data will **not** be saved in the RAM-chunk on disk, and on restart it will be restored from a binary log (if any), or lost (if binary logging is disabled).
+
+# Unfreezing a table
+
+Unfreezing is much simpler; it just re-enables previously blocked operations and also restarts an internal compaction service. All the operations that are waiting for a table unfreeze get unfrozen too and finish their operations normally.

manual/Securing_and_compacting_an_index/Isolation_during_flushing_and_merging.md

@@ -0,0 +1,9 @@
+# Isolation during flushing and merging
+
+When flushing and compacting a real-time table Manticore provides isolation, so that a changed state doesn't affect the queries that were running when this or that operation started.
+
+For instance, while compacting a table we have a pair of disk chunks that are being merged and also a new chunk produced by merging those two. Then, at one moment we create a new version of the index, where instead of the original pair of chunks the new one is placed. That is done seamlessly, so that if there's a long-running query using the original chunks, it will continue seeing the old version of the index while a new query will see the new version with the resulting merged chunk.
+
+Same is true for flushing a RAM chunk: we merge all suitable RAM segments into a new disk chunk, and finally put a new disk chunk into the set of disk chunks and abandon the participated RAM chunk segments. During this operation, Manticore also provides isolation for those queries that started before the operation began.
+
+Moreover, these operations are also transparent for replaces and updates. If you update an attribute in a document which belongs to a disk chunk which is being merged with another one, the update will be applied both to that chunk and to the resulting chunk after the merge. If you delete a document during a merge - it will be deleted in the original chunk and also the resulting merged chunk will either have the document marked deleted, or it will have no such document at all (if the deletion happened on early stage of the merging).

manual/Server_settings/Searchd.md

@@ -81,7 +81,7 @@ attr_flush_period = 900 # persist updates to disk every 15 minutes
 <!-- example conf auto_optimize -->
 Disables or throttles automatic [OPTIMIZE](../Securing_and_compacting_an_index/Compacting_an_index.md#OPTIMIZE-INDEX).
 
-Since Manticore 4 indexes compaction happens automatically. You can change it with help of searchd setting `auto_optimize` by setting it to:
+Since Manticore 4 indexes compaction happens automatically. You can change it with help of the setting `auto_optimize` by changing it to:
 * 0 to disable automatic indexes compaction (you can still call `OPTIMIZE` manually)
 * 1 to enable it explicitly
 * N to enable it, but let OPTIMIZE start as soon as the number of disk chunks is greater than `# of CPU cores * 2 * N`
@@ -514,7 +514,7 @@ Instance-wide limit of threads one operation can use. By default appropriate ope
 
 You can also set this setting as a session or a global variable during the runtime.
 
-You can also control the behaviour on per-query with help of [threads OPTION](../Searching/Options.md#threads).
+You can also control the behaviour on per-query with help of the [threads OPTION](../Searching/Options.md#threads).
 
 <!-- intro -->
 ##### Example:

manual/Transactions.md

@@ -9,7 +9,7 @@ Transactions are supported for the following commands:
 
 Transactions are not supported for:
 * UPDATE (which is [different](Updating_documents/REPLACE_vs_UPDATE.md) from the REPLACE in that it does in-place attribute update).
-* ALTER - for [updating index schema](Updating_index_schema.md)
+* ALTER - for [updating index schema](Updating_table_schema_and_settings.md)
 * TRUNCATE - for [emptying a real-time index](Emptying_an_index.md)
 * ATTACH - for [attaching a plain index to a real-time index](Adding_data_from_external_storages/Adding_data_from_indexes/Attaching_a_plain_index_to_RT_index.md)
 * CREATE - [for creating an index](Creating_an_index/Local_indexes.md)

manual/Updating_index_schema.md → manual/Updating_table_schema_and_settings.md

@@ -246,3 +246,27 @@ mysql> show index rt settings;
 1 row in set (0.00 sec)
 ```
 <!-- end -->
+
+## Rebuild secondary index
+
+<!-- example ALTER REBUILD SECONDARY -->
+```sql
+ALTER TABLE table REBUILD SECONDARY
+```
+
+`ALTER` can also be used to rebuild secondary indexes in a given table. Sometimes a secondary index can be disabled for a whole index or for one/multiple attributes in the index. `ALTER TABLE table REBUILD SECONDARY` rebuilds the secondary index from the attribute data and enables it again. Secondary indexes get disabled:
+* on `UPDATE` of an attribute: in this case its secondary index gets disabled
+* in case Manticore loads a table with old formatted secondary indexes: in this case secondary indexes will be disabled for the whole table
+
+<!-- request Example -->
+```sql
+ALTER TABLE rt REBUILD SECONDARY;
+```
+
+<!-- response Example -->
+
+```sql
+Query OK, 0 rows affected (0.00 sec)
+```
+
+<!-- end -->

src/CMakeLists.txt

@@ -60,7 +60,7 @@ add_library ( lmanticore STATIC sphinx.cpp sphinxexcerpt.cpp sphinxquery.cpp sph
 		timeout_queue.cpp dynamic_idx.cpp columnarrt.cpp columnarmisc.cpp exprtraits.cpp columnarexpr.cpp
 		sphinx_alter.cpp columnarsort.cpp binlog.cpp chunksearchctx.cpp client_task_info.cpp
 		indexfiles.cpp attrindex_builder.cpp queryfilter.cpp aggregate.cpp secondarylib.cpp costestimate.cpp
-		docidlookup.cpp tracer.cpp )
+		docidlookup.cpp tracer.cpp attrindex_merge.cpp )
 
 add_library ( lstem STATIC sphinxsoundex.cpp sphinxmetaphone.cpp sphinxstemen.cpp sphinxstemru.cpp sphinxstemru.inl
 		sphinxstemcz.cpp sphinxstemar.cpp )
@@ -149,7 +149,7 @@ set ( HEADERS sphinxexcerpt.h sphinxfilter.h sphinxint.h sphinxjsonquery.h sphin
 		indexsettings.h columnarlib.h fileio.h memio.h memio_impl.h queryprofile.h columnarfilter.h columnargrouper.h fileutils.h
 		libutils.h conversion.h columnarsort.h sortcomp.h binlog_defs.h binlog.h ${MANTICORE_BINARY_DIR}/config/config.h
 		chunksearchctx.h indexfiles.h attrindex_builder.h queryfilter.h aggregate.h secondarylib.h
-		costestimate.h docidlookup.h tracer.h )
+		costestimate.h docidlookup.h tracer.h attrindex_merge.h )
 
 set ( SEARCHD_H searchdaemon.h searchdconfig.h searchdddl.h searchdexpr.h searchdha.h searchdreplication.h searchdsql.h
 		searchdtask.h client_task_info.h taskflushattrs.h taskflushbinlog.h taskflushmutable.h taskglobalidf.h

src/attribute.cpp

@@ -469,7 +469,7 @@ std::unique_ptr<BlobRowBuilder_i> sphCreateBlobRowBuilderUpdate ( const ISphSche
 
 static int64_t GetBlobRowOffset ( const CSphRowitem * pDocinfo, int iBlobRowOffset )
 {
-	return sphUnalignedRead ( *((int64_t*)const_cast<CSphRowitem *>(pDocinfo) + iBlobRowOffset) );
+	return sphUnalignedRead ( *( (const int64_t*)pDocinfo + iBlobRowOffset ) );
 }
 
 static int64_t GetBlobRowOffset ( const CSphMatch & tMatch, const CSphAttrLocator & tLocator )
@@ -482,11 +482,11 @@ template <typename T>
 static const BYTE * GetBlobAttr ( int iBlobAttrId, int nBlobAttrs, const BYTE * pRow, int & iLengthBytes )
 {
 	T uLen1 = sphUnalignedRead ( *( (const T*)pRow + iBlobAttrId ) );
-	T uLen0 = iBlobAttrId > 0 ? sphUnalignedRead ( *((const T*)pRow + iBlobAttrId - 1) ) : 0;
-	iLengthBytes = (int)uLen1-uLen0;
-	assert ( iLengthBytes>=0 );
+	T uLen0 = iBlobAttrId > 0 ? sphUnalignedRead ( *( (const T*)pRow + iBlobAttrId - 1 ) ) : 0;
+	iLengthBytes = (int)uLen1 - uLen0;
+	assert ( iLengthBytes >= 0 );
 
-	return iLengthBytes ? (const BYTE *)((const T*)pRow + nBlobAttrs) + uLen0 : nullptr;
+	return iLengthBytes ? (const BYTE*)( (const T*)pRow + nBlobAttrs ) + uLen0 : nullptr;
 }
 
 

src/attrindex_merge.cpp

@@ -0,0 +1,433 @@
+//
+// Copyright (c) 2017-2022, Manticore Software LTD (https://manticoresearch.com)
+// Copyright (c) 2001-2016, Andrew Aksyonoff
+// Copyright (c) 2008-2016, Sphinx Technologies Inc
+// All rights reserved
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License. You should have
+// received a copy of the GPL license along with this program; if you
+// did not, you can find it at http://www.gnu.org/
+//
+
+#include "histogram.h"
+#include "docidlookup.h"
+#include "docstore.h"
+#include "indexfiles.h"
+#include "killlist.h"
+#include "attrindex_builder.h"
+#include "secondarylib.h"
+#include "attrindex_merge.h"
+
+class AttrMerger_c : public AttrMerger_i
+{
+	AttrIndexBuilder_c						m_tMinMax;
+	HistogramContainer_c					m_tHistograms;
+	CSphVector<PlainOrColumnar_t>			m_dAttrsForHistogram;
+	CSphFixedVector<DocidRowidPair_t> 		m_dDocidLookup {0};
+	CSphWriter								m_tWriterSPA;
+	std::unique_ptr<BlobRowBuilder_i>		m_pBlobRowBuilder;
+	std::unique_ptr<DocstoreBuilder_i>		m_pDocstoreBuilder;
+	std::unique_ptr<columnar::Builder_i>	m_pColumnarBuilder;
+	RowID_t									m_tResultRowID = 0;
+	int64_t									m_iTotalBytes = 0;
+
+	MergeCb_c & 							m_tMonitor;
+	CSphString &							m_sError;
+	int64_t									m_iTotalDocs;
+
+	CSphVector<PlainOrColumnar_t>	m_dSiAttrs;
+	std::unique_ptr<SI::Builder_i>	m_pSIdxBuilder;
+
+private:
+	bool CopyPureColumnarAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t>& dRowMap );
+	bool CopyMixedAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t>& dRowMap );
+
+public:
+	AttrMerger_c ( MergeCb_c & tMonitor, CSphString & sError, int64_t iTotalDocs )
+		: m_tMonitor ( tMonitor )
+		, m_sError ( sError )
+		, m_iTotalDocs ( iTotalDocs )
+	{}
+
+	~AttrMerger_c() override {}
+
+	bool Prepare ( const CSphIndex * pSrcIndex, const CSphIndex * pDstIndex ) override;
+	bool CopyAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t>& dRowMap, DWORD uAlive ) override;
+	bool FinishMergeAttributes ( const CSphIndex * pDstIndex, BuildHeader_t& tBuildHeader ) override;
+};
+
+AttrMerger_i * AttrMerger_i::Create ( MergeCb_c & tMonitor, CSphString & sError, int64_t iTotalDocs )
+{
+	return new AttrMerger_c ( tMonitor, sError, iTotalDocs );
+}
+
+bool AttrMerger_c::Prepare ( const CSphIndex * pSrcIndex, const CSphIndex * pDstIndex )
+{
+	auto sSPA = GetExt ( SPH_EXT_SPA, true, pDstIndex );
+	if ( pDstIndex->GetMatchSchema().HasNonColumnarAttrs() && !m_tWriterSPA.OpenFile ( sSPA, m_sError ) )
+		return false;
+
+	if ( pDstIndex->GetMatchSchema().HasBlobAttrs() )
+	{
+		m_pBlobRowBuilder = sphCreateBlobRowBuilder ( pSrcIndex->GetMatchSchema(), GetExt ( SPH_EXT_SPB, true, pDstIndex ), pSrcIndex->GetSettings().m_tBlobUpdateSpace, m_sError );
+		if ( !m_pBlobRowBuilder )
+			return false;
+	}
+
+	if ( pDstIndex->GetDocstore() )
+	{
+		m_pDocstoreBuilder = CreateDocstoreBuilder ( GetExt ( SPH_EXT_SPDS, true, pDstIndex ), pDstIndex->GetDocstore()->GetDocstoreSettings(), m_sError );
+		if ( !m_pDocstoreBuilder )
+			return false;
+
+		for ( int i = 0; i < pDstIndex->GetMatchSchema().GetFieldsCount(); ++i )
+			if ( pDstIndex->GetMatchSchema().IsFieldStored(i) )
+				m_pDocstoreBuilder->AddField ( pDstIndex->GetMatchSchema().GetFieldName(i), DOCSTORE_TEXT );
+
+		for ( int i = 0; i < pDstIndex->GetMatchSchema().GetAttrsCount(); ++i )
+			if ( pDstIndex->GetMatchSchema().IsAttrStored(i) )
+				m_pDocstoreBuilder->AddField ( pDstIndex->GetMatchSchema().GetAttr(i).m_sName, DOCSTORE_ATTR );
+	}
+
+	if ( pDstIndex->GetMatchSchema().HasColumnarAttrs() )
+	{
+		m_pColumnarBuilder = CreateColumnarBuilder ( pDstIndex->GetMatchSchema(), pDstIndex->GetSettings(), GetExt ( SPH_EXT_SPC, true, pDstIndex ), m_sError );
+		if ( !m_pColumnarBuilder )
+			return false;
+	}
+
+	if ( IsSecondaryLibLoaded() )
+	{
+		m_pSIdxBuilder = CreateIndexBuilder ( 64*1024*1024, pDstIndex->GetMatchSchema(), GetExt ( SPH_EXT_SPIDX, true, pDstIndex ).cstr(), m_dSiAttrs, m_sError );
+		if ( !m_pSIdxBuilder.get() )
+			return false;
+	}
+
+	m_tMinMax.Init ( pDstIndex->GetMatchSchema() );
+
+	m_dDocidLookup.Reset ( m_iTotalDocs );
+	CreateHistograms ( m_tHistograms, m_dAttrsForHistogram, pDstIndex->GetMatchSchema() );
+
+	m_tResultRowID = 0;
+	return true;
+}
+
+
+bool AttrMerger_c::CopyPureColumnarAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t> & dRowMap )
+{
+	assert ( !tIndex.GetRawAttrs() );
+	assert ( tIndex.GetMatchSchema().GetAttr ( 0 ).IsColumnar() );
+
+	auto dColumnarIterators = CreateAllColumnarIterators ( tIndex.GetColumnar(), tIndex.GetMatchSchema() );
+	CSphVector<int64_t> dTmp;
+
+	int iChunk = tIndex.m_iChunk;
+	m_tMonitor.SetEvent ( MergeCb_c::E_MERGEATTRS_START, iChunk );
+	auto _ = AtScopeExit ( [this, iChunk] { m_tMonitor.SetEvent ( MergeCb_c::E_MERGEATTRS_FINISHED, iChunk ); } );
+	for ( RowID_t tRowID = 0, tRows = (RowID_t)dRowMap.GetLength64(); tRowID < tRows; ++tRowID )
+	{
+		if ( dRowMap[tRowID] == INVALID_ROWID )
+			continue;
+
+		if ( m_tMonitor.NeedStop() )
+			return false;
+
+		// limit granted by caller code
+		assert ( m_tResultRowID != INVALID_ROWID );
+
+		ARRAY_FOREACH ( i, dColumnarIterators )
+		{
+			auto& tIt = dColumnarIterators[i];
+			Verify ( AdvanceIterator ( tIt.first, tRowID ) );
+			SetColumnarAttr ( i, tIt.second, m_pColumnarBuilder.get(), tIt.first, dTmp );
+		}
+
+		BuildStoreHistograms ( nullptr, nullptr, dColumnarIterators, m_dAttrsForHistogram, m_tHistograms );
+
+		if ( m_pDocstoreBuilder )
+			m_pDocstoreBuilder->AddDoc ( m_tResultRowID, tIndex.GetDocstore()->GetDoc ( tRowID, nullptr, -1, false ) );
+
+		if ( m_pSIdxBuilder.get() )
+		{
+			m_pSIdxBuilder->SetRowID ( m_tResultRowID );
+			BuilderStoreAttrs ( nullptr, nullptr, dColumnarIterators, m_dSiAttrs, m_pSIdxBuilder.get(), dTmp );
+		}
+
+		m_dDocidLookup[m_tResultRowID] = { dColumnarIterators[0].first->Get(), m_tResultRowID };
+		++m_tResultRowID;
+	}
+	return true;
+}
+
+bool AttrMerger_c::CopyMixedAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t>& dRowMap )
+{
+	auto dColumnarIterators = CreateAllColumnarIterators ( tIndex.GetColumnar(), tIndex.GetMatchSchema() );
+	CSphVector<int64_t> dTmp;
+
+	int iColumnarIdLoc = tIndex.GetMatchSchema().GetAttr ( 0 ).IsColumnar() ? 0 : - 1;
+	const CSphRowitem* pRow = tIndex.GetRawAttrs();
+	assert ( pRow );
+	int iStride = tIndex.GetMatchSchema().GetRowSize();
+	CSphFixedVector<CSphRowitem> dTmpRow ( iStride );
+	auto iStrideBytes = dTmpRow.GetLengthBytes();
+	const CSphColumnInfo* pBlobLocator = tIndex.GetMatchSchema().GetAttr ( sphGetBlobLocatorName() );
+
+	int iChunk = tIndex.m_iChunk;
+	m_tMonitor.SetEvent ( MergeCb_c::E_MERGEATTRS_START, iChunk );
+	auto _ = AtScopeExit ( [this, iChunk] { m_tMonitor.SetEvent ( MergeCb_c::E_MERGEATTRS_FINISHED, iChunk ); } );
+	for ( RowID_t tRowID = 0, tRows = (RowID_t)dRowMap.GetLength64(); tRowID < tRows; ++tRowID, pRow += iStride )
+	{
+		if ( dRowMap[tRowID] == INVALID_ROWID )
+			continue;
+
+		if ( m_tMonitor.NeedStop() )
+			return false;
+
+		// limit granted by caller code
+		assert ( m_tResultRowID != INVALID_ROWID );
+
+		m_tMinMax.Collect ( pRow );
+
+		if ( m_pBlobRowBuilder )
+		{
+			const BYTE* pOldBlobRow = tIndex.GetRawBlobAttrs() + sphGetRowAttr ( pRow, pBlobLocator->m_tLocator );
+			uint64_t	uNewOffset	= m_pBlobRowBuilder->Flush ( pOldBlobRow );
+
+			memcpy ( dTmpRow.Begin(), pRow, iStrideBytes );
+			sphSetRowAttr ( dTmpRow.Begin(), pBlobLocator->m_tLocator, uNewOffset );
+
+			m_tWriterSPA.PutBytes ( dTmpRow.Begin(), iStrideBytes );
+		} else if ( iStrideBytes )
+			m_tWriterSPA.PutBytes ( pRow, iStrideBytes );
+
+		ARRAY_FOREACH ( i, dColumnarIterators )
+		{
+			auto& tIt = dColumnarIterators[i];
+			Verify ( AdvanceIterator ( tIt.first, tRowID ) );
+			SetColumnarAttr ( i, tIt.second, m_pColumnarBuilder.get(), tIt.first, dTmp );
+		}
+
+		DocID_t tDocID = iColumnarIdLoc >= 0 ? dColumnarIterators[iColumnarIdLoc].first->Get() : sphGetDocID ( pRow );
+
+		BuildStoreHistograms ( pRow, tIndex.GetRawBlobAttrs(), dColumnarIterators, m_dAttrsForHistogram, m_tHistograms );
+
+		if ( m_pDocstoreBuilder )
+			m_pDocstoreBuilder->AddDoc ( m_tResultRowID, tIndex.GetDocstore()->GetDoc ( tRowID, nullptr, -1, false ) );
+
+		if ( m_pSIdxBuilder.get() )
+		{
+			m_pSIdxBuilder->SetRowID ( m_tResultRowID );
+			BuilderStoreAttrs ( pRow, tIndex.GetRawBlobAttrs(), dColumnarIterators, m_dSiAttrs, m_pSIdxBuilder.get(), dTmp );
+		}
+
+		m_dDocidLookup[m_tResultRowID] = { tDocID, m_tResultRowID };
+		++m_tResultRowID;
+	}
+	return true;
+}
+
+
+bool AttrMerger_c::CopyAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t>& dRowMap, DWORD uAlive )
+{
+	if ( !uAlive )
+		return true;
+
+	// that is very empyric, however is better than nothing.
+	m_iTotalBytes += tIndex.GetStats().m_iTotalBytes * ( (float)uAlive / (float)dRowMap.GetLength64() );
+
+	if ( !tIndex.GetRawAttrs() )
+		return CopyPureColumnarAttributes( tIndex, dRowMap );
+	return CopyMixedAttributes ( tIndex, dRowMap );
+}
+
+
+bool AttrMerger_c::FinishMergeAttributes ( const CSphIndex * pDstIndex, BuildHeader_t& tBuildHeader )
+{
+	m_tMinMax.FinishCollect();
+	assert ( m_tResultRowID==m_iTotalDocs );
+	tBuildHeader.m_iDocinfo = m_iTotalDocs;
+	tBuildHeader.m_iTotalDocuments = m_iTotalDocs;
+	tBuildHeader.m_iTotalBytes = m_iTotalBytes;
+
+	m_dDocidLookup.Sort ( CmpDocidLookup_fn() );
+	if ( !WriteDocidLookup ( GetExt ( SPH_EXT_SPT, true, pDstIndex ), m_dDocidLookup, m_sError ) )
+		return false;
+
+	if ( pDstIndex->GetMatchSchema().HasNonColumnarAttrs() )
+	{
+		if ( m_iTotalDocs )
+		{
+			tBuildHeader.m_iMinMaxIndex = m_tWriterSPA.GetPos() / sizeof(CSphRowitem);
+			const auto& dMinMaxRows		 = m_tMinMax.GetCollected();
+			m_tWriterSPA.PutBytes ( dMinMaxRows.Begin(), dMinMaxRows.GetLengthBytes64() );
+			tBuildHeader.m_iDocinfoIndex = ( dMinMaxRows.GetLength() / pDstIndex->GetMatchSchema().GetRowSize() / 2 ) - 1;
+		}
+
+		m_tWriterSPA.CloseFile();
+		if ( m_tWriterSPA.IsError() )
+			return false;
+	}
+
+	if ( m_pBlobRowBuilder && !m_pBlobRowBuilder->Done(m_sError) )
+		return false;
+
+	std::string sErrorSTL;
+	if ( m_pColumnarBuilder && !m_pColumnarBuilder->Done(sErrorSTL) )
+	{
+		m_sError = sErrorSTL.c_str();
+		return false;
+	}
+
+	if ( !m_tHistograms.Save ( GetExt ( SPH_EXT_SPHI, true, pDstIndex ), m_sError ) )
+		return false;
+
+	if ( !CheckDocsCount ( m_tResultRowID, m_sError ) )
+		return false;
+
+	if ( m_pDocstoreBuilder )
+		m_pDocstoreBuilder->Finalize();
+
+	std::string sError;
+	if ( m_pSIdxBuilder.get() && !m_pSIdxBuilder->Done ( sError ) )
+	{
+		m_sError = sError.c_str();
+		return false;
+	}
+
+	return WriteDeadRowMap ( GetExt ( SPH_EXT_SPM, true, pDstIndex ), m_tResultRowID, m_sError );
+}
+
+
+class SiBuilder_c
+{
+public:
+	SiBuilder_c (  MergeCb_c & tMonitor, CSphString & sError )
+		: m_tMonitor ( tMonitor )
+		, m_sError ( sError )
+	{}
+
+	bool CopyAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t> & dRowMap );
+
+	CSphString m_sFilename;
+
+private:
+	MergeCb_c & m_tMonitor;
+	CSphString & m_sError;
+
+	CSphVector<PlainOrColumnar_t>	m_dSiAttrs;
+	std::unique_ptr<SI::Builder_i>	m_pSIdxBuilder;
+
+	bool CopyPureColumnarAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t> & dRowMap );
+	bool CopyMixedAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t> & dRowMap );
+};
+
+bool SiBuilder_c::CopyPureColumnarAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t> & dRowMap )
+{
+	assert ( !tIndex.GetRawAttrs() );
+	assert ( tIndex.GetMatchSchema().GetAttr ( 0 ).IsColumnar() );
+
+	auto dColumnarIterators = CreateAllColumnarIterators ( tIndex.GetColumnar(), tIndex.GetMatchSchema() );
+	CSphVector<int64_t> dTmp;
+
+	int iChunk = tIndex.m_iChunk;
+	m_tMonitor.SetEvent ( MergeCb_c::E_MERGEATTRS_START, iChunk );
+	auto _ = AtScopeExit ( [this, iChunk] { m_tMonitor.SetEvent ( MergeCb_c::E_MERGEATTRS_FINISHED, iChunk ); } );
+
+	for ( RowID_t tRowID = 0, tRows = (RowID_t)dRowMap.GetLength64(); tRowID<tRows; ++tRowID )
+	{
+		if ( dRowMap[tRowID] == INVALID_ROWID )
+			continue;
+
+		if ( m_tMonitor.NeedStop() )
+			return false;
+
+		ARRAY_FOREACH ( i, dColumnarIterators )
+		{
+			auto& tIt = dColumnarIterators[i];
+			Verify ( AdvanceIterator ( tIt.first, tRowID ) );
+		}
+
+		m_pSIdxBuilder->SetRowID ( tRowID );
+		BuilderStoreAttrs ( nullptr, nullptr, dColumnarIterators, m_dSiAttrs, m_pSIdxBuilder.get(), dTmp );
+	}
+	return true;
+}
+
+bool SiBuilder_c::CopyMixedAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t> & dRowMap )
+{
+	auto dColumnarIterators = CreateAllColumnarIterators ( tIndex.GetColumnar(), tIndex.GetMatchSchema() );
+	CSphVector<int64_t> dTmp;
+
+	const CSphRowitem * pRow = tIndex.GetRawAttrs();
+	assert ( pRow );
+	int iStride = tIndex.GetMatchSchema().GetRowSize();
+
+	int iChunk = tIndex.m_iChunk;
+	m_tMonitor.SetEvent ( MergeCb_c::E_MERGEATTRS_START, iChunk );
+	auto _ = AtScopeExit ( [this, iChunk] { m_tMonitor.SetEvent ( MergeCb_c::E_MERGEATTRS_FINISHED, iChunk ); } );
+
+	for ( RowID_t tRowID = 0, tRows = (RowID_t)dRowMap.GetLength64(); tRowID<tRows; ++tRowID, pRow += iStride )
+	{
+		if ( dRowMap[tRowID] == INVALID_ROWID )
+			continue;
+
+		if ( m_tMonitor.NeedStop() )
+			return false;
+
+		ARRAY_FOREACH ( i, dColumnarIterators )
+		{
+			auto& tIt = dColumnarIterators[i];
+			Verify ( AdvanceIterator ( tIt.first, tRowID ) );
+		}
+
+		m_pSIdxBuilder->SetRowID ( tRowID );
+		BuilderStoreAttrs ( pRow, tIndex.GetRawBlobAttrs(), dColumnarIterators, m_dSiAttrs, m_pSIdxBuilder.get(), dTmp );
+	}
+	return true;
+}
+
+
+bool SiBuilder_c::CopyAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t> & dRowMap )
+{
+	if ( IsSecondaryLibLoaded() )
+	{
+		m_sFilename = GetExt ( SPH_EXT_SPIDX, true, &tIndex );
+		m_pSIdxBuilder = CreateIndexBuilder ( 64*1024*1024, tIndex.GetMatchSchema(), m_sFilename.cstr(), m_dSiAttrs, m_sError );
+	} else
+	{
+		m_sError = "secondary index library not loaded";
+	}
+
+	if ( !m_pSIdxBuilder )
+		return false;
+
+	bool bOk = false;
+	if ( !tIndex.GetRawAttrs() )
+		bOk = CopyPureColumnarAttributes( tIndex, dRowMap );
+	else
+		bOk = CopyMixedAttributes ( tIndex, dRowMap );
+
+	if ( !bOk )
+		return false;
+
+	std::string sError;
+	if ( m_pSIdxBuilder.get() && !m_pSIdxBuilder->Done ( sError ) )
+	{
+		m_sError = sError.c_str();
+		return false;
+	}
+
+	return true;
+}
+
+bool SiRecreate ( MergeCb_c & tMonitor, const CSphIndex & tIndex, const VecTraits_T<RowID_t> & dRowMap, CSphString & sFile, CSphString & sError )
+{
+	SiBuilder_c tBuilder ( tMonitor, sError );
+	if ( !tBuilder.CopyAttributes ( tIndex, dRowMap ) )
+		return false;
+
+	sFile = tBuilder.m_sFilename;
+
+	return true;
+}

src/attrindex_merge.h

@@ -0,0 +1,32 @@
+//
+// Copyright (c) 2017-2022, Manticore Software LTD (https://manticoresearch.com)
+// Copyright (c) 2001-2016, Andrew Aksyonoff
+// Copyright (c) 2008-2016, Sphinx Technologies Inc
+// All rights reserved
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License. You should have
+// received a copy of the GPL license along with this program; if you
+// did not, you can find it at http://www.gnu.org/
+//
+
+#pragma once
+
+#include "sphinx.h"
+#include "indexformat.h"
+
+class AttrMerger_i
+{
+public:
+	AttrMerger_i () {}
+	virtual ~AttrMerger_i() {}
+
+	virtual bool Prepare ( const CSphIndex * pSrcIndex, const CSphIndex * pDstIndex ) = 0;
+	virtual bool CopyAttributes ( const CSphIndex & tIndex, const VecTraits_T<RowID_t> & dRowMap, DWORD uAlive ) = 0;
+	virtual bool FinishMergeAttributes ( const CSphIndex * pDstIndex, BuildHeader_t & tBuildHeader ) = 0;
+
+	static AttrMerger_i * Create ( MergeCb_c & tMonitor, CSphString & sError, int64_t iTotalDocs );
+};
+
+bool SiRecreate ( MergeCb_c & tMonitor, const CSphIndex & tIndex, const VecTraits_T<RowID_t> & dRowMap, CSphString & sFile, CSphString & sError );
+

src/datareader.cpp

@@ -298,7 +298,7 @@ public:
 	// returns depended reader sharing same mmap as maker
 	FileBlockReader_c * MakeReader ( BYTE *, int ) final
 	{
-		auto pReader = new ThinMMapReader_c ( m_tBackendFile.GetWritePtr(),
+		auto pReader = new ThinMMapReader_c ( m_tBackendFile.GetReadPtr(),
 			m_tBackendFile.GetLength64(), m_tBackendFile.GetFileName() );
 		if ( m_iPos )
 			pReader->SeekTo ( m_iPos, 0 );

src/ddl.l

@@ -79,9 +79,11 @@ ALPHA				[a-zA-Z_]+
 "MULTI64"			{ YYSTOREBOUNDS; return TOK_MULTI64; }
 "NOT"				{ YYSTOREBOUNDS; return TOK_NOT; }
 "PLUGIN"			{ YYSTOREBOUNDS; return TOK_PLUGIN; }
+"REBUILD"			{ YYSTOREBOUNDS; return TOK_REBUILD; }
 "RECONFIGURE"		{ YYSTOREBOUNDS; return TOK_RECONFIGURE; }
 "RETURNS"			{ YYSTOREBOUNDS; return TOK_RETURNS; }
 "RTINDEX"			{ YYSTOREBOUNDS; return TOK_RTINDEX; }
+"SECONDARY"			{ YYSTOREBOUNDS; return TOK_SECONDARY; }
 "SONAME"			{ YYSTOREBOUNDS; return TOK_SONAME; }
 "STORED"			{ YYSTOREBOUNDS; return TOK_STORED; }
 "STRING"			{ YYSTOREBOUNDS; return TOK_STRING; }

src/ddl.y

@@ -50,9 +50,11 @@
 %token	TOK_MULTI64
 %token	TOK_NOT
 %token	TOK_PLUGIN
+%token	TOK_REBUILD
 %token	TOK_RECONFIGURE
 %token	TOK_RETURNS
 %token	TOK_RTINDEX
+%token	TOK_SECONDARY
 %token	TOK_SONAME
 %token	TOK_STORED
 %token	TOK_STRING
@@ -184,6 +186,12 @@ alter:
 			pParser->ToString ( tStmt.m_sCluster, $3 );
 			pParser->ToString ( tStmt.m_sSetName, $5 );
 		}
+	| TOK_ALTER TOK_TABLE ident TOK_REBUILD TOK_SECONDARY
+		{
+			SqlStmt_t & tStmt = *pParser->m_pStmt;
+			tStmt.m_eStmt = STMT_ALTER_REBUILD_SI;
+			pParser->ToString ( tStmt.m_sIndex, $3 );
+		}
 	;
 
 //////////////////////////////////////////////////////////////////////////

src/docidlookup.h

@@ -56,7 +56,7 @@ class LookupReader_c
 {
 public:
 			LookupReader_c() = default;
-			LookupReader_c ( const BYTE * pData );
+			explicit LookupReader_c ( const BYTE * pData );
 
 
 	void	SetData ( const BYTE * pData );
@@ -246,4 +246,15 @@ private:
 CSphVector<RowidIterator_i *> CreateLookupIterator ( CSphVector<SecondaryIndexInfo_t> & dSIInfo, const CSphVector<CSphFilterSettings> & dFilters, const BYTE * pDocidLookup, uint32_t uTotalDocs );
 bool	WriteDocidLookup ( const CSphString & sFilename, const VecTraits_T<DocidRowidPair_t> & dLookup, CSphString & sError );
 
+struct CmpDocidLookup_fn
+{
+	static inline bool IsLess ( const DocidRowidPair_t & a, const DocidRowidPair_t & b )
+	{
+		if ( a.m_tDocID==b.m_tDocID )
+			return a.m_tRowID < b.m_tRowID;
+
+		return (uint64_t)a.m_tDocID < (uint64_t)b.m_tDocID;
+	}
+};
+
 #endif

src/docs_collector.cpp

@@ -125,15 +125,14 @@ public:
 
 /// public iface
 DocsCollector_c::DocsCollector_c ( const CSphQuery& tQuery, bool bJson, const CSphString& sIndex, const cServedIndexRefPtr_c& pDesc, CSphString* pError )
-	: m_pImpl { new Impl_c ( tQuery, bJson, sIndex, pDesc, pError ) }
-{
-	assert ( m_pImpl );
-}
+	: m_pImpl { std::make_unique<Impl_c> ( tQuery, bJson, sIndex, pDesc, pError ) }
+{}
 
-DocsCollector_c::~DocsCollector_c()
-{
-	SafeDelete ( m_pImpl );
-}
+DocsCollector_c::~DocsCollector_c() = default;
+
+DocsCollector_c::DocsCollector_c ( DocsCollector_c&& rhs ) noexcept
+	: m_pImpl ( std::exchange ( rhs.m_pImpl, nullptr ) )
+{}
 
 bool DocsCollector_c::GetValuesChunk ( CSphVector<DocID_t>& dValues, int iValues )
 {

src/docs_collector.h

@@ -17,16 +17,13 @@
 class DocsCollector_c : public ISphNoncopyable
 {
 	class Impl_c;
-	Impl_c* m_pImpl;
+	std::unique_ptr<Impl_c> m_pImpl;
 
 public:
 	DocsCollector_c ( const CSphQuery& tQuery, bool bJson, const CSphString& sIndex, const cServedIndexRefPtr_c& pDesc, CSphString* pError );
+	DocsCollector_c ( DocsCollector_c&& rhs ) noexcept;
 	~DocsCollector_c();
 
-	DocsCollector_c (DocsCollector_c&& rhs) noexcept
-		: m_pImpl ( std::exchange ( rhs.m_pImpl, nullptr ) )
-	{}
-
 	bool GetValuesChunk ( CSphVector<DocID_t>& dValues, int iValues );
 
 	// beware, that slice lives together with this class, and will become undefined once it destroyed.

src/fileutils.h

@@ -202,7 +202,7 @@ public:
 #else
 		assert ( m_iFD==-1 );
 #endif
-		assert ( !this->GetWritePtr() && !this->GetLength64() );
+		assert ( !this->GetReadPtr() && !this->GetLength64() );
 
 		T * pData = NULL;
 		int64_t iCount = 0;
@@ -297,8 +297,8 @@ public:
 		this->MemUnlock();
 
 #if _WIN32
-		if ( this->GetWritePtr() )
-			::UnmapViewOfFile ( this->GetWritePtr() );
+		if ( this->GetReadPtr() )
+			::UnmapViewOfFile ( this->GetReadPtr() );
 
 		if ( m_iMap!=INVALID_HANDLE_VALUE )
 			::CloseHandle ( m_iMap );
@@ -308,7 +308,7 @@ public:
 			::CloseHandle ( m_iFD );
 		m_iFD = INVALID_HANDLE_VALUE;
 #else
-		if ( this->GetWritePtr() )
+		if ( this->GetReadPtr() )
 			::munmap ( this->GetWritePtr(), this->GetLengthBytes() );
 
 		SafeClose ( m_iFD );
@@ -319,7 +319,7 @@ public:
 
 	bool Resize ( uint64_t uNewSize, CSphString & sWarning, CSphString & sError )
 	{
-		if ( !this->GetWritePtr() )
+		if ( !this->GetReadPtr() )
 			return false;
 
 		bool bMlock = this->m_bMemLocked;
@@ -329,7 +329,7 @@ public:
 #if _WIN32
 		assert ( m_iMap );
 
-		::UnmapViewOfFile ( this->GetWritePtr() );
+		::UnmapViewOfFile ( this->GetReadPtr() );
 		::CloseHandle ( m_iMap );
 
 		m_iMap = ::CreateFileMapping ( m_iFD, nullptr, m_bWrite ? PAGE_READWRITE : PAGE_READONLY, (DWORD)( uNewSize >> 32 ), (DWORD) ( uNewSize & 0xFFFFFFFFULL ), nullptr );
@@ -347,9 +347,8 @@ public:
 			Reset();
 			return false;
 		}
-#endif
 
-#if !_WIN32
+#else
 		if ( sphSeek ( m_iFD, uNewSize, SEEK_SET ) < 0 )
 		{
 			sError.SetSprintf ( "failed to seek '%s': %s (length=" UINT64_FMT ")", m_sFilename.cstr(), strerror(errno), uNewSize );
@@ -381,7 +380,7 @@ public:
 			return false;
 		}
 
-		if ( this->GetWritePtr() )
+		if ( this->GetReadPtr() )
 			::munmap ( this->GetWritePtr(), this->GetLengthBytes() );
 #endif
 #endif
@@ -423,11 +422,11 @@ public:
 
 	bool Flush ( bool bWaitComplete, CSphString & sError ) const
 	{
-		if ( !this->GetWritePtr() )
+		if ( !this->GetReadPtr() )
 			return true;
 
 #if _WIN32
-		if ( !::FlushViewOfFile ( this->GetWritePtr(), this->GetLengthBytes() ) )
+		if ( !::FlushViewOfFile ( this->GetReadPtr(), this->GetLengthBytes() ) )
 		{
 			sError.SetSprintf ( "FlushViewOfFile failed for '%s': errno %u", m_sFilename.cstr(), ::GetLastError() );
 			return false;

src/global_idf.cpp

@@ -156,7 +156,7 @@ DWORD CSphGlobalIDF::GetDocs ( const CSphString& sWord ) const
 	int64_t iStart = 0;
 	int64_t iEnd = m_iTotalWords - 1;
 
-	const IDFWord_t* pWords = ( IDFWord_t* ) m_pWords.GetWritePtr ();
+	auto pWords = (const IDFWord_t*)m_pWords.GetReadPtr();
 
 	if ( m_pHash.GetLengthBytes ())
 	{

src/index_converter.cpp

@@ -853,7 +853,7 @@ AttrConverter_t::AttrConverter_t ( const Index_t & tSrc, const CSphSchema & tDst
 	// persist MVA
 	if ( !tSrc.m_tMvaArena.IsEmpty() )
 	{
-		const DWORD * pMvaArena = tSrc.m_tMvaArena.GetWritePtr();
+		const DWORD * pMvaArena = tSrc.m_tMvaArena.GetReadPtr();
 		DWORD uDocs = *pMvaArena;
 		if ( uDocs )
 			m_pMvaUpdates = (pMvaArena+1) + uDocs * sizeof(SphDocID_t)/sizeof(DWORD);
@@ -867,7 +867,7 @@ CSphRowitem * AttrConverter_t::NextRow()
 	else
 		return nullptr;
 
-	const CSphRowitem * pSrcRow = m_tIndex.m_tAttr.GetWritePtr() + m_iCurRow * m_iSrcStride;
+	const CSphRowitem * pSrcRow = m_tIndex.m_tAttr.GetReadPtr() + m_iCurRow * m_iSrcStride;
 	m_tCurDocID = DOCINFO2ID ( pSrcRow );
 	const CSphRowitem * pAttrs = DOCINFO2ATTRS ( pSrcRow );
 
@@ -886,7 +886,7 @@ CSphRowitem * AttrConverter_t::NextRow()
 			const BYTE * pStr = nullptr;
 			int iLen = 0;
 			if ( uOff )
-				iLen = sphUnpackStr ( m_tIndex.m_tString.GetWritePtr() + uOff, &pStr );
+				iLen = sphUnpackStr ( m_tIndex.m_tString.GetReadPtr() + uOff, &pStr );
 
 			assert ( m_pBlob );
 			m_pBlob->SetAttr( iBlobAttr++, (const BYTE*)pStr, iLen, m_sError );
@@ -899,13 +899,13 @@ CSphRowitem * AttrConverter_t::NextRow()
 			{
 				if ( !m_tIndex.m_bArenaProhibit && ( uOff & MVA_ARENA_FLAG ) )
 				{
-					assert ( m_pMvaUpdates && m_pMvaUpdates<m_tIndex.m_tMvaArena.GetWritePtr() + m_tIndex.m_tMvaArena.GetLength64() );
+					assert ( m_pMvaUpdates && m_pMvaUpdates<m_tIndex.m_tMvaArena.GetReadPtr() + m_tIndex.m_tMvaArena.GetLength64() );
 					pMva = m_pMvaUpdates;
 					int iCount = *m_pMvaUpdates;
 					m_pMvaUpdates += iCount + 1;
 				} else
 				{
-					pMva = m_tIndex.m_tMva.GetWritePtr() + uOff;
+					pMva = m_tIndex.m_tMva.GetReadPtr() + uOff;
 				}
 			}
 
@@ -984,16 +984,6 @@ private:
 	bool ConvertDictionary ( Index_t & tIndex, CSphString & sError );
 };
 
-struct CmpDocidLookup_fn
-{
-	static inline bool IsLess ( const DocidRowidPair_t & a, const DocidRowidPair_t & b )
-	{
-		if ( a.m_tDocID==b.m_tDocID )
-			return a.m_tRowID < b.m_tRowID;
-
-		return (uint64_t)a.m_tDocID < (uint64_t)b.m_tDocID;
-	}
-};
 
 bool ConverterPlain_t::WriteLookup ( Index_t & tIndex, CSphString & sError )
 {
@@ -1629,7 +1619,7 @@ bool ConverterPlain_t::WriteKillList ( const Index_t & tIndex, bool bIgnoreKlist
 	{
 		dKillList.Resize ( tIndex.m_tKillList.GetLength () );
 		ARRAY_FOREACH ( i, dKillList )
-			dKillList[i] = tIndex.m_tKillList.GetWritePtr()[i];
+			dKillList[i] = tIndex.m_tKillList.GetReadPtr()[i];
 	}
 
 	CSphString sName = tIndex.GetFilename(SPH_EXT_SPK);

src/indexfiles.h

@@ -52,6 +52,9 @@ struct IndexFileExt_t
 CSphVector<IndexFileExt_t>	sphGetExts();
 const char*					sphGetExt ( ESphExt eExt );
 
+class CSphIndex;
+CSphString GetExt ( ESphExt eExt, bool bTemp, const CSphIndex * pIndex );
+
 /// encapsulates all common actions over index files in general (copy/rename/delete etc.)
 class IndexFiles_c : public ISphNoncopyable
 {

src/indexformat.cpp

@@ -402,7 +402,7 @@ void CWordlist::DebugPopulateCheckpoints()
 	if ( !m_pCpReader )
 		return;
 
-	const BYTE * pCur = m_tBuf.GetWritePtr() + m_iDictCheckpointsOffset;
+	const BYTE * pCur = m_tBuf.GetReadPtr() + m_iDictCheckpointsOffset;
 	ARRAY_FOREACH ( i, m_dCheckpoints )
 		pCur = m_pCpReader->ReadEntry ( pCur, m_dCheckpoints[i] );
 
@@ -414,7 +414,7 @@ const CSphWordlistCheckpoint * CWordlist::FindCheckpointCrc ( SphWordID_t iWordI
 {
 	if ( m_pCpReader ) // FIXME!!! fall to regular checkpoints after data got read
 	{
-		MappedCheckpoint_fn tPred ( m_dCheckpoints.Begin(), m_tBuf.GetWritePtr() + m_iDictCheckpointsOffset, m_pCpReader );
+		MappedCheckpoint_fn tPred ( m_dCheckpoints.Begin(), m_tBuf.GetReadPtr() + m_iDictCheckpointsOffset, m_pCpReader );
 		return sphSearchCheckpointCrc( iWordID, m_dCheckpoints, std::move(tPred));
 	}
 
@@ -425,7 +425,7 @@ const CSphWordlistCheckpoint * CWordlist::FindCheckpointWrd ( const char* sWord,
 {
 	if ( m_pCpReader ) // FIXME!!! fall to regular checkpoints after data got read
 	{
-		MappedCheckpoint_fn tPred ( m_dCheckpoints.Begin(), m_tBuf.GetWritePtr() + m_iDictCheckpointsOffset, m_pCpReader );
+		MappedCheckpoint_fn tPred ( m_dCheckpoints.Begin(), m_tBuf.GetReadPtr() + m_iDictCheckpointsOffset, m_pCpReader );
 		return sphSearchCheckpointWrd ( sWord, iWordLen, bStarMode, m_dCheckpoints, std::move ( tPred ) );
 	}
 
@@ -493,14 +493,14 @@ const BYTE * CWordlist::AcquireDict ( const CSphWordlistCheckpoint * pCheckpoint
 	SphOffset_t iOff = pCheckpoint->m_iWordlistOffset;
 	if ( m_pCpReader )
 	{
-		MappedCheckpoint_fn tPred ( m_dCheckpoints.Begin(), m_tBuf.GetWritePtr() + m_iDictCheckpointsOffset, m_pCpReader );
+		MappedCheckpoint_fn tPred ( m_dCheckpoints.Begin(), m_tBuf.GetReadPtr() + m_iDictCheckpointsOffset, m_pCpReader );
 		iOff = tPred ( pCheckpoint ).m_iWordlistOffset;
 	}
 
 	assert ( !m_tBuf.IsEmpty() );
 	assert ( iOff>0 && iOff<(int64_t)m_tBuf.GetLengthBytes() );
 
-	return m_tBuf.GetWritePtr()+iOff;
+	return m_tBuf.GetReadPtr()+iOff;
 }
 
 
@@ -570,7 +570,7 @@ void CWordlist::GetInfixedWords ( const char * sSubstring, int iSubLen, const ch
 	// lookup key1
 	// OPTIMIZE? maybe lookup key2 and reduce checkpoint set size, if possible?
 	CSphVector<DWORD> dPoints;
-	if ( !sphLookupInfixCheckpoints ( sSubstring, iBytes1, m_tBuf.GetWritePtr(), m_dInfixBlocks, m_iInfixCodepointBytes, dPoints ) )
+	if ( !sphLookupInfixCheckpoints ( sSubstring, iBytes1, m_tBuf.GetReadPtr(), m_dInfixBlocks, m_iInfixCodepointBytes, dPoints ) )
 		return;
 
 	DictEntryDiskPayload_t tDict2Payload ( tArgs.m_bPayload, tArgs.m_eHitless );
@@ -583,7 +583,7 @@ void CWordlist::GetInfixedWords ( const char * sSubstring, int iSubLen, const ch
 	ARRAY_FOREACH ( i, dPoints )
 	{
 		// OPTIMIZE? add a quicker path than a generic wildcard for "*infix*" case?
-		KeywordsBlockReader_c tDictReader ( m_tBuf.GetWritePtr() + m_dCheckpoints[dPoints[i]-1].m_iWordlistOffset, m_iSkiplistBlockSize );
+		KeywordsBlockReader_c tDictReader ( m_tBuf.GetReadPtr() + m_dCheckpoints[dPoints[i]-1].m_iWordlistOffset, m_iSkiplistBlockSize );
 		while ( tDictReader.UnpackWord() )
 		{
 			if ( sphInterrupted () )
@@ -607,7 +607,7 @@ void CWordlist::GetInfixedWords ( const char * sSubstring, int iSubLen, const ch
 
 void CWordlist::SuffixGetChekpoints ( const SuggestResult_t & , const char * sSuffix, int iLen, CSphVector<DWORD> & dCheckpoints ) const
 {
-	sphLookupInfixCheckpoints ( sSuffix, iLen, m_tBuf.GetWritePtr(), m_dInfixBlocks, m_iInfixCodepointBytes, dCheckpoints );
+	sphLookupInfixCheckpoints ( sSuffix, iLen, m_tBuf.GetReadPtr(), m_dInfixBlocks, m_iInfixCodepointBytes, dCheckpoints );
 }
 
 
@@ -615,7 +615,7 @@ void CWordlist::SetCheckpoint ( SuggestResult_t & tRes, DWORD iCP ) const
 {
 	assert ( tRes.m_pWordReader );
 	KeywordsBlockReader_c * pReader = (KeywordsBlockReader_c *)tRes.m_pWordReader;
-	pReader->Reset ( m_tBuf.GetWritePtr() + m_dCheckpoints[iCP-1].m_iWordlistOffset );
+	pReader->Reset ( m_tBuf.GetReadPtr() + m_dCheckpoints[iCP-1].m_iWordlistOffset );
 }
 
 

src/indextool.cpp

@@ -868,18 +868,18 @@ static void ApplyKilllist ( IndexInfo_t & tTarget, const IndexInfo_t & tKiller,
 {
 	if ( tSettings.m_uFlags & KillListTarget_t::USE_DOCIDS )
 	{
-		LookupReaderIterator_c tTargetReader ( tTarget.m_tLookup.GetWritePtr() );
-		LookupReaderIterator_c tKillerReader ( tKiller.m_tLookup.GetWritePtr() );
+		LookupReaderIterator_c tTargetReader ( tTarget.m_tLookup.GetReadPtr() );
+		LookupReaderIterator_c tKillerReader ( tKiller.m_tLookup.GetReadPtr() );
 
-		KillByLookup ( tTargetReader, tKillerReader, tTarget.m_tDeadRowMap, [] ( DocID_t ) {} );
+		KillByLookup ( tTargetReader, tKillerReader, tTarget.m_tDeadRowMap );
 	}
 
 	if ( tSettings.m_uFlags & KillListTarget_t::USE_KLIST )
 	{
-		LookupReaderIterator_c tTargetReader ( tTarget.m_tLookup.GetWritePtr() );
+		LookupReaderIterator_c tTargetReader ( tTarget.m_tLookup.GetReadPtr() );
 		DocidListReader_c tKillerReader ( tKiller.m_dKilllist );
 
-		KillByLookup ( tTargetReader, tKillerReader, tTarget.m_tDeadRowMap, [] ( DocID_t ) {} );
+		KillByLookup ( tTargetReader, tKillerReader, tTarget.m_tDeadRowMap );
 	}
 }
 

src/jsonqueryfilter.cpp

@@ -848,10 +848,16 @@ std::unique_ptr<FilterTreeNode_t> FilterTreeConstructor_c::ConstructRangeFilter
 	if ( !bIntFilter )
 	{
 		if ( tFilter.m_bOpenRight )
+		{
 			tFilter.m_fMaxValue = FLT_MAX;
+			tFilter.m_bHasEqualMax = true;
+		}
 
 		if ( tFilter.m_bOpenLeft )
+		{
 			tFilter.m_fMinValue = -FLT_MAX;
+			tFilter.m_bHasEqualMin = true;
+		}
 	}
 
 	return pFilterNode;

src/killlist.h

@@ -37,7 +37,7 @@ protected:
 	bool			Set ( RowID_t tRowID, DWORD * pData );
 	inline bool		IsSet ( RowID_t tRowID, const DWORD * pData ) const
 	{
-		if ( !m_bHaveDead )
+		if ( !m_bHaveDead || tRowID==INVALID_ROWID )
 			return false;
 
 		assert ( tRowID < m_uRows );
@@ -63,7 +63,7 @@ public:
 	bool		Set ( RowID_t tRowID );
 	inline bool	IsSet ( RowID_t tRowID ) const
 	{
-		return DeadRowMap_c::IsSet ( tRowID, m_tData.GetWritePtr() );
+		return DeadRowMap_c::IsSet ( tRowID, m_tData.GetReadPtr() );
 	}
 
 	int64_t		GetLengthBytes() const override;
@@ -124,7 +124,7 @@ public:
 		return true;
 	}
 
-	static inline void HintDocID ( DocID_t tDocID ) {}
+	static inline void HintDocID ( DocID_t ) {}
 
 private:
 	const DocID_t * m_pIterator {nullptr};
@@ -132,43 +132,49 @@ private:
 };
 
 
-template <typename TARGET, typename KILLER, typename MAP, typename FNHOOK>
-int KillByLookup ( TARGET & tTargetReader, KILLER & tKillerReader, MAP & tDeadRowMap, FNHOOK fnHook )
+template<typename READER1, typename READER2, typename FUNCTOR>
+void Intersect ( READER1& tReader1, READER2& tReader2, FUNCTOR&& fnFunctor )
 {
-	RowID_t tTargetRowID = INVALID_ROWID;
+	RowID_t tRowID1 = INVALID_ROWID;
+	DocID_t tDocID1 = 0, tDocID2 = 0;
+	bool bHaveDocs1 = tReader1.Read ( tDocID1, tRowID1 );
+	bool bHaveDocs2 = tReader2.ReadDocID ( tDocID2 );
 
-	DocID_t tKillerDocID = 0, tTargetDocID = 0;
-	bool bHaveKillerDocs = tKillerReader.ReadDocID ( tKillerDocID );
-	bool bHaveTargetDocs = tTargetReader.Read ( tTargetDocID, tTargetRowID );
-
-	int iKilled = 0;
-
-	while ( bHaveKillerDocs && bHaveTargetDocs )
+	while ( bHaveDocs1 && bHaveDocs2 )
 	{
-		if ( tKillerDocID < tTargetDocID )
+		if ( tDocID1 < tDocID2 )
 		{
-			tKillerReader.HintDocID ( tTargetDocID );
-			bHaveKillerDocs = tKillerReader.ReadDocID ( tKillerDocID );
-		}
-		else if ( tKillerDocID > tTargetDocID )
+			tReader1.HintDocID ( tDocID2 );
+			bHaveDocs1 = tReader1.Read ( tDocID1, tRowID1 );
+		} else if ( tDocID1 > tDocID2 )
 		{
-			tTargetReader.HintDocID ( tKillerDocID );
-			bHaveTargetDocs = tTargetReader.Read ( tTargetDocID, tTargetRowID );
-		}
-		else
+			tReader2.HintDocID ( tDocID1 );
+			bHaveDocs2 = tReader2.ReadDocID ( tDocID2 );
+		} else
 		{
-			if ( tDeadRowMap.Set ( tTargetRowID ) )
-			{
-				fnHook ( tKillerDocID );
-				++iKilled;
-			}
-
-			bHaveKillerDocs = tKillerReader.ReadDocID ( tKillerDocID );
-			bHaveTargetDocs = tTargetReader.Read ( tTargetDocID, tTargetRowID );
+			fnFunctor ( tRowID1, tDocID1, tReader2 );
+			bHaveDocs1 = tReader1.Read ( tDocID1, tRowID1 );
+			bHaveDocs2 = tReader2.ReadDocID ( tDocID2 );
 		}
 	}
+}
 
-	return iKilled;
+template<typename TARGETREADER, typename KILLERREADER, typename FNACTION>
+int ProcessIntersected ( TARGETREADER& tReader1, KILLERREADER& tReader2, FNACTION fnAction )
+{
+	int iProcessed = 0;
+	Intersect ( tReader1, tReader2, [&iProcessed, fnAction = std::move ( fnAction )] ( RowID_t tRowID, DocID_t tDocID, KILLERREADER& ) {
+		if ( fnAction ( tRowID, tDocID ) )
+			++iProcessed;
+	} );
+
+	return iProcessed;
+}
+
+template <typename TARGET, typename KILLER, typename MAP>
+int KillByLookup ( TARGET & tTargetReader, KILLER & tKillerReader, MAP & tDeadRowMap )
+{
+	return ProcessIntersected ( tTargetReader, tKillerReader, [&tDeadRowMap] ( RowID_t tRowID, DocID_t ) { return tDeadRowMap.Set ( tRowID ); } );
 }
 
 

src/networking_daemon.cpp

@@ -338,38 +338,32 @@ class CSphNetLoop::Impl_c
 /////////////////////////////////////////////////////////////////////////////
 
 CSphNetLoop::CSphNetLoop ( const VecTraits_T<Listener_t> & dListeners )
-{
-	m_pImpl = new Impl_c ( dListeners, this );
-}
+	: m_pImpl { new Impl_c ( dListeners, this ) }
+{}
 
 CSphNetLoop::~CSphNetLoop ()
 {
-	SafeDelete ( m_pImpl );
 	sphLogDebugv ( "~CSphNetLoop() (%p) completed", this );
 }
 
 void CSphNetLoop::LoopNetPoll ()
 {
 	ScopedRole_c thPoll ( NetPoollingThread );
-	assert ( m_pImpl );
 	m_pImpl->LoopNetPoll();
 }
 
 void CSphNetLoop::StopNetLoop()
 {
-	assert ( m_pImpl );
 	m_pImpl->StopNetLoop ();
 };
 
 void CSphNetLoop::AddAction ( ISphNetAction * pElem ) EXCLUDES ( NetPoollingThread )
 {
-	assert ( m_pImpl );
 	m_pImpl->AddAction ( pElem );
 }
 
 void CSphNetLoop::RemoveEvent ( NetPollEvent_t * pEvent ) REQUIRES ( NetPoollingThread )
 {
-	assert ( m_pImpl );
 	m_pImpl->RemoveEvent ( pEvent );
 }
 

src/networking_daemon.h

@@ -72,7 +72,7 @@ using WakeupEventRefPtr_c = CSphRefcountedPtr<CSphWakeupEvent>;
 class CSphNetLoop : public ISphRefcountedMT
 {
 	class Impl_c;
-	Impl_c * m_pImpl = nullptr;
+	std::unique_ptr<Impl_c> m_pImpl;
 
 protected:
 	~CSphNetLoop () override;

src/searchd.cpp

@@ -5143,7 +5143,7 @@ public:
 
 	void							RunQueries ();					///< run all queries, get all results
 	void							RunCollect ( const CSphQuery & tQuery, const CSphString & sIndex, CSphString * pErrors, CSphVector<BYTE> * pCollectedDocs );
-	void							SetQuery ( int iQuery, const CSphQuery & tQuery, ISphTableFunc * pTableFunc );
+	void							SetQuery ( int iQuery, const CSphQuery & tQuery, std::unique_ptr<ISphTableFunc> pTableFunc );
 	void							SetQueryParser ( std::unique_ptr<QueryParser_i> pParser, QueryType_e eQueryType );
 	void							SetProfile ( QueryProfile_c * pProfile );
 	AggrResult_t *					GetResult ( int iResult ) { return m_dAggrResults.Begin() + iResult; }
@@ -5156,7 +5156,7 @@ public:
 	CSphVector<SearchFailuresLog_c>	m_dFailuresSet;					///< failure logs for each query
 	CSphVector<CSphVector<int64_t>>	m_dAgentTimes;					///< per-agent time stats
 	KeepCollection_c				m_dAcquired;					/// locked indexes
-	CSphFixedVector<ISphTableFunc *>	m_dTables;
+	CSphFixedVector<std::unique_ptr<ISphTableFunc>>	m_dTables;
 	SqlStmt_t *						m_pStmt = nullptr;				///< original (one) statement to take extra options
 
 protected:
@@ -5219,54 +5219,45 @@ private:
 };
 
 PubSearchHandler_c::PubSearchHandler_c ( int iQueries, std::unique_ptr<QueryParser_i> pQueryParser, QueryType_e eQueryType, bool bMaster )
+	: m_pImpl { std::make_unique<SearchHandler_c> ( iQueries, std::move ( pQueryParser ), eQueryType, bMaster ) }
 {
-	m_pImpl = new SearchHandler_c ( iQueries, std::move ( pQueryParser ), eQueryType, bMaster );
+	assert ( m_pImpl );
 }
 
-PubSearchHandler_c::~PubSearchHandler_c ()
-{
-	delete m_pImpl;
-}
+PubSearchHandler_c::~PubSearchHandler_c () = default;
 
 void PubSearchHandler_c::RunQueries ()
 {
-	assert ( m_pImpl );
 	m_pImpl->RunQueries();
 }
 
-void PubSearchHandler_c::SetQuery ( int iQuery, const CSphQuery & tQuery, ISphTableFunc * pTableFunc )
+void PubSearchHandler_c::SetQuery ( int iQuery, const CSphQuery & tQuery, std::unique_ptr<ISphTableFunc> pTableFunc )
 {
-	assert ( m_pImpl );
-	m_pImpl->SetQuery ( iQuery, tQuery, pTableFunc );
+	m_pImpl->SetQuery ( iQuery, tQuery, std::move(pTableFunc) );
 }
 
 void PubSearchHandler_c::SetProfile ( QueryProfile_c * pProfile )
 {
-	assert ( m_pImpl );
 	m_pImpl->SetProfile ( pProfile );
 }
 
 void PubSearchHandler_c::SetStmt ( SqlStmt_t & tStmt )
 {
-	assert ( m_pImpl );
 	m_pImpl->m_pStmt = &tStmt;
 }
 
 AggrResult_t * PubSearchHandler_c::GetResult ( int iResult )
 {
-	assert ( m_pImpl );
 	return m_pImpl->GetResult (iResult);
 }
 
 void PubSearchHandler_c::PushIndex ( const CSphString& sIndex, const cServedIndexRefPtr_c& pDesc )
 {
-	assert ( m_pImpl );
 	m_pImpl->m_dAcquired.AddIndex ( sIndex, pDesc );
 }
 
 void PubSearchHandler_c::RunCollect ( const CSphQuery& tQuery, const CSphString& sIndex, CSphString* pErrors, CSphVector<BYTE>* pCollectedDocs )
 {
-	assert ( m_pImpl );
 	m_pImpl->RunCollect ( tQuery, sIndex, pErrors, pCollectedDocs );
 }
 
@@ -5280,8 +5271,6 @@ SearchHandler_c::SearchHandler_c ( int iQueries, std::unique_ptr<QueryParser_i>
 	m_dAgentTimes.Resize ( iQueries );
 	m_bMaster = bMaster;
 	m_bFederatedUser = false;
-	ARRAY_FOREACH ( i, m_dTables )
-		m_dTables[i] = nullptr;
 
 	SetQueryParser ( std::move ( pQueryParser ), eQueryType );
 	m_dResults.Resize ( iQueries );
@@ -5344,9 +5333,6 @@ public:
 
 SearchHandler_c::~SearchHandler_c ()
 {
-	ARRAY_FOREACH ( i, m_dTables )
-		SafeDelete ( m_dTables[i] );
-
 	auto dPointed = hazard::GetListOfPointed ( m_dQueries );
 	if ( !dPointed.IsEmpty () )
 	{
@@ -5457,12 +5443,12 @@ void SearchHandler_c::RunActionQuery ( const CSphQuery & tQuery, const CSphStrin
 		LogQuery ( m_dQueries[0], m_dAggrResults[0], m_dAgentTimes[0] );
 }
 
-void SearchHandler_c::SetQuery ( int iQuery, const CSphQuery & tQuery, ISphTableFunc * pTableFunc )
+void SearchHandler_c::SetQuery ( int iQuery, const CSphQuery & tQuery, std::unique_ptr<ISphTableFunc> pTableFunc )
 {
 	m_dQueries[iQuery] = tQuery;
 	m_dQueries[iQuery].m_pQueryParser = m_pQueryParser.get();
 	m_dQueries[iQuery].m_eQueryType = m_eQueryType;
-	m_dTables[iQuery] = pTableFunc;
+	m_dTables[iQuery] = std::move ( pTableFunc );
 }
 
 
@@ -7201,7 +7187,7 @@ void SearchHandler_c::RunSubset ( int iStart, int iEnd )
 	for ( int i=0; i<iQueries; ++i )
 	{
 		AggrResult_t & tRes = m_dNAggrResults[i];
-		ISphTableFunc * pTableFunc = m_dTables[iStart+i];
+		auto& pTableFunc = m_dTables[iStart+i];
 
 		// FIXME! log such queries properly?
 		if ( pTableFunc )
@@ -7491,9 +7477,9 @@ public:
 };
 
 
-ISphTableFunc * CreateRemoveRepeats()
+std::unique_ptr<ISphTableFunc> CreateRemoveRepeats()
 {
-	return new CSphTableFuncRemoveRepeats;
+	return std::make_unique<CSphTableFuncRemoveRepeats>();
 }
 
 #undef LOC_ERROR1
@@ -7515,8 +7501,8 @@ static const char * g_dSqlStmts[] =
 	"facet", "alter_reconfigure", "show_index_settings", "flush_index", "reload_plugins", "reload_index",
 	"flush_hostnames", "flush_logs", "reload_indexes", "sysfilters", "debug", "alter_killlist_target",
 	"alter_index_settings", "join_cluster", "cluster_create", "cluster_delete", "cluster_index_add",
-	"cluster_index_delete", "cluster_update", "explain", "import_table", "lock_indexes", "unlock_indexes",
-	"show_settings", "kill",
+	"cluster_index_delete", "cluster_update", "explain", "import_table", "freeze_indexes", "unfreeze_indexes",
+	"show_settings", "alter_rebuild_si", "kill",
 };
 
 
@@ -13451,10 +13437,7 @@ void HandleMysqlMultiStmt ( const CSphVector<SqlStmt_t> & dStmt, CSphQueryResult
 	auto& tSess = session::Info();
 
 	// select count
-	int iSelect = 0;
-	ARRAY_FOREACH ( i, dStmt )
-		if ( dStmt[i].m_eStmt==STMT_SELECT )
-			iSelect++;
+	int iSelect = dStmt.count_of ( [] ( const auto& tStmt ) { return tStmt.m_eStmt == STMT_SELECT; } );
 
 	CSphQueryResultMeta tPrevMeta = tLastMeta;
 
@@ -13467,22 +13450,25 @@ void HandleMysqlMultiStmt ( const CSphVector<SqlStmt_t> & dStmt, CSphQueryResult
 	QueryProfile_c tProfile;
 
 	iSelect = 0;
-	ARRAY_FOREACH ( i, dStmt )
-	{
-		if ( dStmt[i].m_eStmt==STMT_SELECT )
-		{
-			tHandler.SetQuery ( iSelect, dStmt[i].m_tQuery, dStmt[i].m_pTableFunc );
-			dStmt[i].m_pTableFunc = nullptr;
-			iSelect++;
-		}
-		else if ( dStmt[i].m_eStmt==STMT_SET && dStmt[i].m_eSet==SET_LOCAL )
+	for ( auto& tStmt: dStmt )
+		switch ( tStmt.m_eStmt )
 		{
-			CSphString sSetName ( dStmt[i].m_sSetName );
-			sSetName.ToLower();
-			if ( sSetName=="profiling" )
-				tSess.SetProfile ( ParseProfileFormat ( dStmt[i] ) );
+		case STMT_SELECT:
+			{
+				tHandler.SetQuery ( iSelect, tStmt.m_tQuery, std::move ( tStmt.m_pTableFunc ) );
+				++iSelect;
+				break;
+			}
+		case STMT_SET:
+			if ( tStmt.m_eSet == SET_LOCAL )
+			{
+				CSphString sSetName ( tStmt.m_sSetName );
+				sSetName.ToLower();
+				if ( sSetName == "profiling" )
+					tSess.SetProfile ( ParseProfileFormat ( tStmt ) );
+			}
+		default: break;
 		}
-	}
 
 	// use first meta for faceted search
 	bool bUseFirstMeta = ( tHandler.m_dQueries.GetLength()>1 && !tHandler.m_dQueries[0].m_bFacet && tHandler.m_dQueries[1].m_bFacet );
@@ -14432,7 +14418,7 @@ void HandleMysqlDebug ( RowBuffer_i &tOut, Str_t sCommand, const QueryProfile_c
 	using namespace DebugCmd;
 	CSphString sError;
 	bool bVipConn = session::GetVip ();
-	auto tCmd = DebugCmd::ParseDebugCmd ( sCommand, sError );
+	auto tCmd = ParseDebugCmd ( sCommand, sError );
 
 	if ( bVipConn )
 	{
@@ -15356,8 +15342,14 @@ static void RemoveAttrFromIndex ( const SqlStmt_t& tStmt, CSphIndex* pIdx, CSphS
 		pIdx->AddRemoveField ( false, sAttrToRemove, 0, sError );
 }
 
+enum class Alter_e
+{
+	AddColumn,
+	DropColumn,
+	RebuildSI
+};
 
-static void HandleMysqlAlter ( RowBuffer_i & tOut, const SqlStmt_t & tStmt, bool bAdd )
+static void HandleMysqlAlter ( RowBuffer_i & tOut, const SqlStmt_t & tStmt, Alter_e eAction )
 {
 	if ( !sphCheckWeCanModify ( tStmt.m_sStmt, tOut ) )
 		return;
@@ -15366,7 +15358,7 @@ static void HandleMysqlAlter ( RowBuffer_i & tOut, const SqlStmt_t & tStmt, bool
 	SearchFailuresLog_c dErrors;
 	CSphString sError;
 
-	if ( bAdd && tStmt.m_eAlterColType==SPH_ATTR_NONE )
+	if ( eAction==Alter_e::AddColumn && tStmt.m_eAlterColType==SPH_ATTR_NONE )
 	{
 		sError.SetSprintf ( "unsupported attribute type '%d'", tStmt.m_eAlterColType );
 		tOut.Error ( tStmt.m_sStmt, sError.cstr() );
@@ -15410,10 +15402,14 @@ static void HandleMysqlAlter ( RowBuffer_i & tOut, const SqlStmt_t & tStmt, bool
 
 		CSphString sAddError;
 
-		if ( bAdd )
+		if ( eAction==Alter_e::AddColumn )
 			AddAttrToIndex ( tStmt, WIdx_c ( pServed ), sAddError );
-		else
+		else if ( eAction==Alter_e::DropColumn )
 			RemoveAttrFromIndex ( tStmt, WIdx_c ( pServed ), sAddError );
+		else if ( eAction==Alter_e::RebuildSI )
+		{
+			WIdx_c ( pServed )->AlterSI ( sAddError );
+		}
 
 		if ( !sAddError.IsEmpty() )
 			dErrors.Submit ( sName, nullptr, sAddError.cstr() );
@@ -15942,7 +15938,7 @@ void HandleMysqlImportTable ( RowBuffer_i & tOut, const SqlStmt_t & tStmt, CSphS
 }
 
 //////////////////////////////////////////////////////////////////////////
-void HandleMysqlLockIndexes ( RowBuffer_i& tOut, const CSphString& sIndexes, CSphString& sWarningOut )
+void HandleMysqlFreezeIndexes ( RowBuffer_i& tOut, const CSphString& sIndexes, CSphString& sWarningOut )
 {
 	// search through specified local indexes
 	StrVec_t dIndexes, dNonlockedIndexes, dIndexFiles;
@@ -15965,7 +15961,7 @@ void HandleMysqlLockIndexes ( RowBuffer_i& tOut, const CSphString& sIndexes, CSp
 	if ( !dNonlockedIndexes.IsEmpty() )
 	{
 		StringBuilder_c sWarning;
-		sWarning << "Some indexes are not suitable for locking: ";
+		sWarning << "Some indexes are not suitable for freezing: ";
 		sWarning.StartBlock();
 		dNonlockedIndexes.for_each ( [&sWarning] ( const auto& sValue ) { sWarning << sValue; } );
 		sWarning.FinishBlocks ();
@@ -15982,7 +15978,7 @@ void HandleMysqlLockIndexes ( RowBuffer_i& tOut, const CSphString& sIndexes, CSp
 	tOut.Eof ( false, iWarnings );
 }
 
-void HandleMysqlUnlockIndexes ( RowBuffer_i& tOut, const CSphString& sIndexes, CSphString& sWarningOut )
+void HandleMysqlUnfreezeIndexes ( RowBuffer_i& tOut, const CSphString& sIndexes, CSphString& sWarningOut )
 {
 	// search through specified local indexes
 	StrVec_t dIndexes;
@@ -16163,8 +16159,7 @@ bool ClientSession_c::Execute ( Str_t sQuery, RowBuffer_i & tOut )
 
 			StatCountCommand ( SEARCHD_COMMAND_SEARCH );
 			SearchHandler_c tHandler ( 1, sphCreatePlainQueryParser(), QUERY_SQL, true );
-			tHandler.SetQuery ( 0, dStmt.Begin()->m_tQuery, dStmt.Begin()->m_pTableFunc );
-			dStmt.Begin()->m_pTableFunc = nullptr;
+			tHandler.SetQuery ( 0, dStmt.Begin()->m_tQuery, std::move ( dStmt.Begin()->m_pTableFunc ) );
 			tHandler.m_pStmt = pStmt;
 
 			if ( tSess.IsProfile() )
@@ -16400,11 +16395,15 @@ bool ClientSession_c::Execute ( Str_t sQuery, RowBuffer_i & tOut )
 		return false; // do not profile this call, keep last query profile
 
 	case STMT_ALTER_ADD:
-		HandleMysqlAlter ( tOut, *pStmt, true );
+		HandleMysqlAlter ( tOut, *pStmt, Alter_e::AddColumn );
 		return true;
 
 	case STMT_ALTER_DROP:
-		HandleMysqlAlter ( tOut, *pStmt, false );
+		HandleMysqlAlter ( tOut, *pStmt, Alter_e::DropColumn );
+		return true;
+
+	case STMT_ALTER_REBUILD_SI:
+		HandleMysqlAlter ( tOut, *pStmt, Alter_e::RebuildSI );
 		return true;
 
 	case STMT_SHOW_PLAN:
@@ -16514,12 +16513,12 @@ bool ClientSession_c::Execute ( Str_t sQuery, RowBuffer_i & tOut )
 		HandleMysqlImportTable ( tOut, *pStmt, m_tLastMeta.m_sWarning );
 		return true;
 
-	case STMT_LOCK:
-		HandleMysqlLockIndexes ( tOut, pStmt->m_sIndex, m_tLastMeta.m_sWarning);
+	case STMT_FREEZE:
+		HandleMysqlFreezeIndexes ( tOut, pStmt->m_sIndex, m_tLastMeta.m_sWarning);
 		return true;
 
-	case STMT_UNLOCK:
-		HandleMysqlUnlockIndexes ( tOut, pStmt->m_sIndex, m_tLastMeta.m_sWarning );
+	case STMT_UNFREEZE:
+		HandleMysqlUnfreezeIndexes ( tOut, pStmt->m_sIndex, m_tLastMeta.m_sWarning );
 		return true;
 
 	case STMT_SHOW_SETTINGS:

src/searchdaemon.h

@@ -1272,7 +1272,7 @@ public:
 						~PubSearchHandler_c();
 
 	void				RunQueries ();					///< run all queries, get all results
-	void				SetQuery ( int iQuery, const CSphQuery & tQuery, ISphTableFunc * pTableFunc );
+	void				SetQuery ( int iQuery, const CSphQuery & tQuery, std::unique_ptr<ISphTableFunc> pTableFunc );
 	void				SetProfile ( QueryProfile_c * pProfile );
 	void				SetStmt ( SqlStmt_t & tStmt );
 	AggrResult_t *		GetResult ( int iResult );
@@ -1281,7 +1281,7 @@ public:
 	void				RunCollect( const CSphQuery& tQuery, const CSphString& sIndex, CSphString* pErrors, CSphVector<BYTE>* pCollectedDocs );
 
 private:
-	SearchHandler_c *	m_pImpl = nullptr;
+	std::unique_ptr<SearchHandler_c>	m_pImpl;
 };
 
 
@@ -1391,7 +1391,7 @@ void LogSphinxqlError ( const char * sStmt, const Str_t& sError );
 // that is used from sphinxql command over API
 void RunSingleSphinxqlCommand ( Str_t sCommand, ISphOutputBuffer & tOut );
 
-ISphTableFunc *		CreateRemoveRepeats();
+std::unique_ptr<ISphTableFunc>		CreateRemoveRepeats();
 
 struct PercolateOptions_t
 {

src/searchdreplication.cpp

@@ -664,6 +664,7 @@ static void UpdateGroupView ( const wsrep_view_info_t * pView, ReplicationCluste
 	if ( pCluster->m_sViewNodes!=sBuf.cstr() )
 	{
 		ScopedMutex_t tLock ( pCluster->m_tViewNodesLock );
+		sphLogDebugRpl ( "view nodes changed: %s > %s", sBuf.cstr(), pCluster->m_sViewNodes.cstr() );
 		pCluster->m_sViewNodes = sBuf.cstr();
 	}
 }
@@ -1046,6 +1047,7 @@ static bool ReplicateClusterInit ( ReplicationArgs_t & tArgs, CSphString & sErro
 			sIndexes += sIndex.cstr();
 		sphLogDebugRpl ( "cluster '%s', new %d, indexes '%s', nodes '%s'", tArgs.m_pCluster->m_sName.cstr(), (int)tArgs.m_bNewCluster, sIndexes.cstr(), tArgs.m_pCluster->m_sClusterNodes.scstr() );
 	}
+	tArgs.m_pCluster->m_sViewNodes = sIncoming; // till view_handler_cb got called by Galera
 
 	auto pRecvArgs = std::make_unique<ReceiverCtx_t>();
 	pRecvArgs->m_pCluster = tArgs.m_pCluster;
@@ -1452,6 +1454,7 @@ void ReplicateClustersDelete() EXCLUDES ( g_tClustersLock )
 static void DeleteClusterByName ( const CSphString& sCluster ) EXCLUDES ( g_tClustersLock )
 {
 	Threads::SccWL_t wLock( g_tClustersLock );
+	sphLogDebugRpl ( "deleting cluster %s", sCluster.cstr() );
 	g_hClusters.Delete ( sCluster );
 }
 
@@ -2164,7 +2167,7 @@ static bool ValidateUpdate ( const ReplicationCommand_t & tCmd, CSphString & sEr
 	const ISphSchema& tSchema = RIdx_c ( pServed )->GetMatchSchema();
 
 	assert ( tCmd.m_pUpdateAPI );
-	return IndexUpdateHelper_c::Update_CheckAttributes ( *tCmd.m_pUpdateAPI, tSchema, sError );
+	return Update_CheckAttributes ( *tCmd.m_pUpdateAPI, tSchema, sError );
 }
 
 CommitMonitor_c::~CommitMonitor_c()
@@ -3765,6 +3768,8 @@ bool RemoteClusterDelete ( const CSphString & sCluster, CSphString & sError ) EX
 			return false;
 		}
 
+		sphLogDebugRpl ( "remote delte cluster %s", sCluster.cstr() );
+
 		pCluster = g_hClusters[sCluster];
 
 		// remove cluster from cache without delete of cluster itself
@@ -5623,12 +5628,13 @@ bool ClusterGetNodes ( const CSphString & sClusterNodes, const CSphString & sClu
 			// FIXME!!! no need to wait all replies in case any node get nodes list
 			// just break on 1st successful reply
 			// however need a way for distributed loop to finish as it can not break early
+			// FIXME!!! validate the nodes are the same
 			const PQRemoteReply_t & tRes = PQRemoteBase_c::GetRes ( *pAgent );
-			sNodes = tRes.m_sNodes;
+			if ( sNodes.IsEmpty() && !tRes.m_sNodes.IsEmpty() )
+				sNodes = tRes.m_sNodes;
 		}
 	}
 
-
 	bool bGotNodes = ( !sNodes.IsEmpty() );
 	if ( !bGotNodes )
 	{

src/searchdsql.cpp

@@ -30,12 +30,6 @@ SqlStmt_t::SqlStmt_t()
 }
 
 
-SqlStmt_t::~SqlStmt_t()
-{
-	SafeDelete ( m_pTableFunc );
-}
-
-
 bool SqlStmt_t::AddSchemaItem ( const char * psName )
 {
 	m_dInsertSchema.Add ( psName );
@@ -1490,7 +1484,7 @@ bool sphParseSqlQuery ( const char * sQuery, int iLen, CSphVector<SqlStmt_t> & d
 			CSphString & sFunc = dStmt[iStmt].m_sTableFunc;
 			sFunc.ToUpper();
 
-			ISphTableFunc * pFunc = nullptr;
+			std::unique_ptr<ISphTableFunc> pFunc;
 			if ( sFunc=="REMOVE_REPEATS" )
 				pFunc = CreateRemoveRepeats();
 
@@ -1500,11 +1494,9 @@ bool sphParseSqlQuery ( const char * sQuery, int iLen, CSphVector<SqlStmt_t> & d
 				return false;
 			}
 			if ( !pFunc->ValidateArgs ( dStmt[iStmt].m_dTableFuncArgs, tQuery, sError ) )
-			{
-				SafeDelete ( pFunc );
 				return false;
-			}
-			dStmt[iStmt].m_pTableFunc = pFunc;
+
+			dStmt[iStmt].m_pTableFunc = std::move ( pFunc );
 		}
 
 		// validate filters

src/searchdsql.h

@@ -134,9 +134,10 @@ enum SqlStmt_e
 	STMT_CLUSTER_ALTER_UPDATE,
 	STMT_EXPLAIN,
 	STMT_IMPORT_TABLE,
-	STMT_LOCK,
-	STMT_UNLOCK,
+	STMT_FREEZE,
+	STMT_UNFREEZE,
 	STMT_SHOW_SETTINGS,
+	STMT_ALTER_REBUILD_SI,
 	STMT_KILL,
 
 	STMT_TOTAL
@@ -190,7 +191,7 @@ struct SqlStmt_t
 
 											   // SELECT specific
 	CSphQuery				m_tQuery;
-	ISphTableFunc *			m_pTableFunc = nullptr;
+	std::unique_ptr<ISphTableFunc>			m_pTableFunc;
 
 	CSphString				m_sTableFunc;
 	StrVec_t				m_dTableFuncArgs;
@@ -269,7 +270,6 @@ public:
 	CSphVector<int64_t>		m_dIntSubkeys;
 
 	SqlStmt_t ();
-	~SqlStmt_t();
 
 	bool AddSchemaItem ( const char * psName );
 	// check if the number of fields which would be inserted is in accordance to the given schema

src/sphinx.h

@@ -537,7 +537,6 @@ struct CSphQuery
 	bool			m_bNotOnlyAllowed = false;	///< whether allow single full-text not operator
 	CSphString		m_sStore;					///< don't delete result, just store in given uservar by name
 
-	ISphTableFunc *	m_pTableFunc = nullptr;		///< post-query NOT OWNED, WILL NOT BE FREED in dtor.
 	CSphFilterSettings	m_tHaving;				///< post aggregate filtering (got applied only on master)
 
 	int				m_iSQLSelectStart = -1;	///< SQL parser helper
@@ -892,7 +891,7 @@ struct CSphMultiQueryArgs : public ISphNoncopyable
 
 struct RowToUpdateData_t
 {
-	const CSphRowitem*	m_pRow;	/// row in the index
+	RowID_t 			m_tRow; /// row in the index
 	int					m_iIdx;	/// idx in updateset
 };
 
@@ -905,34 +904,61 @@ struct PostponedUpdate_t
 	RowsToUpdateData_t		m_dRowsToUpdate;
 };
 
+struct UpdateContext_t;
+using BlockerFn = std::function<bool()>;
+
+// common attribute update code for both RT and plain indexes
 // an index or a part of an index that has its own row ids
 class IndexSegment_c
 {
 protected:
-	mutable IndexSegment_c * m_pKillHook = nullptr; // if set, killed docids will be emerged also here.
+	mutable IndexSegment_c* m_pKillHook = nullptr; // if set, killed docids will be emerged also here.
+	enum
+	{
+		ATTRS_UPDATED			= ( 1UL<<0 ),
+		ATTRS_BLOB_UPDATED		= ( 1UL<<1 ),
+		ATTRS_ROWMAP_UPDATED	= ( 1UL<<2 )
+	};
 
-public:
-	// stuff for dispatch races between changes and updates
-	mutable std::atomic<bool>		m_bAttrsBusy { false };
-	CSphVector<PostponedUpdate_t>	m_dPostponedUpdates;
+private:
+	virtual bool		Update_WriteBlobRow ( UpdateContext_t & tCtx, RowID_t tRowID, ByteBlob_t tBlob,
+			int nBlobAttrs, const CSphAttrLocator & tBlobRowLoc, bool & bCritical, CSphString & sError ) {return false;};
+
+	static bool			Update_InplaceJson ( const RowsToUpdate_t& dRows, UpdateContext_t & tCtx, CSphString & sError, bool bDryRun );
+	bool				Update_Blobs ( const RowsToUpdate_t& dRows, UpdateContext_t & tCtx, bool & bCritical, CSphString & sError );
+	static void			Update_Plain ( const RowsToUpdate_t& dRows, UpdateContext_t & tCtx );
 
 public:
-	virtual int		Kill ( DocID_t tDocID ) { return 0; }
-	virtual int		KillMulti ( const VecTraits_T<DocID_t> & dKlist ) { return 0; };
-	virtual			~IndexSegment_c() {};
+	virtual int			Kill ( DocID_t  /*tDocID*/ ) { return 0; }
+	virtual int			KillMulti ( const VecTraits_T<DocID_t> &  /*dKlist*/ ) { return 0; };
+	virtual int			CheckThenKillMulti ( const VecTraits_T<DocID_t>& dKlist, BlockerFn&& /*fnWatcher*/ ) { return KillMulti ( dKlist ); };
+	virtual				~IndexSegment_c() = default;
 
-	inline void SetKillHook ( IndexSegment_c * pKillHook ) const
+	inline void			SetKillHook ( IndexSegment_c * pKillHook ) const
 	{
 		m_pKillHook = pKillHook;
 	}
 
+public:
+	bool Update_UpdateAttributes ( const RowsToUpdate_t& dRows, UpdateContext_t& tCtx, bool& bCritical, CSphString& sError );
+
+	/// apply serie of updates, assuming them prepared (no need to full-scan attributes), and index is offline, i.e. no concurrency
+	virtual void UpdateAttributesOffline ( VecTraits_T<PostponedUpdate_t>& dPostUpdates ) {}
+
 	inline void ResetPostponedUpdates()
 	{
 		m_bAttrsBusy = false;
 		m_dPostponedUpdates.Reset();
 	}
+
+public:
+	// stuff for dispatch races between changes and updates
+	mutable std::atomic<bool>		m_bAttrsBusy { false };
+	CSphVector<PostponedUpdate_t>	m_dPostponedUpdates;
 };
 
+bool Update_CheckAttributes ( const CSphAttrUpdate& tUpd, const ISphSchema& tSchema, CSphString& sError );
+
 // helper - collects killed documents
 struct KillAccum_t final : public IndexSegment_c
 {
@@ -966,7 +992,7 @@ struct UpdateContext_t
 	const HistogramContainer_c *			m_pHistograms {nullptr};
 	CSphRowitem *							m_pAttrPool {nullptr};
 	BYTE *									m_pBlobPool {nullptr};
-	IndexSegment_c *						m_pSegment {nullptr};
+	int 									m_iStride {0};
 
 	CSphFixedVector<UpdatedAttribute_t>		m_dUpdatedAttrs;	// manipulation schema (1 item per column of schema)
 
@@ -977,36 +1003,12 @@ struct UpdateContext_t
 
 
 	UpdateContext_t ( AttrUpdateInc_t & tUpd, const ISphSchema & tSchema );
-};
-
 
-// common attribute update code for both RT and plain indexes
-class IndexUpdateHelper_c
-{
-protected:
-	enum
-	{
-		ATTRS_UPDATED			= ( 1UL<<0 ),
-		ATTRS_BLOB_UPDATED		= ( 1UL<<1 ),
-		ATTRS_ROWMAP_UPDATED	= ( 1UL<<2 )
-	};
-
-	virtual				~IndexUpdateHelper_c() {}
-
-	virtual bool		Update_WriteBlobRow ( UpdateContext_t & tCtx, CSphRowitem * pDocinfo, const BYTE * pBlob,
-			int iLength, int nBlobAttrs, const CSphAttrLocator & tBlobRowLoc, bool & bCritical, CSphString & sError ) = 0;
-
-	static void			Update_PrepareListOfUpdatedAttributes ( UpdateContext_t & tCtx, CSphString & sError );
-	static bool			Update_InplaceJson ( const RowsToUpdate_t& dRows, UpdateContext_t & tCtx, CSphString & sError, bool bDryRun );
-	bool				Update_Blobs ( const RowsToUpdate_t& dRows, UpdateContext_t & tCtx, bool & bCritical, CSphString & sError );
-	static void			Update_Plain ( const RowsToUpdate_t& dRows, UpdateContext_t & tCtx );
-	static bool			Update_HandleJsonWarnings ( UpdateContext_t & tCtx, int iUpdated, CSphString & sWarning, CSphString & sError );
-
-public:
-	static bool			Update_CheckAttributes ( const CSphAttrUpdate & tUpd, const ISphSchema & tSchema, CSphString & sError );
+	void PrepareListOfUpdatedAttributes ( CSphString& sError );
+	bool HandleJsonWarnings ( int iUpdated, CSphString& sWarning, CSphString& sError ) const;
+	CSphRowitem* GetDocinfo ( RowID_t tRowID ) const;
 };
 
-
 class DocstoreAddField_i;
 void SetupDocstoreFields ( DocstoreAddField_i & tFields, const CSphSchema & tSchema );
 bool CheckStoredFields ( const CSphSchema & tSchema, const CSphIndexSettings & tSettings, CSphString & sError );
@@ -1046,6 +1048,7 @@ class CSphSource;
 struct CSphSourceStats;
 class DebugCheckError_i;
 struct AttrAddRemoveCtx_t;
+class Docstore_i;
 
 namespace SI
 {
@@ -1150,12 +1153,10 @@ public:
 	/// returns non-negative amount of actually found and updated records on success
 	/// on failure, -1 is returned and GetLastError() contains error message
 	int							UpdateAttributes ( AttrUpdateSharedPtr_t pUpd, bool & bCritical, CSphString & sError, CSphString & sWarning );
+	int							UpdateAttributes ( AttrUpdateInc_t & tUpd, bool & bCritical, CSphString & sError, CSphString & sWarning );
 
 	/// update accumulating state
-	virtual int					UpdateAttributes ( AttrUpdateInc_t & tUpd, bool & bCritical, CSphString & sError, CSphString & sWarning ) = 0;
-
-	/// apply serie of updates, assuming them prepared (no need to full-scan attributes), and index is offline, i.e. no concurrency
-	virtual void				UpdateAttributesOffline ( VecTraits_T<PostponedUpdate_t> & dUpdates, IndexSegment_c * pSeg ) = 0;
+	virtual int					CheckThenUpdateAttributes ( AttrUpdateInc_t& tUpd, bool& bCritical, CSphString& sError, CSphString& sWarning, BlockerFn&& /*fnWatcher*/ ) = 0;
 
 	virtual Binlog::CheckTnxResult_t ReplayTxn ( Binlog::Blop_e eOp, CSphReader & tReader, CSphString & sError, Binlog::CheckTxn_fn&& fnCheck ) = 0;
 	/// saves memory-cached attributes, if there were any updates to them
@@ -1218,6 +1219,12 @@ public:
 	virtual HistogramContainer_c * Debug_GetHistograms() const { return nullptr; }
 	virtual SI::Index_i *		Debug_GetSI() const { return nullptr; }
 
+	virtual Docstore_i *			GetDocstore() const { return nullptr; }
+	virtual columnar::Columnar_i *	GetColumnar() const { return nullptr; }
+	virtual const DWORD *			GetRawAttrs() const { return nullptr; }
+	virtual const BYTE *			GetRawBlobAttrs() const { return nullptr; }
+	virtual bool					AlterSI ( CSphString & sError ) { return true; }
+
 public:
 	int64_t						m_iTID = 0;				///< last committed transaction id
 	int							m_iChunk = 0;
@@ -1264,6 +1271,7 @@ protected:
 };
 
 const CSphSourceStats& GetStubStats();
+bool CheckDocsCount ( int64_t iDocs, CSphString & sError );
 
 // dummy implementation which makes most of the things optional (makes all non-disk idxes much simpler)
 class CSphIndexStub : public CSphIndex
@@ -1286,8 +1294,7 @@ public:
 	void				GetStatus ( CSphIndexStatus* ) const override {}
 	bool				GetKeywords ( CSphVector <CSphKeywordInfo> & , const char * , const GetKeywordsSettings_t & tSettings, CSphString * ) const override { return false; }
 	bool				FillKeywords ( CSphVector <CSphKeywordInfo> & ) const override { return true; }
-	int					UpdateAttributes ( AttrUpdateInc_t&, bool &, CSphString & , CSphString & ) override { return -1; }
-	void				UpdateAttributesOffline ( VecTraits_T<PostponedUpdate_t> & dUpdates, IndexSegment_c * pSeg ) override {}
+	int					CheckThenUpdateAttributes ( AttrUpdateInc_t&, bool &, CSphString & , CSphString &, BlockerFn&& ) override { return -1; }
 	Binlog::CheckTnxResult_t ReplayTxn ( Binlog::Blop_e, CSphReader &, CSphString &, Binlog::CheckTxn_fn&& ) override { return {}; }
 	bool				SaveAttributes ( CSphString & ) const override { return true; }
 	DWORD				GetAttributeStatus () const override { return 0; }

src/sphinxint.h

@@ -1416,8 +1416,8 @@ BYTE PrereadMapping ( const char * sIndexName, const char * sFor, bool bMlock, b
 	if ( bOnDisk || tBuf.IsEmpty() )
 		return g_uHash;
 
-	const BYTE * pCur = (BYTE *)tBuf.GetWritePtr();
-	const BYTE * pEnd = (BYTE *)tBuf.GetWritePtr() + tBuf.GetLengthBytes();
+	auto pCur = (const BYTE*)tBuf.GetReadPtr();
+	const BYTE * pEnd = pCur + tBuf.GetLengthBytes();
 	const int iHalfPage = 2048;
 
 	g_uHash = 0xff;

src/sphinxql.l

@@ -140,7 +140,7 @@ FLOAT_CONSTANT      {INT}\.{INT}?{EXP}?|{INT}?\.{INT}{EXP}|{INT}{EXP}
 "LEVEL"				{ YYSTOREBOUNDS; return TOK_LEVEL; }
 "LIKE"				{ YYSTOREBOUNDS; return TOK_LIKE; }
 "LOGS"				{ YYSTOREBOUNDS; return TOK_LOGS; }
-"LOCK"				{ YYSTOREBOUNDS; return TOK_LOCK; }
+"FREEZE"			{ YYSTOREBOUNDS; return TOK_FREEZE; }
 "LIMIT"				{ YYSTOREBOUNDS; return TOK_LIMIT; }
 "MATCH"				{ YYSTOREBOUNDS; return TOK_MATCH; }
 "MAX"				{ YYSTOREBOUNDS; return TOK_MAX; }
@@ -191,7 +191,7 @@ FLOAT_CONSTANT      {INT}\.{INT}?{EXP}?|{INT}?\.{INT}{EXP}|{INT}{EXP}
 "TRUE"				{ YYSTOREBOUNDS; return TOK_TRUE; }
 "TRUNCATE"			{ YYSTOREBOUNDS; return TOK_TRUNCATE; }
 "UNCOMMITTED"		{ YYSTOREBOUNDS; return TOK_UNCOMMITTED; }
-"UNLOCK"    		{ YYSTOREBOUNDS; return TOK_UNLOCK; }
+"UNFREEZE"    		{ YYSTOREBOUNDS; return TOK_UNFREEZE; }
 "UPDATE"			{ YYSTOREBOUNDS; return TOK_UPDATE; }
 "USE"				{ YYSTOREBOUNDS; return TOK_USE; }
 "VALUES"			{ YYSTOREBOUNDS; return TOK_VALUES; }

src/sphinxql.y

@@ -65,6 +65,7 @@
 %token	TOK_FOR
 %token	TOK_FORCE
 %token	TOK_FROM
+%token	TOK_FREEZE
 %token	TOK_GLOBAL
 %token	TOK_GROUP
 %token	TOK_GROUPBY
@@ -86,7 +87,6 @@
 %token	TOK_LEVEL
 %token	TOK_LIKE
 %token	TOK_LIMIT
-%token	TOK_LOCK
 %token	TOK_LOGS
 %token	TOK_MATCH
 %token	TOK_MAX
@@ -108,6 +108,7 @@
 %token	TOK_RAND
 %token	TOK_RAMCHUNK
 %token	TOK_READ
+%token	TOK_REBUILD
 %token	TOK_REGEX
 %token	TOK_RECONFIGURE
 %token	TOK_RELOAD
@@ -116,6 +117,7 @@
 %token	TOK_REMAP
 %token	TOK_ROLLBACK
 %token	TOK_RTINDEX
+%token	TOK_SECONDARY
 %token	TOK_SELECT
 %token	TOK_SERIALIZABLE
 %token	TOK_SET
@@ -136,7 +138,7 @@
 %token	TOK_TRUE
 %token	TOK_TRUNCATE
 %token	TOK_UNCOMMITTED
-%token	TOK_UNLOCK
+%token	TOK_UNFREEZE
 %token	TOK_UPDATE
 %token	TOK_USE
 %token	TOK_VALUES
@@ -221,8 +223,8 @@ statement:
 	| delete_cluster
 	| explain_query
 	| TOK_DDLCLAUSE	{ pParser->m_bGotDDLClause = true; }
-	| lock_indexes
-	| unlock_indexes
+	| freeze_indexes
+	| unfreeze_indexes
 	| kill_connid
 	;
 
@@ -251,17 +253,17 @@ reserved_no_option:
 	| TOK_CHARACTER | TOK_CHUNK | TOK_CLUSTER | TOK_COLLATION | TOK_COLUMN | TOK_COMMIT
 	| TOK_COMMITTED | TOK_COUNT | TOK_CREATE | TOK_DATABASES | TOK_DELETE
 	| TOK_DESC | TOK_DESCRIBE  | TOK_DOUBLE
-	| TOK_FLOAT | TOK_FLUSH | TOK_FOR| TOK_GLOBAL | TOK_GROUP
+	| TOK_FLOAT | TOK_FLUSH | TOK_FOR | TOK_FREEZE | TOK_GLOBAL | TOK_GROUP
 	| TOK_GROUP_CONCAT | TOK_GROUPBY | TOK_HAVING | TOK_HOSTNAMES | TOK_INDEX | TOK_INDEXOF | TOK_INSERT
 	| TOK_INT | TOK_INTEGER | TOK_INTO | TOK_ISOLATION | TOK_LEVEL
-	| TOK_LIKE | TOK_LOCK | TOK_LOGS | TOK_MATCH | TOK_MAX | TOK_META | TOK_MIN | TOK_MULTI
+	| TOK_LIKE | TOK_LOGS | TOK_MATCH | TOK_MAX | TOK_META | TOK_MIN | TOK_MULTI
 	| TOK_MULTI64 | TOK_OPTIMIZE | TOK_PLAN
-	| TOK_PLUGINS | TOK_PROFILE | TOK_RAMCHUNK | TOK_RAND | TOK_READ
+	| TOK_PLUGINS | TOK_PROFILE | TOK_RAMCHUNK | TOK_RAND | TOK_READ | TOK_REBUILD
 	| TOK_RECONFIGURE | TOK_REMAP | TOK_REPEATABLE | TOK_REPLACE
-	| TOK_ROLLBACK | TOK_RTINDEX | TOK_SERIALIZABLE | TOK_SESSION | TOK_SET
+	| TOK_ROLLBACK | TOK_RTINDEX | TOK_SECONDARY | TOK_SERIALIZABLE | TOK_SESSION | TOK_SET
 	| TOK_SETTINGS | TOK_SHOW | TOK_SONAME | TOK_START | TOK_STATUS | TOK_STRING
 	| TOK_SUM | TOK_TABLE | TOK_TABLES | TOK_THREADS | TOK_TO | TOK_TRUNCATE
-	| TOK_UNCOMMITTED | TOK_UNLOCK | TOK_UPDATE | TOK_VALUES | TOK_VARIABLES
+	| TOK_UNCOMMITTED | TOK_UNFREEZE | TOK_UPDATE | TOK_VALUES | TOK_VARIABLES
 	| TOK_WARNINGS | TOK_WEIGHT | TOK_WHERE | TOK_WITH | TOK_WITHIN | TOK_KILL
 	;
 
@@ -2012,17 +2014,17 @@ explain_query:
 			}
 	;
 
-lock_indexes:
-	TOK_LOCK one_or_more_indexes
+freeze_indexes:
+	TOK_FREEZE one_or_more_indexes
 		{
-			pParser->m_pStmt->m_eStmt = STMT_LOCK;
+			pParser->m_pStmt->m_eStmt = STMT_FREEZE;
 		}
 	;
 
-unlock_indexes:
-	TOK_UNLOCK one_or_more_indexes
+unfreeze_indexes:
+	TOK_UNFREEZE one_or_more_indexes
 		{
-			pParser->m_pStmt->m_eStmt = STMT_UNLOCK;
+			pParser->m_pStmt->m_eStmt = STMT_UNFREEZE;
 		}
 	;
 

src/sphinxrt.cpp

@@ -243,8 +243,9 @@ SphAttr_t InsertDocData_t::GetID() const
 //////////////////////////////////////////////////////////////////////////
 
 
-RtSegment_t::RtSegment_t ( DWORD uDocs )
+RtSegment_t::RtSegment_t ( DWORD uDocs, const ISphSchema& tSchema )
 	: m_tDeadRowMap ( uDocs )
+	, m_tSchema { tSchema }
 {
 }
 
@@ -316,6 +317,14 @@ const CSphRowitem * RtSegment_t::GetDocinfoByRowID ( RowID_t tRowID ) const NO_T
 	return m_dRows.GetLength() ? &m_dRows[tRowID*GetStride()] : nullptr;
 }
 
+RowID_t RtSegment_t::GetAliveRowidByDocid ( DocID_t tDocID ) const
+{
+	RowID_t* pRowID = m_tDocIDtoRowID.Find ( tDocID );
+	if ( !pRowID || m_tDeadRowMap.IsSet ( *pRowID ) )
+		return INVALID_ROWID;
+	return *pRowID;
+}
+
 
 RowID_t RtSegment_t::GetRowidByDocid ( DocID_t tDocID ) const
 {
@@ -1056,11 +1065,47 @@ CSphVector<int> GetChunkIds ( const VecTraits_T<DiskChunkRefPtr_t> & dChunks )
 	return dIds;
 }
 
-enum class WriteState_e : int
+class SaveState_c
 {
-	ENABLED,	// normal
-	DISCARD,	// disabled, current result will not be necessary (can escape to don't waste resources)
-	DISABLED,	// disabled, current stage must be completed first
+public:
+
+	enum States_e : BYTE {
+		ENABLED,	// normal
+		DISCARD,	// disabled, current result will not be necessary (can escape to don't waste resources)
+		DISABLED,	// disabled, current stage must be completed first
+	};
+
+	explicit SaveState_c ( States_e eValue )
+		: m_tValue { eValue, false } {}
+
+	void SetState ( States_e eState )
+	{
+		m_tValue.ModifyValueAndNotifyAll ( [eState] ( Value_t& t ) { t.m_eValue = eState; } );
+	}
+	void SetShutdownFlag ()
+	{
+		m_tValue.ModifyValueAndNotifyAll ( [] ( Value_t& t ) { t.m_bShutdown = true; } );
+	}
+
+	bool Is (States_e eValue) const { return m_tValue.GetValue().m_eValue==eValue; }
+
+	// sleep and return true when expected state achieved.
+	// sleep and return false if shutdown expected
+	bool WaitStateOrShutdown ( States_e uState ) const
+	{
+		return uState == m_tValue.Wait ( [uState] ( const Value_t& tVal ) { return tVal.m_bShutdown || ( tVal.m_eValue == uState ); } ).m_eValue;
+	}
+private:
+	struct Value_t
+	{
+		States_e m_eValue;
+		bool m_bShutdown;
+		Value_t ( States_e eValue, bool bShutdown )
+			: m_eValue { eValue }
+			, m_bShutdown { bShutdown }
+		{}
+	};
+	Coro::Waitable_T<Value_t> m_tValue;
 };
 
 enum class MergeSeg_e : BYTE
@@ -1071,7 +1116,7 @@ enum class MergeSeg_e : BYTE
 	EXIT 	= 4,	// shutdown and exit
 };
 
-class RtIndex_c final : public RtIndex_i, public ISphNoncopyable, public ISphWordlist, public ISphWordlistSuggest, public IndexUpdateHelper_c, public IndexAlterHelper_c, public DebugCheckHelper_c
+class RtIndex_c final : public RtIndex_i, public ISphNoncopyable, public ISphWordlist, public ISphWordlistSuggest, public IndexAlterHelper_c, public DebugCheckHelper_c
 {
 public:
 						RtIndex_c ( const CSphSchema & tSchema, const char * sIndexName, int64_t iRamSize, const char * sPath, bool bKeywordDict );
@@ -1082,7 +1127,7 @@ public:
 	bool				DeleteDocument ( const VecTraits_T<DocID_t> & dDocs, CSphString & sError, RtAccum_t * pAccExt ) final;
 	bool				Commit ( int * pDeleted, RtAccum_t * pAccExt, CSphString* pError = nullptr ) final;
 	void				RollBack ( RtAccum_t * pAccExt ) final;
-	int					CommitReplayable ( RtSegment_t * pNewSeg, const CSphVector<DocID_t> & dAccKlist ); // returns total killed documents
+	int					CommitReplayable ( RtSegment_t * pNewSeg, const VecTraits_T<DocID_t> & dAccKlist ); // returns total killed documents
 	void				ForceRamFlush ( const char * szReason ) final;
 	bool				IsFlushNeed() const final;
 	bool				ForceDiskChunk() final;
@@ -1137,7 +1182,7 @@ public:
 	void				PostSetup() final;
 	bool				IsRT() const final { return true; }
 
-	int					UpdateAttributes ( AttrUpdateInc_t & tUpd, bool & bCritical, CSphString & sError, CSphString & sWarning ) final;
+	int					CheckThenUpdateAttributes ( AttrUpdateInc_t & tUpd, bool & bCritical, CSphString & sError, CSphString & sWarning, BlockerFn&& ) final;
 	bool				SaveAttributes ( CSphString & sError ) const final;
 	DWORD				GetAttributeStatus () const final { return m_uDiskAttrStatus; }
 
@@ -1200,7 +1245,7 @@ private:
 	std::atomic<int64_t>		m_iRamChunksAllocatedRAM { 0 };
 
 	std::atomic<bool>			m_bOptimizeStop { false };
-	Threads::Coro::Waitable_T<int>	m_tOptimizeRuns {0};
+	Coro::Waitable_T<int>		m_tOptimizeRuns {0};
 	friend class OptimizeGuard_c;
 
 	int64_t						m_iRtMemLimit;
@@ -1231,7 +1276,7 @@ private:
 	int							m_iMaxCodepointLength = 0;
 	TokenizerRefPtr_c			m_pTokenizerIndexing;
 	bool						m_bPreallocPassedOk = true;
-	std::atomic<WriteState_e>	m_eSaving { WriteState_e::ENABLED };
+	SaveState_c					m_tSaving { SaveState_c::ENABLED };
 	bool						m_bHasFiles = false;
 
 	// fixme! make this *Lens atomic together with disk/ram data, to avoid any kind of race among them
@@ -1253,9 +1298,6 @@ private:
 	void						MergeKeywords ( RtSegment_t & tSeg, const RtSegment_t & tSeg1, const RtSegment_t & tSeg2, const VecTraits_T<RowID_t> & dRowMap1, const VecTraits_T<RowID_t> & dRowMap2 ) const;
 	RtSegment_t *				MergeTwoSegments ( const RtSegment_t * pA, const RtSegment_t * pB ) const;
 	static void					CopyWord ( RtSegment_t& tDstSeg, RtWord_t& tDstWord, RtDocWriter_c& tDstDoc, const RtSegment_t& tSrcSeg, const RtWord_t* pSrcWord, const VecTraits_T<RowID_t>& dRowMap );
-	void						UpdateAttributesOffline ( VecTraits_T<PostponedUpdate_t>& dUpdates, IndexSegment_c * pSeg ) override;
-
-	bool						UpdateAttributesInRamSegment ( const RowsToUpdate_t& dRows, UpdateContext_t& tCtx, bool& bCritical, CSphString& sError );
 
 	void						DeleteFieldFromDict ( RtSegment_t * pSeg, int iKillField );
 	void						AddFieldToRamchunk ( const CSphString & sFieldName, DWORD uFieldFlags, const CSphSchema & tOldSchema, const CSphSchema & tNewSchema );
@@ -1289,13 +1331,12 @@ private:
 	bool						ReadNextWord ( SuggestResult_t & tRes, DictWord_t & tWord ) const final;
 
 	ConstRtSegmentRefPtf_t		AdoptSegment ( RtSegment_t * pNewSeg );
-	int							ApplyKillList ( const CSphVector<DocID_t> & dAccKlist ) REQUIRES ( m_tWorkers.SerialChunkAccess() );
+	int							ApplyKillList ( const VecTraits_T<DocID_t> & dAccKlist ) REQUIRES ( m_tWorkers.SerialChunkAccess() );
 
 	bool						AddRemoveColumnarAttr ( RtGuard_t & tGuard, bool bAdd, const CSphString & sAttrName, ESphAttr eAttrType, const CSphSchema & tOldSchema, const CSphSchema & tNewSchema, CSphString & sError );
 	void						AddRemoveRowwiseAttr ( RtGuard_t & tGuard, bool bAdd, const CSphString & sAttrName, ESphAttr eAttrType, const CSphSchema & tOldSchema, const CSphSchema & tNewSchema, CSphString & sError );
 
-	bool						Update_WriteBlobRow ( UpdateContext_t& tCtx, CSphRowitem* pDocinfo, const BYTE* pBlob, int iLength, int nBlobAttrs, const CSphAttrLocator& tBlobRowLoc, bool& bCritical, CSphString& sError ) override;
-	bool						Update_DiskChunks ( AttrUpdateInc_t& tUpd, const DiskChunkSlice_t& dDiskChunks, CSphString& sError );
+	bool						Update_DiskChunks ( AttrUpdateInc_t& tUpd, const DiskChunkSlice_t& dDiskChunks, CSphString& sError ) REQUIRES ( m_tWorkers.SerialChunkAccess() );
 
 	void						GetIndexFiles ( StrVec_t& dFiles, StrVec_t& dExt, const FilenameBuilder_i* = nullptr ) const override;
 	DocstoreBuilder_i::Doc_t *	FetchDocFields ( DocstoreBuilder_i::Doc_t & tStoredDoc, const InsertDocData_t & tDoc, CSphSource_StringVector & tSrc, CSphVector<CSphVector<BYTE>> & dTmpAttrStorage ) const;
@@ -1321,7 +1362,7 @@ private:
 	void						SetMemLimit ( int64_t iMemLimit );
 	void						RecalculateRateLimit ( int64_t iSaved, int64_t iInserted, bool bEmergent );
 	void						AlterSave ( bool bSaveRam );
-	void 						BinlogCommit ( RtSegment_t * pSeg, const CSphVector<DocID_t> & dKlist );
+	void 						BinlogCommit ( RtSegment_t * pSeg, const VecTraits_T<DocID_t> & dKlist );
 	bool						StopOptimize();
 	void						UpdateUnlockedCount();
 	bool						CheckSegmentConsistency ( const RtSegment_t* pNewSeg, bool bSilent=true ) const;
@@ -1350,6 +1391,7 @@ private:
 	// Manage alter state
 	void						RaiseAlterGeneration();
 	int							GetAlterGeneration() const override;
+	bool						AlterSI ( CSphString & sError ) override;
 };
 
 
@@ -1382,6 +1424,8 @@ RtIndex_c::~RtIndex_c ()
 		// From serial worker resuming on Wait() will happen after whole merger coroutine finished.
 		ScopedScheduler_c tSerialFiber { m_tWorkers.SerialChunkAccess() };
 		TRACE_SCHED ( "rt", "~RtIndex_c" );
+		m_tSaving.SetShutdownFlag ();
+		Threads::Coro::Reschedule();
 		StopMergeSegmentsWorker();
 		m_tNSavesNow.Wait ( [] ( int iVal ) { return iVal==0; } );
 	}
@@ -1390,10 +1434,10 @@ RtIndex_c::~RtIndex_c ()
 	bool bValid = m_pTokenizer && m_pDict && m_bPreallocPassedOk;
 
 	if ( bValid )
-	{
-		SaveRamChunk();
+		bValid &= SaveRamChunk();
+
+	if ( bValid )
 		SaveMeta();
-	}
 
 	if ( m_iLockFD>=0 )
 		::close ( m_iLockFD );
@@ -1411,15 +1455,14 @@ RtIndex_c::~RtIndex_c ()
 		sFile.SetSprintf ( "%s%s", m_sPath.cstr(), sphGetExt ( SPH_EXT_SETTINGS ) );
 		::unlink ( sFile.cstr() );
 	}
+	if ( !bValid )
+		return;
 
 	tmSave = sphMicroTimer() - tmSave;
-	if ( tmSave>=1000 && bValid )
-	{
-		sphInfo ( "rt: index %s: ramchunk saved in %d.%03d sec",
-			m_sIndexName.cstr(), (int)(tmSave/1000000), (int)((tmSave/1000)%1000) );
-	}
+	if ( tmSave>=1000 )
+		sphInfo ( "rt: index %s: ramchunk saved in %d.%03d sec", m_sIndexName.cstr(), (int)(tmSave/1000000), (int)((tmSave/1000)%1000) );
 
-	if ( !sphInterrupted() && bValid )
+	if ( !sphInterrupted() )
 		sphLogDebug ( "closed index %s, valid %d, deleted %d, time %d.%03d sec", m_sIndexName.cstr(), (int)bValid, (int)m_bIndexDeleted, (int)(tmSave/1000000), (int)((tmSave/1000)%1000) );
 }
 
@@ -1435,8 +1478,10 @@ int RtIndex_c::GetAlterGeneration() const
 
 void RtIndex_c::UpdateUnlockedCount()
 {
-	if ( !m_bDebugCheck )
-		m_tUnLockedSegments.UpdateValueAndNotifyAll ( (int)m_tRtChunks.RamSegs()->count_of ( [] ( auto& dSeg ) { return !dSeg->m_iLocked; } ) );
+	if ( m_bDebugCheck )
+		return;
+
+	m_tUnLockedSegments.UpdateValueAndNotifyAll ( (int)m_tRtChunks.RamSegs()->count_of ( [] ( auto& dSeg ) { return !dSeg->m_iLocked; } ) );
 }
 
 void RtIndex_c::ProcessDiskChunk ( int iChunk, VisitChunk_fn&& fnVisitor ) const
@@ -1479,7 +1524,7 @@ bool RtIndex_c::IsFlushNeed() const
 	if ( Binlog::IsActive() && m_iTID<=m_iSavedTID )
 		return false;
 
-	return m_eSaving.load(std::memory_order_relaxed)==WriteState_e::ENABLED;
+	return m_tSaving.Is ( SaveState_c::ENABLED );
 }
 
 static int64_t SegmentsGetUsedRam ( const ConstRtSegmentSlice_t& dSegments )
@@ -2009,7 +2054,7 @@ RtSegment_t * CreateSegment ( RtAccum_t* pAcc, int iWordsCheckpoint, ESphHitless
 		return nullptr;
 
 	MEMORY ( MEM_RT_ACCUM );
-	auto * pSeg = new RtSegment_t ( pAcc->m_uAccumDocs );
+	auto * pSeg = new RtSegment_t ( pAcc->m_uAccumDocs, pAcc->m_pIndex->GetInternalSchema() );
 	FakeWL_t tFakeLock {pSeg->m_tLock};
 	CreateSegmentHits ( *pAcc, pSeg, iWordsCheckpoint, eHitless, dHitlessWords );
 
@@ -2457,7 +2502,7 @@ RtSegment_t* RtIndex_c::MergeTwoSegments ( const RtSegment_t* pA, const RtSegmen
 	auto pColumnarBuilder = CreateColumnarBuilderRT(m_tSchema);
 	RtAttrMergeContext_t tCtx ( nBlobAttrs, tNextRowID, pColumnarBuilder.get() );
 
-	auto * pSeg = new RtSegment_t (0);
+	auto * pSeg = new RtSegment_t (0, m_tSchema);
 	FakeWL_t _ { pSeg->m_tLock }; // as pSeg is just created - we don't need real guarding and use fake lock to mute thread safety warnings
 
 	assert ( !!pA->m_pDocstore==!!pB->m_pDocstore );
@@ -2538,12 +2583,12 @@ namespace GatherUpdates {
 	const VecTraits_T<PostponedUpdate_t>& AccessPostponedUpdates ( const ConstDiskChunkRefPtr_t& pChunk ) { return pChunk->Cidx ().m_dPostponedUpdates; }
 
 	template<typename CHUNK_OR_SEG>
-	CSphVector<PostponedUpdate_t> FromChunksOrSegments ( VecTraits_T<CHUNK_OR_SEG> dChunksOrSegmengs )
+	CSphVector<PostponedUpdate_t> FromChunksOrSegments ( VecTraits_T<CHUNK_OR_SEG> dChunksOrSegments )
 	{
 		CSphVector<PostponedUpdate_t> dResult;
 		CSphVector<HashedUpd_t> dUpdates;
 
-		for ( const auto& dSeg : dChunksOrSegmengs )
+		for ( const auto& dSeg : dChunksOrSegments )
 		{
 			const VecTraits_T<PostponedUpdate_t>& dPostponedUpdates = AccessPostponedUpdates (dSeg);
 			if ( dPostponedUpdates.IsEmpty () )
@@ -2574,59 +2619,37 @@ namespace GatherUpdates {
 }; // namespace
 
 
-bool RtIndex_c::UpdateAttributesInRamSegment ( const RowsToUpdate_t& dRows, UpdateContext_t& tCtx, bool& bCritical, CSphString& sError ) REQUIRES ( m_tWorkers.SerialChunkAccess() )
-{
-	TRACE_SCHED ( "rt", "UpdateAttributesInRamSegment" );
-	if ( tCtx.m_tUpd.m_pUpdate->m_bStrict )
-		if ( !Update_InplaceJson ( dRows, tCtx, sError, true ) )
-			return false;
-
-	// second pass
-	int iSaveWarnings = tCtx.m_iJsonWarnings;
-	tCtx.m_iJsonWarnings = 0;
-	Update_InplaceJson ( dRows, tCtx, sError, false );
-	tCtx.m_iJsonWarnings += iSaveWarnings;
-
-	if ( !Update_Blobs ( dRows, tCtx, bCritical, sError ) )
-		return false;
-
-	Update_Plain ( dRows, tCtx );
-	return true;
-}
-
 // that is 2-nd part of postponed updates. We may have one or several update set, stored from old segments.
-void RtIndex_c::UpdateAttributesOffline ( VecTraits_T<PostponedUpdate_t> & dUpdates, IndexSegment_c * pSeg ) NO_THREAD_SAFETY_ANALYSIS
+void RtSegment_t::UpdateAttributesOffline ( VecTraits_T<PostponedUpdate_t>& dPostUpdates ) NO_THREAD_SAFETY_ANALYSIS
 {
-	if ( dUpdates.IsEmpty() )
+	if ( dPostUpdates.IsEmpty() )
 		return;
 
-	assert ( pSeg && "for RT index UpdateAttributesOffline should be called only with non-null pSeg!" );
-	auto * pSegment = (RtSegment_t *) pSeg;
-
 	CSphString sError;
 	bool bCritical;
-	for ( auto & tUpdate : dUpdates )
+
+	assert ( GetStride() == m_tSchema.GetRowSize() );
+	for ( auto & tPostUpdate : dPostUpdates )
 	{
-		AttrUpdateInc_t tUpdInc { std::move ( tUpdate.m_pUpdate ) };
+		AttrUpdateInc_t tUpdInc { std::move ( tPostUpdate.m_pUpdate ) };
 		UpdateContext_t tCtx ( tUpdInc, m_tSchema );
-		Update_PrepareListOfUpdatedAttributes ( tCtx, sError );
+		tCtx.PrepareListOfUpdatedAttributes ( sError );
 
 		// actualize list of updates in context of new segment
 		const auto & dDocids = tUpdInc.m_pUpdate->m_dDocids;
-		ARRAY_FOREACH ( i, tUpdate.m_dRowsToUpdate )
+		ARRAY_FOREACH ( i, tPostUpdate.m_dRowsToUpdate )
 		{
-			auto& tRow = tUpdate.m_dRowsToUpdate[i];
-			auto pRow = pSegment->FindAliveRow ( dDocids[tRow.m_iIdx] );
-			if ( pRow )
-				tRow.m_pRow = pRow;
+			auto& tRow = tPostUpdate.m_dRowsToUpdate[i];
+			auto tRowID = GetAliveRowidByDocid ( dDocids[tRow.m_iIdx] );
+			if ( tRowID==INVALID_ROWID )
+				tPostUpdate.m_dRowsToUpdate.RemoveFast ( i-- );
 			else
-				tUpdate.m_dRowsToUpdate.RemoveFast ( i-- );
+				tRow.m_tRow = tRowID;
 		}
 
-		tCtx.m_pAttrPool = pSegment->m_dRows.begin();
-		tCtx.m_pBlobPool = pSegment->m_dBlobs.begin();
-		tCtx.m_pSegment = pSeg;
-		UpdateAttributesInRamSegment ( tUpdate.m_dRowsToUpdate, tCtx, bCritical, sError );
+		tCtx.m_pAttrPool = m_dRows.begin();
+		tCtx.m_pBlobPool = m_dBlobs.begin();
+		Update_UpdateAttributes ( tPostUpdate.m_dRowsToUpdate, tCtx, bCritical, sError );
 	}
 }
 
@@ -2744,7 +2767,7 @@ ConstRtSegmentRefPtf_t RtIndex_c::AdoptSegment ( RtSegment_t * pNewSeg )
 
 // CommitReplayable -> ApplyKillList
 // AttachDiskIndex -> ApplyKillList
-int RtIndex_c::ApplyKillList ( const CSphVector<DocID_t> & dAccKlist )
+int RtIndex_c::ApplyKillList ( const VecTraits_T<DocID_t> & dAccKlist )
 {
 	if ( dAccKlist.IsEmpty() )
 		return 0;
@@ -2754,8 +2777,27 @@ int RtIndex_c::ApplyKillList ( const CSphVector<DocID_t> & dAccKlist )
 
 	int iKilled = 0;
 	auto pChunks = m_tRtChunks.DiskChunks();
-	for ( auto& pChunk : *pChunks )
-		iKilled += pChunk->CastIdx().KillMulti ( dAccKlist );
+
+	if ( m_tSaving.Is ( SaveState_c::ENABLED ) )
+		for ( auto& pChunk : *pChunks )
+			iKilled += pChunk->CastIdx().KillMulti ( dAccKlist );
+	else
+	{
+		// if saving is disabled, and we NEED to actually mark a doc in disk chunk as deleted,
+		// we'll pause that action, waiting until index is unlocked.
+		bool bNeedWait = true;
+		bool bEnabled = false;
+		for ( auto& pChunk : *pChunks )
+			iKilled += pChunk->CastIdx().CheckThenKillMulti ( dAccKlist, [this,&bNeedWait, &bEnabled]()
+			{
+				if ( bNeedWait )
+				{
+					bNeedWait = false;
+					bEnabled = m_tSaving.WaitStateOrShutdown ( SaveState_c::ENABLED );
+				}
+				return bEnabled;
+			});
+	}
 
 	auto pSegs = m_tRtChunks.RamSegs();
 	for ( auto& pSeg : *pSegs )
@@ -2969,7 +3011,7 @@ bool RtIndex_c::MergeSegmentsStep ( MergeSeg_e eVal ) REQUIRES ( m_tWorkers.Seri
 			dOld.Add ( pA );
 			dOld.Add ( pB );
 			auto dUpdates = GatherUpdates::FromChunksOrSegments ( dOld );
-			UpdateAttributesOffline ( dUpdates, pMerged );
+			pMerged->UpdateAttributesOffline ( dUpdates );
 		}
 	}
 
@@ -3041,7 +3083,7 @@ int CommitID() {
 }
 } // namespace
 
-int RtIndex_c::CommitReplayable ( RtSegment_t * pNewSeg, const CSphVector<DocID_t> & dAccKlist ) REQUIRES_SHARED ( pNewSeg->m_tLock )
+int RtIndex_c::CommitReplayable ( RtSegment_t * pNewSeg, const VecTraits_T<DocID_t> & dAccKlist ) REQUIRES_SHARED ( pNewSeg->m_tLock )
 {
 	// store statistics, because pNewSeg just might get merged
 	const int iId = CommitID();
@@ -3181,18 +3223,6 @@ struct SaveDiskDataContext_t : public BuildHeader_t
 };
 
 
-struct CmpDocidLookup_fn
-{
-	static inline bool IsLess ( const DocidRowidPair_t & a, const DocidRowidPair_t & b )
-	{
-		if ( a.m_tDocID==b.m_tDocID )
-			return a.m_tRowID < b.m_tRowID;
-
-		return (uint64_t)a.m_tDocID < (uint64_t)b.m_tDocID;
-	}
-};
-
-
 bool RtIndex_c::WriteAttributes ( SaveDiskDataContext_t & tCtx, CSphString & sError ) const
 {
 	CSphString sSPA, sSPB, sSPT, sSPHI, sSPDS, sSPC, sSIdx;
@@ -3767,7 +3797,7 @@ bool RtIndex_c::SaveDiskHeader ( SaveDiskDataContext_t & tCtx, const ChunkStats_
 
 void RtIndex_c::SaveMeta ( int64_t iTID, VecTraits_T<int> dChunkNames )
 {
-	if ( m_eSaving.load ( std::memory_order_relaxed ) != WriteState_e::ENABLED )
+	if ( !m_tSaving.Is ( SaveState_c::ENABLED ) )
 		return;
 
 	// sanity check
@@ -3886,7 +3916,7 @@ int64_t RtIndex_c::GetMemCount ( PRED&& fnPred ) const
 // i.e. create new disk chunk from ram segments
 bool RtIndex_c::SaveDiskChunk ( bool bForced, bool bEmergent, bool bBootstrap ) REQUIRES ( m_tWorkers.SerialChunkAccess() )
 {
-	if ( m_eSaving.load(std::memory_order_relaxed) != WriteState_e::ENABLED ) // fixme! review, m.b. refactor
+	if ( !m_tSaving.Is ( SaveState_c::ENABLED ) ) // fixme! review, m.b. refactor
 		return !bBootstrap;
 
 	assert ( Coro::CurrentScheduler() == m_tWorkers.SerialChunkAccess() );
@@ -4011,7 +4041,7 @@ bool RtIndex_c::SaveDiskChunk ( bool bForced, bool bEmergent, bool bBootstrap )
 	if ( !dUpdates.IsEmpty () )
 	{
 		RTLOGV << "SaveDiskChunk: apply postponed updates";
-		pNewChunk->UpdateAttributesOffline ( dUpdates, pNewChunk.get() );
+		pNewChunk->UpdateAttributesOffline ( dUpdates );
 		dUpdates.Reset();
 	}
 
@@ -4635,8 +4665,8 @@ void RtIndex_c::SaveRamFieldLengths ( CSphWriter& wrChunk ) const
 
 bool RtIndex_c::SaveRamChunk ()
 {
-	if ( m_eSaving.load ( std::memory_order_relaxed ) != WriteState_e::ENABLED )
-		return true;
+	if ( !m_tSaving.Is ( SaveState_c::ENABLED ) )
+		return false;
 
 	MEMORY ( MEM_INDEX_RT );
 
@@ -4714,7 +4744,7 @@ bool RtIndex_c::LoadRamChunk ( DWORD uVersion, bool bRebuildInfixes, bool bFixup
 	{
 		DWORD uRows = rdChunk.GetDword();
 
-		RtSegmentRefPtf_t pSeg {new RtSegment_t ( uRows )};
+		RtSegmentRefPtf_t pSeg { new RtSegment_t ( uRows, m_tSchema ) };
 		pSeg->m_uRows = uRows;
 		pSeg->m_tAliveRows.store ( rdChunk.GetDword (), std::memory_order_relaxed );
 
@@ -7690,12 +7720,12 @@ CSphFixedVector<RowsToUpdateData_t> Update_CollectRowPtrs ( UpdateContext_t & tC
 		if ( !tCtx.m_tUpd.m_dUpdated.BitGet ( i ) )
 			ARRAY_CONSTFOREACH ( j, tSegments )
 			{
-				auto pRow = tSegments[j]->FindAliveRow ( dDocids[i] );
-				if ( !pRow )
+				auto tRowID = tSegments[j]->GetAliveRowidByDocid( dDocids[i] );
+				if ( tRowID==INVALID_ROWID )
 					continue;
 
 				auto& dUpd = dUpdateSets[j].Add();
-				dUpd.m_pRow = pRow;
+				dUpd.m_tRow = tRowID;
 				dUpd.m_iIdx = i;
 			}
 	return dUpdateSets;
@@ -7708,52 +7738,59 @@ CSphFixedVector<RowsToUpdateData_t> Update_CollectRowPtrs ( UpdateContext_t & tC
 // to final resulting chunk/segment. That bit set before merging attributes and exists till the end of segment's lifetime.
 // Here is first part of postponed merge - after update we collect docs updated in segment and store them into vec of
 // updates (as it might happen be more than one update during the operation)
-static void AddDerivedUpdate ( const RowsToUpdate_t & dRows, const UpdateContext_t & tCtx )
+void RtSegment_t::MaybeAddPostponedUpdate ( const RowsToUpdate_t& dRows, const UpdateContext_t& tCtx )
 {
-	auto & tUpd = tCtx.m_tUpd;
+	if ( !m_bAttrsBusy.load ( std::memory_order_acquire ) )
+		return;
+
+	// segment is now saving/merging - add postponed update.
+	auto& tUpd = tCtx.m_tUpd;
 	// count exact N of affected rows (no need to waste space for reserve in this route at all)
-	auto iRows = dRows.count_of ( [&tUpd] ( auto & i ) { return tUpd.m_dUpdated.BitGet ( i.m_iIdx ); } );
+	auto iRows = dRows.count_of ( [&tUpd] ( auto& i ) { return tUpd.m_dUpdated.BitGet ( i.m_iIdx ); } );
 
 	if ( !iRows )
 		return;
 
-	auto * pSegment = (RtSegment_t *) tCtx.m_pSegment;
-	assert (pSegment);
-
-	auto& tNewDerivedUpdate = pSegment->m_dPostponedUpdates.Add();
-	tNewDerivedUpdate.m_pUpdate = MakeReusableUpdate ( tUpd.m_pUpdate );
-	tNewDerivedUpdate.m_dRowsToUpdate.Reserve (iRows);
+	auto& tNewPostponedUpdate = m_dPostponedUpdates.Add();
+	tNewPostponedUpdate.m_pUpdate = MakeReusableUpdate ( tUpd.m_pUpdate );
+	tNewPostponedUpdate.m_dRowsToUpdate.Reserve ( iRows );
 
 	// collect indexes of actually updated rows and docids
-	dRows.for_each ( [&tUpd, &tNewDerivedUpdate] ( auto & i ) {
+	dRows.for_each ( [&tUpd, &tNewPostponedUpdate] ( const auto& i ) {
 		if ( tUpd.m_dUpdated.BitGet ( i.m_iIdx ) )
-			tNewDerivedUpdate.m_dRowsToUpdate.Add ().m_iIdx = i.m_iIdx;
+			tNewPostponedUpdate.m_dRowsToUpdate.Add().m_iIdx = i.m_iIdx;
 	});
 }
 
-bool RtIndex_c::Update_DiskChunks ( AttrUpdateInc_t& tUpd, const DiskChunkSlice_t& dDiskChunks, CSphString & sError )
+bool RtIndex_c::Update_DiskChunks ( AttrUpdateInc_t& tUpd, const DiskChunkSlice_t& dDiskChunks, CSphString & sError ) REQUIRES ( m_tWorkers.SerialChunkAccess() )
 {
+	assert ( Coro::CurrentScheduler() == m_tWorkers.SerialChunkAccess() );
 	bool bCritical = false;
 	CSphString sWarning;
 
-	// That seems to be only place where order of disk chunks is important.
-	// About deduplication: order of new-born disk chunks is important, as they may contain replaces for older docs.
-	// Since we don't consider killed documents during update, it is important to update from freshest to oldest,
-	// this order ensures that actual documents updated. However, for optimized (merged/splitted) chunks that is not
-	// important as kill-lists applied during merge, and so resulting chunks have only 'freshest' version, because
-	// all previous are effectively excluded during merge pass.
-	// (If we refactor updates to scan rows taking in account kill-lists, order will not be important at all).
-	for ( int iChunk = dDiskChunks.GetLength()-1; iChunk>=0; --iChunk )
+	bool bEnabled = m_tSaving.Is ( SaveState_c::ENABLED );
+	bool bNeedWait = !bEnabled;
+
+	// if saving is disabled, and we NEED to actually update a disk chunk,
+	// we'll pause that action, waiting until index is unlocked.
+	BlockerFn fnBlock = [this, &bNeedWait, &bEnabled]() {
+		if ( bNeedWait )
+		{
+			bNeedWait = false;
+			bEnabled = m_tSaving.WaitStateOrShutdown ( SaveState_c::ENABLED );
+		}
+		return bEnabled;
+	};
+
+	for ( auto& pDiskChunk : dDiskChunks )
 	{
 		if ( tUpd.AllApplied () )
 			break;
 
-		auto& pDiskChunk = dDiskChunks[iChunk];
-
 		// acquire fine-grain lock
 		SccWL_t wLock ( pDiskChunk->m_tLock );
 
-		int iRes = pDiskChunk->CastIdx().UpdateAttributes ( tUpd, bCritical, sError, sWarning );
+		int iRes = pDiskChunk->CastIdx().CheckThenUpdateAttributes ( tUpd, bCritical, sError, sWarning, bNeedWait ? fnBlock : nullptr );
 
 		// FIXME! need to handle critical failures here (chunk is unusable at this point)
 		assert ( !bCritical );
@@ -7769,41 +7806,37 @@ bool RtIndex_c::Update_DiskChunks ( AttrUpdateInc_t& tUpd, const DiskChunkSlice_
 	return true;
 }
 
-// thread-safe, as segment is locked up level before calling UpdateAttributesInRamSegment
-bool RtIndex_c::Update_WriteBlobRow ( UpdateContext_t & tCtx, CSphRowitem * pDocinfo, const BYTE * pBlob, int iLength,
+// thread-safe, as segment is locked up level before calling RAM segment update
+bool RtSegment_t::Update_WriteBlobRow ( UpdateContext_t & tCtx, RowID_t tRowID, ByteBlob_t tBlob,
 		int nBlobAttrs, const CSphAttrLocator & tBlobRowLoc, bool & bCritical, CSphString & sError ) NO_THREAD_SAFETY_ANALYSIS
 {
 	// fixme! Ensure pSegment->m_tLock acquired exclusively...
-	auto* pSegment = (RtSegment_t*)tCtx.m_pSegment;
-	assert ( pSegment );
+	auto pDocinfo = tCtx.GetDocinfo ( tRowID );
+	BYTE* pExistingBlob = m_dBlobs.begin() + sphGetRowAttr ( pDocinfo, tBlobRowLoc );
+	DWORD uExistingBlobLen = sphGetBlobTotalLen ( pExistingBlob, nBlobAttrs );
 
 	bCritical = false;
 
-	CSphTightVector<BYTE> & dBlobPool = pSegment->m_dBlobs;
-
-	BYTE * pExistingBlob = dBlobPool.Begin() + sphGetRowAttr ( pDocinfo, tBlobRowLoc );
-	DWORD uExistingBlobLen = sphGetBlobTotalLen ( pExistingBlob, nBlobAttrs );
-
 	// overwrite old record
-	if ( (DWORD)iLength<=uExistingBlobLen )
+	if ( (DWORD)tBlob.second<=uExistingBlobLen )
 	{
-		memcpy ( pExistingBlob, pBlob, iLength );
+		memcpy ( pExistingBlob, tBlob.first, tBlob.second );
 		return true;
 	}
 
-	int iPoolSize = dBlobPool.GetLength();
-	dBlobPool.Append ( pBlob, iLength );
+	int iPoolSize = m_dBlobs.GetLength();
+	m_dBlobs.Append ( tBlob );
 
 	sphSetRowAttr ( pDocinfo, tBlobRowLoc, iPoolSize );
 
 	// update blob pool ptrs since they could have changed after the resize
-	tCtx.m_pBlobPool = dBlobPool.begin();
+	tCtx.m_pBlobPool = m_dBlobs.begin();
 	return true;
 }
 
 
 // FIXME! might be inconsistent in case disk chunk update fails
-int RtIndex_c::UpdateAttributes ( AttrUpdateInc_t & tUpd, bool & bCritical, CSphString & sError, CSphString & sWarning )
+int RtIndex_c::CheckThenUpdateAttributes ( AttrUpdateInc_t& tUpd, bool& bCritical, CSphString& sError, CSphString& sWarning, BlockerFn&& fnWatcher )
 {
 	const auto& tUpdc = *tUpd.m_pUpdate;
 	assert ( tUpdc.m_dRowOffset.IsEmpty() || tUpdc.m_dDocids.GetLength()==tUpdc.m_dRowOffset.GetLength() );
@@ -7815,19 +7848,12 @@ int RtIndex_c::UpdateAttributes ( AttrUpdateInc_t & tUpd, bool & bCritical, CSph
 	if ( m_tRtChunks.IsEmpty() )
 		return 0;
 
-	if ( m_eSaving.load ( std::memory_order_relaxed ) == WriteState_e::DISABLED )
-	{
-		sError = "index is locked now, try again later";
-		return -1;
-	}
-
 	int iUpdated = tUpd.m_iAffected;
-
-	UpdateContext_t tCtx ( tUpd, m_tSchema );
-	if ( !Update_CheckAttributes ( *tCtx.m_tUpd.m_pUpdate, tCtx.m_tSchema, sError ) )
+	if ( !Update_CheckAttributes ( *tUpd.m_pUpdate, m_tSchema, sError ) )
 		return -1;
 
-	Update_PrepareListOfUpdatedAttributes ( tCtx, sError );
+	UpdateContext_t tCtx ( tUpd, m_tSchema );
+	tCtx.PrepareListOfUpdatedAttributes ( sError );
 
 	// do update in serial fiber. That ensures no concurrency with set of chunks changing, however need to dispatch
 	// with changers themselves (merge segments, merge chunks, save disk chunks).
@@ -7842,18 +7868,21 @@ int RtIndex_c::UpdateAttributes ( AttrUpdateInc_t & tUpd, bool & bCritical, CSph
 		if ( dRamUpdateSets[i].IsEmpty() )
 			continue;
 
+		if ( fnWatcher && !fnWatcher() )
+			return -1;
+
 		auto* pSeg = const_cast<RtSegment_t*> ( (const RtSegment_t*)tGuard.m_dRamSegs[i] );
 		SccWL_t wLock ( pSeg->m_tLock );
 
+		assert ( pSeg->GetStride() == m_tSchema.GetRowSize() );
+
 		// point context to target segment
 		tCtx.m_pAttrPool = pSeg->m_dRows.begin();
 		tCtx.m_pBlobPool = pSeg->m_dBlobs.begin();
-		tCtx.m_pSegment = pSeg;
-		if ( !UpdateAttributesInRamSegment ( dRamUpdateSets[i], tCtx, bCritical, sError ) )
+		if ( !pSeg->Update_UpdateAttributes ( dRamUpdateSets[i], tCtx, bCritical, sError ) )
 			return -1;
 
-		if ( pSeg->m_bAttrsBusy.load ( std::memory_order_acquire ) )
-			AddDerivedUpdate ( dRamUpdateSets[i], tCtx ); // segment is now saving/merging - add postponed update.
+		pSeg->MaybeAddPostponedUpdate( dRamUpdateSets[i], tCtx );
 
 		if ( tUpd.AllApplied () )
 			break;
@@ -7866,7 +7895,7 @@ int RtIndex_c::UpdateAttributes ( AttrUpdateInc_t & tUpd, bool & bCritical, CSph
 	Binlog::CommitUpdateAttributes ( &m_iTID, m_sIndexName.cstr(), tUpdc );
 
 	iUpdated = tUpd.m_iAffected - iUpdated;
-	if ( !Update_HandleJsonWarnings ( tCtx, iUpdated, sWarning, sError ) )
+	if ( !tCtx.HandleJsonWarnings ( iUpdated, sWarning, sError ) )
 		return -1;
 
 	// all done
@@ -7881,7 +7910,7 @@ bool RtIndex_c::SaveAttributes ( CSphString & sError ) const
 
 	const auto& pDiskChunks = m_tRtChunks.DiskChunks();
 
-	if ( pDiskChunks->IsEmpty() || ( m_eSaving.load ( std::memory_order_relaxed ) == WriteState_e::DISCARD ) )
+	if ( pDiskChunks->IsEmpty() || m_tSaving.Is ( SaveState_c::DISCARD ) )
 		return true;
 
 	for ( auto& pChunk : *pDiskChunks )
@@ -8187,7 +8216,7 @@ bool RtIndex_c::AttachDiskIndex ( CSphIndex* pIndex, bool bTruncate, bool & bFat
 		return false;
 
 	// safeguards
-	// we do not support some of the disk index features in RT just yet
+	// we do not support some disk index features in RT just yet
 #define LOC_ERROR(_arg) { sError = _arg; return false; }
 	const CSphIndexSettings & tSettings = pIndex->GetSettings();
 	if ( tSettings.m_iStopwordStep!=1 )
@@ -8627,7 +8656,7 @@ static int64_t NumAliveDocs ( const CSphIndex& dChunk )
 	return dChunk.GetStats().m_iTotalDocuments - tStatus.m_iDead;
 }
 
-void RtIndex_c::BinlogCommit ( RtSegment_t * pSeg, const CSphVector<DocID_t> & dKlist ) REQUIRES ( pSeg->m_tLock )
+void RtIndex_c::BinlogCommit ( RtSegment_t * pSeg, const VecTraits_T<DocID_t> & dKlist ) REQUIRES ( pSeg->m_tLock )
 {
 //	Tracer::AsyncOp tTracer ( "rt", "RtIndex_c::BinlogCommit" );
 	Binlog::Commit ( Binlog::COMMIT, &m_iTID, m_sIndexName.cstr(), false, [pSeg,&dKlist,this] (CSphWriter& tWriter) REQUIRES ( pSeg->m_tLock )
@@ -8691,7 +8720,7 @@ Binlog::CheckTnxResult_t RtIndex_c::ReplayCommit ( CSphReader & tReader, CSphStr
 	DWORD uRows = tReader.UnzipOffset();
 	if ( uRows )
 	{
-		pSeg = new RtSegment_t(uRows);
+		pSeg = new RtSegment_t(uRows, m_tSchema);
 		FakeWL_t _ ( pSeg->m_tLock );
 		pSeg->m_uRows = uRows;
 		pSeg->m_tAliveRows.store ( uRows, std::memory_order_relaxed );
@@ -8883,7 +8912,7 @@ bool RtIndex_c::CompressOneChunk ( int iChunkID, int* pAffected )
 	auto& dUpdates = pVictim->CastIdx().m_dPostponedUpdates;
 	if ( !dUpdates.IsEmpty() )
 	{
-		tCompressed.UpdateAttributesOffline ( dUpdates, &tCompressed );
+		tCompressed.UpdateAttributesOffline ( dUpdates );
 		dUpdates.Reset();
 	}
 
@@ -9052,8 +9081,8 @@ bool RtIndex_c::SplitOneChunk ( int iChunkID, const char* szUvarFilter, int* pAf
 	auto& dUpdates = pVictim->CastIdx().m_dPostponedUpdates;
 	if ( !dUpdates.IsEmpty() )
 	{
-		tIndexI.UpdateAttributesOffline ( dUpdates, &tIndexI );
-		tIndexE.UpdateAttributesOffline ( dUpdates, &tIndexE );
+		tIndexI.UpdateAttributesOffline ( dUpdates );
+		tIndexE.UpdateAttributesOffline ( dUpdates );
 		dUpdates.Reset();
 	}
 
@@ -9150,7 +9179,7 @@ bool RtIndex_c::MergeTwoChunks ( int iAID, int iBID, int* pAffected )
 	auto dUpdates = GatherUpdates::FromChunksOrSegments ( tUpdated );
 	if ( !dUpdates.IsEmpty() )
 	{
-		tMerged.UpdateAttributesOffline ( dUpdates, &tMerged );
+		tMerged.UpdateAttributesOffline ( dUpdates );
 		dUpdates.Reset();
 	}
 
@@ -9617,7 +9646,7 @@ int	RtIndex_c::Kill ( DocID_t tDocID )
 }
 
 
-int RtIndex_c::KillMulti ( const VecTraits_T<DocID_t> & dKlist )
+int RtIndex_c::KillMulti ( const VecTraits_T<DocID_t> & /*dKlist*/ )
 {
 	assert ( 0 && "No external kills for RT");
 	return 0;
@@ -9768,13 +9797,13 @@ bool RtIndex_c::CopyExternalFiles ( int /*iPostfix*/, StrVec_t & dCopied )
 void RtIndex_c::ProhibitSave()
 {
 	StopOptimize();
-	m_eSaving.store ( WriteState_e::DISCARD, std::memory_order_relaxed );
+	m_tSaving.SetState ( SaveState_c::DISCARD );
 	std::atomic_thread_fence ( std::memory_order_release );
 }
 
 void RtIndex_c::EnableSave()
 {
-	m_eSaving.store ( WriteState_e::ENABLED, std::memory_order_relaxed );
+	m_tSaving.SetState ( SaveState_c::ENABLED );
 	m_bOptimizeStop.store ( false, std::memory_order_relaxed );
 	std::atomic_thread_fence ( std::memory_order_release );
 }
@@ -9786,7 +9815,9 @@ void RtIndex_c::LockFileState ( CSphVector<CSphString>& dFiles )
 	ForceRamFlush ( "forced" );
 	CSphString sError;
 	SaveAttributes ( sError ); // fixme! report error, better discard whole locking
-	m_eSaving.store ( WriteState_e::DISABLED, std::memory_order_relaxed );
+	// that will ensure, if current txn is applying, it will be finished (especially kill pass) before we continue.
+	ScopedScheduler_c tSerialFiber ( m_tWorkers.SerialChunkAccess() );
+	m_tSaving.SetState ( SaveState_c::DISABLED );
 	std::atomic_thread_fence ( std::memory_order_release );
 	GetIndexFiles ( dFiles, dFiles );
 }
@@ -10076,4 +10107,21 @@ void RtIndex_c::RecalculateRateLimit ( int64_t iSaved, int64_t iInserted, bool b
 	m_iSoftRamLimit = m_iRtMemLimit * m_fSaveRateLimit;
 
 	TRACE_COUNTER ( "mem", perfetto::CounterTrack ( "Ratio", "%" ), m_fSaveRateLimit );
-}
+}
+
+bool RtIndex_c::AlterSI ( CSphString & sError )
+{
+	// strength single-fiber access (don't rely upon to upstream w-lock)
+	ScopedScheduler_c tSerialFiber ( m_tWorkers.SerialChunkAccess() );
+	TRACE_SCHED ( "rt", "alter-si" );
+
+	auto pChunks = m_tRtChunks.DiskChunks();
+	for ( auto & tChunk : *pChunks )
+	{
+		if ( !tChunk->CastIdx().AlterSI ( sError ) )
+			return false;
+	}
+
+	RaiseAlterGeneration();
+	return true;
+}

src/sphinxrt.h

@@ -250,10 +250,11 @@ public:
 	DeadRowMap_Ram_c				m_tDeadRowMap;
 	std::unique_ptr<DocstoreRT_i>	m_pDocstore;
 	std::unique_ptr<ColumnarRT_i>	m_pColumnar;
+	const ISphSchema&				m_tSchema;
 
 	mutable bool					m_bConsistent{false};
 
-							explicit RtSegment_t ( DWORD uDocs );
+							RtSegment_t ( DWORD uDocs, const ISphSchema& tSchema );
 
 	int64_t					GetUsedRam() const;				// get cached ram usage counter
 	void					UpdateUsedRam() const;			// recalculate ram usage, update index ram counter
@@ -262,6 +263,7 @@ public:
 
 	const CSphRowitem * 	FindAliveRow ( DocID_t tDocid ) const;
 	const CSphRowitem *		GetDocinfoByRowID ( RowID_t tRowID ) const;
+	RowID_t					GetAliveRowidByDocid ( DocID_t tDocid ) const;
 	RowID_t					GetRowidByDocid ( DocID_t tDocID ) const;
 
 	int						Kill ( DocID_t tDocID ) override;
@@ -270,12 +272,16 @@ public:
 	void					SetupDocstore ( const CSphSchema * pSchema );
 	void					BuildDocID2RowIDMap ( const CSphSchema & tSchema );
 
+	void					MaybeAddPostponedUpdate ( const RowsToUpdate_t& dRows, const UpdateContext_t& tCtx );
+	void					UpdateAttributesOffline ( VecTraits_T<PostponedUpdate_t>& dPostUpdates ) final;
+
 private:
 	mutable int64_t			m_iUsedRam = 0;			///< ram usage counter
 
 							~RtSegment_t () final;
 
 	void					FixupRAMCounter ( int64_t iDelta ) const;
+	bool					Update_WriteBlobRow ( UpdateContext_t& tCtx, RowID_t tRowID, ByteBlob_t tBlob, int nBlobAttrs, const CSphAttrLocator& tBlobRowLoc, bool& bCritical, CSphString& sError ) final;
 };
 
 using RtSegmentRefPtf_t = CSphRefcountedPtr<RtSegment_t>;

src/sphinxsort.cpp

@@ -6091,12 +6091,12 @@ void QueueCreator_c::ReplaceGroupbyStrWithExprs ( CSphMatchComparatorState & tSt
 			const CSphColumnInfo & tAttr = tSorterSchema.GetAttr(iRemap);
 			const_cast<CSphColumnInfo &>(tAttr).m_eStage = SPH_EVAL_PRESORT;
 		}
-		else
+		else if ( !pGroupStrBase->IsColumnar() )
 		{
 			CSphString sRemapCol;
 			sRemapCol.SetSprintf ( "%s%s", g_sIntAttrPrefix, pGroupStrBase->m_sName.cstr() );
-
 			iRemap = tSorterSchema.GetAttrIndex ( sRemapCol.cstr() );
+
 			if ( iRemap==-1 )
 			{
 				CSphColumnInfo tRemapCol ( sRemapCol.cstr(), SPH_ATTR_STRINGPTR );
@@ -6108,10 +6108,13 @@ void QueueCreator_c::ReplaceGroupbyStrWithExprs ( CSphMatchComparatorState & tSt
 			}
 		}
 
-		tState.m_eKeypart[i] = SPH_KEYPART_STRINGPTR;
-		tState.m_tLocator[i] = tSorterSchema.GetAttr(iRemap).m_tLocator;
-		tState.m_dAttrs[i] = iRemap;
-		tState.m_dRemapped.BitSet ( i );
+		if ( iRemap!=-1 )
+		{
+			tState.m_eKeypart[i] = SPH_KEYPART_STRINGPTR;
+			tState.m_tLocator[i] = tSorterSchema.GetAttr(iRemap).m_tLocator;
+			tState.m_dAttrs[i] = iRemap;
+			tState.m_dRemapped.BitSet ( i );
+		}
 	}
 }
 

src/std/stringbuilder_impl.h

@@ -296,7 +296,7 @@ inline StringBuilder_c& operator<< ( StringBuilder_c& tOut, timestamp_t tVal )
 template<typename INT, int iPrec>
 inline StringBuilder_c& operator<< ( StringBuilder_c& tOut, FixedFrac_T<INT, iPrec>&& tVal )
 {
-	tOut.template IFtoA(tVal);
+	tOut.template IFtoA<INT, iPrec>(tVal);
 	return tOut;
 }
 

src/threads_detached.cpp

@@ -61,6 +61,8 @@ void Detached::MakeAloneIteratorAvailable ()
 //#endif
 }
 
+static int64_t g_tmShutdownAllAlonesDelta = 3; // max allowed wait in seconds
+
 void Detached::ShutdownAllAlones()
 {
 #if !_WIN32
@@ -71,6 +73,9 @@ void Detached::ShutdownAllAlones()
 	}
 	int iTurn = 1;
 
+	int64_t tmStart = sphMicroTimer();
+	int64_t tmEnd = tmStart + g_tmShutdownAllAlonesDelta * 1000000;
+
 	while ( iThreads > 0 )
 	{
 		{
@@ -109,6 +114,13 @@ void Detached::ShutdownAllAlones()
 		}
 
 		++iTurn;
+
+		int64_t tmCur = sphMicroTimer(); 
+		if ( tmCur>tmEnd )
+		{
+			sphWarning ( "ShutdownAllAlones exits by timeout (%.3f seconds) but still has %d alone threads", ( (tmCur-tmStart)/1000000.0f ), iThreads );
+			break;
+		}
 	}
 #endif
 }

test/test_431/test.xml

@@ -107,6 +107,7 @@ desc META:all;
 select * from META:all;
 select count(*) from META:all group by property;
 select count(distinct title) from META:all group by brand_id;
+select count(distinct title) from META:all;
 </sphinxql></queries>
 
 </test>