Missing size1.mrk file for column with Nested structure

[harlinb]

I have a column that looks like:

column Nested
(
  name String,
  names Array(String),
  types Array(Enum8('PU' = 1, 'US' = 2, 'OTHER' = 3))
)

After waiting for a significant amount of data to be ingested, when I restart clickhouse I recieve the following error messages:

2018.03.15 21:36:40.677379 [ 27 ] <Error> void DB::MergeTreeData::loadDataParts(bool): Code: 226, e.displayText() = DB::Exception: No column.size1.mrk file checksum for column 20180315-0_1_1_0 in part /var/lib/clickhouse/data/default/table/20180315-0_1_1_0/, e.what() = DB::Exception, Stack trace:
0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x15) [0x7317e35]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x1e) [0x19caa8e]
2. /usr/bin/clickhouse-server() [0x688e05a]
3. /usr/bin/clickhouse-server(DB::DataTypeArray::enumerateStreams(std::function<void (std::vector<DB::IDataType::Substream, std::allocator<DB::IDataType::Substream> > const&)>, std::vector<DB::IDataType::Substream, std::allocator<DB::IDataType::Substream> >) const+0xae) [0x6012d8e]
4. /usr/bin/clickhouse-server(DB::DataTypeArray::enumerateStreams(std::function<void (std::vector<DB::IDataType::Substream, std::allocator<DB::IDataType::Substream> > const&)>, std::vector<DB::IDataType::Substream, std::allocator<DB::IDataType::Substream> >) const+0x30c) [0x6012fec]
5. /usr/bin/clickhouse-server(DB::MergeTreeDataPart::checkConsistency(bool)+0x62e) [0x689473e]
6. /usr/bin/clickhouse-server(DB::MergeTreeData::loadDataParts(bool)+0x475) [0x6875b55]
7. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::StorageReplicatedMergeTree(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::NamesAndTypesList const&, DB::NamesAndTypesList const&, DB::NamesAndTypesList const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, DB::ColumnDefault, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, DB::ColumnDefault> > > const&, DB::Context&, std::shared_ptr<DB::IAST> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IAST> const&, DB::MergeTreeData::MergingParams const&, DB::MergeTreeSettings const&, bool)+0xcca) [0x6833d5a]
8. /usr/bin/clickhouse-server() [0x6917002]
9. /usr/bin/clickhouse-server(std::_Function_handler<std::shared_ptr<DB::IStorage> (DB::StorageFactory::Arguments const&), std::shared_ptr<DB::IStorage> (*)(DB::StorageFactory::Arguments const&)>::_M_invoke(std::_Any_data const&, DB::StorageFactory::Arguments const&)+0x22) [0x69190f2]
10. /usr/bin/clickhouse-server(DB::StorageFactory::get(DB::ASTCreateQuery&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context&, DB::Context&, DB::NamesAndTypesList const&, DB::NamesAndTypesList const&, DB::NamesAndTypesList const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, DB::ColumnDefault, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, DB::ColumnDefault> > > const&, bool, bool) const+0x23b) [0x67f836b]
11. /usr/bin/clickhouse-server(DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)+0x68a) [0x66b71aa]
12. /usr/bin/clickhouse-server(DB::InterpreterCreateQuery::execute()+0x5a) [0x66b9dea]
13. /usr/bin/clickhouse-server() [0x67652d2]
14. /usr/bin/clickhouse-server(DB::executeQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum)+0x137) [0x67662f7]
15. /usr/bin/clickhouse-server(DB::TCPHandler::runImpl()+0x38d) [0x19d9dad]
16. /usr/bin/clickhouse-server(DB::TCPHandler::run()+0x2a) [0x19dab9a]
17. /usr/bin/clickhouse-server(Poco::Net::TCPServerConnection::start()+0xe) [0x762b2ce]
18. /usr/bin/clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x165) [0x762b695]
19. /usr/bin/clickhouse-server(Poco::PooledThread::run()+0x76) [0x73e74b6]
20. /usr/bin/clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x37) [0x73e3827]
21. /usr/bin/clickhouse-server() [0x76fd52e]
22. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76b9) [0x7f3ea3e2f6b9]
23. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6c) [0x7f3ea385c3dc]
2018.03.15 21:36:40.677412 [ 27 ] <Error> default.table (Data): Considering to remove broken part /var/lib/clickhouse/data/default/table/20180315-0_1_1_0 because it's impossible to repair.

This error appears once for every part that has been merged several times.
After the restart, it queues fetches from the other replicas and gets back into the correct state:

2018.03.15 21:36:47.877837 [ 27 ] <Error> default.table (StorageReplicatedMergeTree): Fetching missing part 20180315-0_1_1_0
2018.03.15 21:36:47.897570 [ 27 ] <Error> default.table (StorageReplicatedMergeTree): Removing missing part from ZooKeeper and queueing a fetch: 20180315-0_1_1_0

After the parts are fetched, everything is back to normal.
If, however I restart every node in the shard, then data is reported to be lost.

2018.03.15 20:06:42.049535 [ 25 ] <Warning> default.table (StorageReplicatedMergeTree, PartCheckThread): Checking part 20180315-0_1_1_0
2018.03.15 20:06:42.049936 [ 25 ] <Warning> default.table (StorageReplicatedMergeTree, PartCheckThread): Checking if anyone has a part covering 20180315-0_1_1_0.
2018.03.15 20:06:42.052618 [ 25 ] <Error> default.table (StorageReplicatedMergeTree, PartCheckThread): No replica has part covering 20180315-0_1_1_0 and a merge is impossible: we didn't find smaller parts with either the same min block or the same max block.
2018.03.15 20:06:42.053984 [ 25 ] <Error> default.table (StorageReplicatedMergeTree, PartCheckThread): Part 20180315-0_1_1_0 is lost forever.

table engine looks like:

engine=ReplicatedMergeTree('/clickhouse/tables/0/table', 'node-03')
PARTITION BY (eventDate, policy)
ORDER BY (Id, timestamp)
SETTINGS index_granularity = 8192

clickhouse version 1.1.54343

[harlinb]

I was able to reproduce this bug.

• Start with a MergeTree table that has a Nested Array similar to above
• Continue to insert data until new parts is merged at least to level 3
• Merges start to remove the size1.mrk file from the Nested field
• Restart clickhouse
• Clickhouse complains that the size1.mrk doesn't exist

in my case I see following files in the merged:
column.size0.bin
column.size0.mrk

but am missing:
column.size1.bin
column.size1.mrk

(The size1 files do exist in all subsequent parts until they have been merged a sufficient number of times)

[alexey-milovidov]

I've reproduced similar issue with:

USE test;

DROP TABLE IF EXISTS nested;
CREATE TABLE nested
(
    column Nested
    (
        name String,
        names Array(String),
        types Array(Enum8('PU' = 1, 'US' = 2, 'OTHER' = 3))
    )
) ENGINE = MergeTree ORDER BY tuple();

INSERT INTO nested VALUES (['Hello', 'World'], [['a'], ['b', 'c']], [['PU', 'US'], ['OTHER']]);

SELECT * FROM nested;

It outputs

┌─column.name───────┬─column.names─┬─column.types─┐
│ ['Hello','World'] │ [[],[]]      │ [[],[]]      │
└───────────────────┴──────────────┴──────────────┘

[alexey-milovidov]

Fixed in master.