[Mysql]reduce number of BinlogSplitMetaEvent transmission between MySqlSoureReader and MySqlSourceEnumerator #1479
Conversation
…eReader and MySqlSourceEnumerator; fixed log in MySqlSourceEnumerator
| @@ -325,7 +325,7 @@ private void fillMetaDataForBinlogSplit(BinlogSplitMetaEvent metadataEvent) { | |||
| receivedMetaGroupId, | |||
| expectedMetaGroupId); | |||
| } | |||
| requestBinlogSplitMetaIfNeeded(binlogSplit); | |||
| requestBinlogSplitMetaIfNeeded(uncompletedBinlogSplits.get(binlogSplit.splitId())); | |||
There was a problem hiding this comment.
I don't understand this change, could you explain more ?
There was a problem hiding this comment.
after fillMetaDataForBinlogSplit, will create a new binlogSplit in uncompletedBinlogSplits and cover last one , should use new binlogSplit to calculate nextMetaGroupId
There was a problem hiding this comment.
if use old binlogSplit，it will cause MySqlSourceReader send repeate BinlogSplitMetaRequestEvent and recevie repeate BinlogSplitMetaEvent
There was a problem hiding this comment.
@lidoudou1993 Could you add a test for the PR？
There was a problem hiding this comment.
I recurrence this problem byMySqlSourceExampleTest in debug model. When set splitMetaGroupSize=5,splitSize=50 and two thousand record in table，it will lead receivedMetaGroupId != expectedMetaGroupId when use old binlogSplit calculate nextMetaGroupId,and MySqlSourceReader need send more BinlogSplitMetaRequestEvent to MySqlSourceEnumerator.
There was a problem hiding this comment.
if use new binlogSplit to calculate nextMetaGroupId,this problem will not happen
There was a problem hiding this comment.
@lidoudou1993 you're right, I catch your point
… next meta (apache#1479) Co-authored-by: lidoudou3 <lidoudou1@jd.com>
reduce number of BinlogSplitMetaEvent transmission between MySqlSoureReader and MySqlSourceEnumerator; fixed log MySqlSourceEnumerator