fllink预写日志和两阶段提交的差别

flink保证sink 端exactly-once有幂等写入和事务写入两种方式。事务写入的实现方式又有预写日志和两阶段提交。请问预写日志和两阶段提交的差别在哪里?两个好像都是先写入缓存,等checkpoint确认了,再写入磁盘。请问两者的差别在哪里?

The WAL sink writes all result records intoapplication state and emits them to the sink system once it receives thenotification that a checkpoint was completed. Since the sink buffersrecords in the state backend, the WAL sink can be used with any kindof sink system. However, it cannot provide bulletproof exactly-onceguarantees, adds to the state size of an application, and the sinksystem has to deal with a spiky writing pattern.In contrast, the 2PC sink requires a sink system that offers transactionalsupport or exposes building blocks to emulate transactions. For eachcheckpoint, the sink starts a transaction and appends all receivedrecords to the transaction, writing them to the sink system withoutcommitting them. When it receives the notification that a checkpointcompleted, it commits the transaction and materializes the writtenresults. The mechanism relies on the ability of a sink to commit atransaction after recovering from a failure that was opened before acompleted checkpoint.The 2PC protocol piggybacks on Flink’s existing checkpointingmechanism. The checkpoint barriers are notifications to start a newtransaction, the notifications of all operators about the success of theirindividual checkpoint are their commit votes, and the messages of theJobManager that notify about the success of a checkpoint are theinstructions to commit the transactions. In contrast to WAL sinks, 2PCsinks can achieve exactly-once output depending on the sink systemand the sink’s implementation. Moreover, a 2PC sink continuouslywrites records to the sink system compared to the spiky writing pattern2of a WAL sink.

你好,我是有问必答小助手,非常抱歉,本次您提出的有问必答问题,技术专家团超时未为您做出解答


本次提问扣除的有问必答次数,已经为您补发到账户,我们后续会持续优化,扩大我们的服务范围,为您带来更好地服务。