Re: about Kafka sink and 2PC function

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Re: about Kafka sink and 2PC function

Andrey Zagrebin-4

This is the contract of 2PC transactions.
Multiple commit retries should result in only one commit which actually happens in the external system.
The external system has to support deduplication of committed transactions, e.g. by some unique id.


On 10 Oct 2019, at 07:15, [hidden email] wrote:

After reading about FlinkKafkaProducer011 and 2PC function in FLINK, I know, 
when snapshotState(),
  • preCommit currentTransaction.
  • add <currentTransaction, newTransaction> to the State.
when Checkpoint done and notifyCheckpointComplete(),
  • producer will commit currentTransaction to brokers.
when initializeState(),
  • restore from State.
  • commit currentTransaction and abort newTransaction.
And I have one question, what happens if program fails after notifyCheckpointComplete() done?
As my opinion, when it recovers, it will re-commit what has committed which results duplicate.