[ANNOUNCE] Weekly Community Update 2019/43

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[ANNOUNCE] Weekly Community Update 2019/43

Konstantin Knauf-2
Dear community,

happy to share this week's community update with updates on some exciting ongoing efforts like unaligned checkpoints, the contribution of a Pulsar connector and the introduction of Executors, a new initiative around queryable state, a couple of bugs and a bit more.

Flink Development

* [development process] Gary encourages everyone involved in Flink development to join the builds@.f.a.o mailing list to receive updates on build stability issues (e.g. from the nightly builds). You can subscribe via [hidden email]. [1]

* [state] Piotrs has proposed a change to FLIP-76 on unaligned checkpoints. For unaligned checkpoints in-flight data needs to be stored as part of the checkpoint. Piotr proposes to continuously persist all stream records to avoid having to store a large amount of in-flight data during checkpointing. Details in the thread [2]

* [state] Vino Yang has revived the discussion on improving queryable state in Flink. Specifically, he proposes to introduce a new component called "QueryableStateLocationService": instead of connecting to the QueryableStateProxy of a Taskmanager a client would connect to the location service to find the correct TaskManager to serve its query. [3]

* [network] The survey on non credit-based flow control is over and it will be dropped in the next Flink release. [4]

* [connectors] For the contribution of the Pulsar connector(s), Yijie kicked off a discussion on a design document for a corresponding FLIP (FLIP-72). After some discussion, the initial FLIP will now cover mostly the sink. The source will be part of future work (after FLIP-27) and the catalog  will get its own FLIP [5].

* [client] Kostas has started (and concluded) a discussion on FLIP-81 to introduce new configuration options for Executors (FLIP-73). FLIP-81 mainly introduces configuration options for parameters, that were previously only exposed via the CLI (e.g. the path to a savepoint for initial state). The vote is currently ongoing [6,7]

* [sql] Two weeks ago Peter Huang started a discussion on FLIP-59 support for functions in Flink's DDL. The primary purpose is the dynamic registration of user defined functions via the DDL. [8]

Notable Bugs

* [FLINK-14074] [1.9.1] [mesos] There is a bug in the way Flink starts Taskmanagers on Mesos: after the initial set of Taskmanagers is started (e.g. during the first job submission) any subsequent job submission will not trigger additional Taskmanagers to be launched although they are needed to run the job. Check the ticket for a workaround. Fixed for 1.9.2. and 1.10.0.  [9]
* [FLINK-15524[ [1.9.1] The JDBC table sink generates an invalid PostgreSQL query for upserts. Fixed in 1.9.2 and 1.10.0. [10]
* [FLINK-14470] [1.9.1] [1.8.2] Not particularly new, but still open and confusing. The Flink WebUI might not show all watermarks for Flink jobs with many tasks. [11]

Events, Blog Posts, Misc

* Alibaba's food delivery business ele.me has published a blog post about their stream processing architecture and use cases for Flink. [10]

* There will be Flink/Spark talk at the next Chicago Big Data [11] on the 7th of November. No idea what it will be about (can't join the group) :)

* At the next Athens Big Data Group on the 14th of November Chaoran Yu of Lightbend will talk about Flink and Spark on Kubernetes. [12]

* There will be full-day meetup with six talks in the Bangalore Kafka Group on the 2nd of November including at least three Flink talks by Timo Walter (Ververica), Shashank Agarwal (Razorpay) and Rasyid Hakim (GoJek).  [13]


Konstantin (@snntrable)


Konstantin Knauf | Solutions Architect

+49 160 91394525

Follow us @VervericaData Ververica


Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time


Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany


Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Tony) Cheng