Flink Docker Kubernetes Gitlab CI CDeployment

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Flink Docker Kubernetes Gitlab CI CDeployment

Nancy Estrada
Hi all,

Has someone successfully run Flink jobs with this type of setup (Gitlab CI CD and Kubernetes)?

Since Flink jobs can’t be dockerized and deployed in a natural way as part of the container (according to Flip-6) I am not very sure of how is the best way of doing this.

We are thinking of having 3 Docker files:  One for the Job manager, one for the Task Manager, and one for a container that will submit the jars and properties File to the Job Manager. With this setup we can still use the official Flink Docker images, and we can automatically  generate the third Docker file with the project specific tasks. This setup allows to have Continuous deployment. However, it implies that we will have a new Flink cluster every time there is a new code version.

What do you think of this set up? Has someone implemented something similar? Is there any documentation about Flink-Gitlab-CI-CD-Kubernetes?

Thank you very much,
Nancy
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Flink Docker Kubernetes Gitlab CI CDeployment

rmetzger0
Hi Nancy,

there is no documentation on a Gitlab Ci integration with a K8s deployment available.

But I'm pretty sure you can deploy Flink using docker containers (check this out: http://flink.apache.org/news/2017/05/16/official-docker-image.html)
I'm CCing Patrick, he has some experience deploying Flink in these environments.


On Mon, May 29, 2017 at 6:50 PM, Nancy Estrada <[hidden email]> wrote:
Hi all,

Has someone successfully run Flink jobs with this type of setup (Gitlab CI
CD and Kubernetes)?

Since Flink jobs can’t be dockerized and deployed in a natural way as part
of the container (according to Flip-6) I am not very sure of how is the best
way of doing this.

We are thinking of having 3 Docker files:  One for the Job manager, one for
the Task Manager, and one for a container that will submit the jars and
properties File to the Job Manager. With this setup we can still use the
official Flink Docker images, and we can automatically  generate the third
Docker file with the project specific tasks. This setup allows to have
Continuous deployment. However, it implies that we will have a new Flink
cluster every time there is a new code version.

What do you think of this set up? Has someone implemented something similar?
Is there any documentation about Flink-Gitlab-CI-CD-Kubernetes?

Thank you very much,
Nancy




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Docker-Kubernetes-Gitlab-CI-CDeployment-tp13371.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Loading...