Looking for example for bucketingSink / StreamingFileSink

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Looking for example for bucketingSink / StreamingFileSink

Avi Levi
Hi Guys,
very new to flink so my apology for the newbie questions :)
but I desperately looking for a good example for streaming to file using bucketingSink / StreamingFileSink . Unfortunately the examples in the documentation are not event compiling (at least not the ones in scala https://issues.apache.org/jira/browse/FLINK-11053 )

I tried using bucketing sink with streamingFileSink (or just streamingFileSink ) and finally tried to implement a writer but with no luck. 
BucketingSink seems to be a perfect fit because I can set the batch interval by time interval or size which is exactly what I need.

This is my last attempt (sample project) which results a lot of "pending" files. 
Any help would be appreciated

Thanks
Avi
Reply | Threaded
Open this post in threaded view
|

Re: Looking for example for bucketingSink / StreamingFileSink

miki haiat
HI Avi , 
Im assuming that the cause  of the "pending" file is because the checkpoint isn't finished successfully [1]
Can you try to change the checkpoint time to 1 min as well .


Thanks,

On Mon, Dec 3, 2018 at 12:45 PM Avi Levi <[hidden email]> wrote:
Hi Guys,
very new to flink so my apology for the newbie questions :)
but I desperately looking for a good example for streaming to file using bucketingSink / StreamingFileSink . Unfortunately the examples in the documentation are not event compiling (at least not the ones in scala https://issues.apache.org/jira/browse/FLINK-11053 )

I tried using bucketing sink with streamingFileSink (or just streamingFileSink ) and finally tried to implement a writer but with no luck. 
BucketingSink seems to be a perfect fit because I can set the batch interval by time interval or size which is exactly what I need.

This is my last attempt (sample project) which results a lot of "pending" files. 
Any help would be appreciated

Thanks
Avi