FileSystem vs RocksDb backend

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

FileSystem vs RocksDb backend

Fritz Budiyanto
Hi All,

If my states fit in the JVM heap, should I be using filesystem backend with HDFS instead of RocksDB ?

Could someone comment on the FS vs RocksDb backend performance ?

I’ve also heard that RocksDb backend is required to support delta checkpointing. Is this true ?
Is delta checkpointing part of 1.3.0 ?


Thanks,
Fritz
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: FileSystem vs RocksDb backend

Fabian Hueske-2
Hi Fritz,

the FSStateBackend is a lot faster than the RocksDBStateBackend.
I don't know exact numbers but if I remember correctly somebody mentioned a 10x difference.

Delta checkpointing is only supported by RocksDB since Flink 1.3.
Given that the FSStateBackend is only suitable for small states, this should not be significant limitation.

Best, Fabian

2017-06-02 19:20 GMT+02:00 Fritz Budiyanto <[hidden email]>:
Hi All,

If my states fit in the JVM heap, should I be using filesystem backend with HDFS instead of RocksDB ?

Could someone comment on the FS vs RocksDb backend performance ?

I’ve also heard that RocksDb backend is required to support delta checkpointing. Is this true ?
Is delta checkpointing part of 1.3.0 ?


Thanks,
Fritz

Loading...