CEP operator in SQL pattern match does not clear it's state

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

CEP operator in SQL pattern match does not clear it's state

Muhammad Hakim
Hi All,
We've been experimenting with `MATCH_RECOGNIZE` clause on Flink 1.8. We tried to have a pattern match within one hour. According to https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html#time-constraint , using `WITHIN` will control the memory usage. We've been running queries using CEP for almost one week and memory usage is not going down. Are we missing something?
Reply | Threaded
Open this post in threaded view
|

Re: CEP operator in SQL pattern match does not clear it's state

Dawid Wysakowicz-2
Hi,

Could you give a bit more details about your program? What is the query
you are using? What is the incoming data?

The "within" clause ensures that all partial matches that were initiated
at time t, are cleared when an event with timestamp (t + w) appears,
where w is the interval specified in "within" clause. The reason why it
is a good practice to use it for memory control is that it ensures that
there will be no partial matches dangling forever.

If the rate of incoming events and the probability of initiating new
partial matches is constant in the input stream I would not expect the
memory consumption to go down, but it should saturate after some time
and remain rather constant, but should not grow infinitely.

I hope it helps.

Best,

Dawid

On 01/10/2019 09:08, Muhammad Hakim wrote:
> Hi All,
> We've been experimenting with `MATCH_RECOGNIZE` clause on Flink 1.8.
> We tried to have a pattern match within one hour. According
> to https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html#time-constraint ,
> using `WITHIN` will control the memory usage. We've been running
> queries using CEP for almost one week and memory usage is not going
> down. Are we missing something?


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CEP operator in SQL pattern match does not clear it's state

Theo Diefenthal
In reply to this post by Muhammad Hakim
Hi Muhammad,

With "not going done" you mean that it's still growing or that it's constant? In case of it being constant, that's pretty much what is expected, right? If it's still growing, the only reason I could come up with: Do you work in event time and have watermarks properly assigned with a boundedTimestampExtractor or ascendingTimestampExtractor?

Best regards
Theo


Von: "Muhammad Hakim" <[hidden email]>
An: "user" <[hidden email]>
Gesendet: Dienstag, 1. Oktober 2019 09:08:22
Betreff: CEP operator in SQL pattern match does not clear it's state

Hi All,
We've been experimenting with `MATCH_RECOGNIZE` clause on Flink 1.8. We tried to have a pattern match within one hour. According to https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html#time-constraint , using `WITHIN` will control the memory usage. We've been running queries using CEP for almost one week and memory usage is not going down. Are we missing something?