In-transit Data Encryption in EMR

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

In-transit Data Encryption in EMR

vinay patil
Hi,

Currently I am looking into configuring in-transit data encryption either using Flink SSL Setup or directly using EMR.

Few Doubts:
   1. Will the existing functionality provided by Amazon to configure in-transit data encrytion work for Flink as well. This is explained here:
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-encryption-enable-security-configuration.html
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-data-encryption-options.html#emr-encryption-intransit

   2. Using Flink SSL Setup: as we know only the IP address of master node on EMR , should we pass only its ip address in the SAN list as given here ? (I think it should work as the yarn-cli command will distribute the truststore and keystore to each TM )
https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/security-ssl.html#use-yarn-cli-to-deploy-the-keystores-and-truststore

Regards,
Vinay Patil
Reply | Threaded
Open this post in threaded view
|

Re: In-transit Data Encryption in EMR

Till Rohrmann
Hi Vinay,

I've pulled my colleague Gordon into the conversation who can probably tell you more about Flink's security features.

Cheers,
Till

On Fri, Jun 2, 2017 at 2:22 PM, vinay patil <[hidden email]> wrote:
Hi,

Currently I am looking into configuring in-transit data encryption either
using Flink SSL Setup or directly using EMR.

Few Doubts:
   1. Will the existing functionality provided by Amazon to configure
in-transit data encrytion work for Flink as well. This is explained here:
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-encryption-enable-security-configuration.html
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-data-encryption-options.html#emr-encryption-intransit

   2. Using Flink SSL Setup: as we know only the IP address of master node
on EMR , should we pass only its ip address in the SAN list as given here ?
(I think it should work as the yarn-cli command will distribute the
truststore and keystore to each TM )
https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/security-ssl.html#use-yarn-cli-to-deploy-the-keystores-and-truststore

Regards,
Vinay Patil



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: In-transit Data Encryption in EMR

vinay patil
Thank you Till.

Gordon can you please help.

Regards,
Vinay Patil

On Fri, Jun 2, 2017 at 9:10 PM, Till Rohrmann [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:
Hi Vinay,

I've pulled my colleague Gordon into the conversation who can probably tell you more about Flink's security features.

Cheers,
Till

On Fri, Jun 2, 2017 at 2:22 PM, vinay patil <[hidden email]> wrote:
Hi,

Currently I am looking into configuring in-transit data encryption either
using Flink SSL Setup or directly using EMR.

Few Doubts:
   1. Will the existing functionality provided by Amazon to configure
in-transit data encrytion work for Flink as well. This is explained here:
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-encryption-enable-security-configuration.html
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-data-encryption-options.html#emr-encryption-intransit

   2. Using Flink SSL Setup: as we know only the IP address of master node
on EMR , should we pass only its ip address in the SAN list as given here ?
(I think it should work as the yarn-cli command will distribute the
truststore and keystore to each TM )
https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/security-ssl.html#use-yarn-cli-to-deploy-the-keystores-and-truststore

Regards,
Vinay Patil



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.




If you reply to this email, your message will be added to the discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13459.html
To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: In-transit Data Encryption in EMR

Tzu-Li (Gordon) Tai
Hi Vinay!

 1. Will the existing functionality provided by Amazon to configure
in-transit data encrytion work for Flink as well. This is explained here:
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-encryption-enable-security-configuration.html
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-data-encryption-options.html#emr-encryption-intransit

I don’t think so. AFAIK, the AWS security configurations needs to be integrated for per-platform’s specific security features, and as of now, there doesn’t seem to be an integration for Flink SSL encryption features, yet.

 2. Using Flink SSL Setup: as we know only the IP address of master node
on EMR , should we pass only its ip address in the SAN list as given here ?
(I think it should work as the yarn-cli command will distribute the
truststore and keystore to each TM )
https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/security-ssl.html#use-yarn-cli-to-deploy-the-keystores-and-truststore

The generated certificate needs to cover all nodes (hostname and IP address). Is it possible for you to use wildcard subdomain names to generate the certificate?
I’m not entirely sure of the subdomain patterns of EMR nodes, but this should be possible.

Cheers,
Gordon

On 5 June 2017 at 12:56:45 PM, vinay patil ([hidden email]) wrote:

Thank you Till.

Gordon can you please help.

Regards,
Vinay Patil

On Fri, Jun 2, 2017 at 9:10 PM, Till Rohrmann [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:
Hi Vinay,

I've pulled my colleague Gordon into the conversation who can probably tell you more about Flink's security features.

Cheers,
Till

On Fri, Jun 2, 2017 at 2:22 PM, vinay patil <[hidden email]> wrote:
Hi,

Currently I am looking into configuring in-transit data encryption either
using Flink SSL Setup or directly using EMR.

Few Doubts:
   1. Will the existing functionality provided by Amazon to configure
in-transit data encrytion work for Flink as well. This is explained here:
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-encryption-enable-security-configuration.html
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-data-encryption-options.html#emr-encryption-intransit

   2. Using Flink SSL Setup: as we know only the IP address of master node
on EMR , should we pass only its ip address in the SAN list as given here ?
(I think it should work as the yarn-cli command will distribute the
truststore and keystore to each TM )
https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/security-ssl.html#use-yarn-cli-to-deploy-the-keystores-and-truststore

Regards,
Vinay Patil



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.




If you reply to this email, your message will be added to the discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13459.html
To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML



View this message in context: Re: In-transit Data Encryption in EMR
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: In-transit Data Encryption in EMR

vinay patil
Hi Gordan,

Thank you for your response.

I have done the necessary configurations by adding all the node ip's from Resource Manager , is this correct ?
Also I will try to check if wildcard works as all our hostname begins with a same pattern.
For ex : SAN=dns:ip-192-168.* should work , right ?


Facing a weird issue when I try to submit the job using the following command:
flink run -m yarn-cluster -yn 4 -ys 4 -yjm 1024 -ytm 4000 -yt deploy-keys/ testFlinkSSL.jar --configFileName conf.yaml

Error is : java.lang.IllegalArgumentException: Wrong FS: hdfs://<some_ip>:8020/user/hadoop/.flink/application_1496660166576_0001/flink-dist_2.10-1.2.0.jar, expected: file:///

I see a JIRA ticket regarding the same but did not find any solution to this.

Regards,
Vinay Patil

Reply | Threaded
Open this post in threaded view
|

Re: In-transit Data Encryption in EMR

vinay patil
Hi Gordon,

The yarn session gets created when I try to run the following command:
yarn-session.sh -n 4 -s 2 -jm 1024 -tm 3000 -d --ship deploy-keys/

However when I try to access the Job Manager UI, it gives me exception as :
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I am able to see the Job Manager UI  when I imported the CA certificate to java truststore on EMR master node :
keytool -keystore /etc/alternatives/jre/lib/security/cacerts -importcert -alias FLINKSSL -file ca.cer


Does this mean that SSL is configured correctly ? I can see in the Job Manager configurations and also in th e logs. Is there any other way to verify ?

Also the keystore and truststore  password should be masked in the logs which is not case.

2017-06-05 14:51:31,135 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.enabled, true
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore, deploy-keys/ca.keystore
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore-password, password
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.key-password, password
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore, deploy-keys/ca.truststore
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore-password, password



Regards,
Vinay Patil
Reply | Threaded
Open this post in threaded view
|

Re: In-transit Data Encryption in EMR

vinay patil
Hi Guys,

I am able to setup SSL correctly, however the following command  does not work correctly and results in the error I had mailed earlier
flink run -m yarn-cluster -yt deploy-keys/ TestJob.jar

Few Doubts: 
1. Can anyone please explain me how do you test if SSL is working correctly ? Currently I am just relying on the logs.

2. Wild Card is not working with the keytool command, can you please let me know what is the issue with the following command:
keytool -genkeypair -alias ca -keystore: -ext SAN=dns:node1.* 


Regards,
Vinay Patil

On Mon, Jun 5, 2017 at 8:43 PM, vinay patil [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:
Hi Gordon,

The yarn session gets created when I try to run the following command:
yarn-session.sh -n 4 -s 2 -jm 1024 -tm 3000 -d --ship deploy-keys/

However when I try to access the Job Manager UI, it gives me exception as :
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I am able to see the Job Manager UI  when I imported the CA certificate to java truststore on EMR master node :
keytool -keystore /etc/alternatives/jre/lib/security/cacerts -importcert -alias FLINKSSL -file ca.cer


Does this mean that SSL is configured correctly ? I can see in the Job Manager configurations and also in th e logs. Is there any other way to verify ?

Also the keystore and truststore  password should be masked in the logs which is not case.

2017-06-05 14:51:31,135 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.enabled, true
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore, deploy-keys/ca.keystore
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore-password, password
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.key-password, password
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore, deploy-keys/ca.truststore
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore-password, password



Regards,
Vinay Patil



If you reply to this email, your message will be added to the discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13490.html
To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: In-transit Data Encryption in EMR

vinay patil
In reply to this post by vinay patil
Hi Guys,

Can anyone please provide me solution to my queries.

On Jun 8, 2017 11:30 PM, "Vinay Patil" <[hidden email]> wrote:
Hi Guys,

I am able to setup SSL correctly, however the following command  does not work correctly and results in the error I had mailed earlier
flink run -m yarn-cluster -yt deploy-keys/ TestJob.jar

Few Doubts: 
1. Can anyone please explain me how do you test if SSL is working correctly ? Currently I am just relying on the logs.

2. Wild Card is not working with the keytool command, can you please let me know what is the issue with the following command:
keytool -genkeypair -alias ca -keystore: -ext SAN=dns:node1.* 


Regards,
Vinay Patil

On Mon, Jun 5, 2017 at 8:43 PM, vinay patil [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:
Hi Gordon,

The yarn session gets created when I try to run the following command:
yarn-session.sh -n 4 -s 2 -jm 1024 -tm 3000 -d --ship deploy-keys/

However when I try to access the Job Manager UI, it gives me exception as :
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I am able to see the Job Manager UI  when I imported the CA certificate to java truststore on EMR master node :
keytool -keystore /etc/alternatives/jre/lib/security/cacerts -importcert -alias FLINKSSL -file ca.cer


Does this mean that SSL is configured correctly ? I can see in the Job Manager configurations and also in th e logs. Is there any other way to verify ?

Also the keystore and truststore  password should be masked in the logs which is not case.

2017-06-05 14:51:31,135 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.enabled, true
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore, deploy-keys/ca.keystore
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore-password, password
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.key-password, password
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore, deploy-keys/ca.truststore
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore-password, password



Regards,
Vinay Patil



If you reply to this email, your message will be added to the discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13490.html
To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: In-transit Data Encryption in EMR

Tzu-Li (Gordon) Tai
Hi Vinay,

Apologies for the inactivity on this thread, I was occupied with some critical fixes for 1.3.1.

1. Can anyone please explain me how do you test if SSL is working correctly ? Currently I am just relying on the logs.

AFAIK, if any of the SSL configuration settings are enabled (*.ssl.enabled) and your job is running fine, then everything should be functioning.

2. Wild Card is not working with the keytool command, can you please let me know what is the issue with the following command:

The wildcard option only works for wildcarding subdomains.
For example, SAN=*.domain.com

On 9 June 2017 at 4:33:46 PM, vinay patil ([hidden email]) wrote:

Hi Guys,

Can anyone please provide me solution to my queries.

On Jun 8, 2017 11:30 PM, "Vinay Patil" <[hidden email]> wrote:
Hi Guys,

I am able to setup SSL correctly, however the following command  does not work correctly and results in the error I had mailed earlier
flink run -m yarn-cluster -yt deploy-keys/ TestJob.jar

Few Doubts: 
1. Can anyone please explain me how do you test if SSL is working correctly ? Currently I am just relying on the logs.

2. Wild Card is not working with the keytool command, can you please let me know what is the issue with the following command:
keytool -genkeypair -alias ca -keystore: -ext SAN=dns:node1.* 


Regards,
Vinay Patil

On Mon, Jun 5, 2017 at 8:43 PM, vinay patil [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:
Hi Gordon,

The yarn session gets created when I try to run the following command:
yarn-session.sh -n 4 -s 2 -jm 1024 -tm 3000 -d --ship deploy-keys/

However when I try to access the Job Manager UI, it gives me exception as :
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I am able to see the Job Manager UI  when I imported the CA certificate to java truststore on EMR master node :
keytool -keystore /etc/alternatives/jre/lib/security/cacerts -importcert -alias FLINKSSL -file ca.cer


Does this mean that SSL is configured correctly ? I can see in the Job Manager configurations and also in th e logs. Is there any other way to verify ?

Also the keystore and truststore  password should be masked in the logs which is not case.

2017-06-05 14:51:31,135 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.enabled, true
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore, deploy-keys/ca.keystore
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore-password, password
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.key-password, password
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore, deploy-keys/ca.truststore
2017-06-05 14:51:31,136 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore-password, password



Regards,
Vinay Patil



If you reply to this email, your message will be added to the discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/In-transit-Data-Encryption-in-EMR-tp13455p13490.html
To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML



View this message in context: Re: In-transit Data Encryption in EMR
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.