Not working with Data Piplelines as of Nov 8

[dchymko]

We had this script stop working on data pipeline on or around Nov 8, 2016 (using the AWS walk through approach with Data pipelines). We also can't get it running on a new instance either. I'm not sure what changed, still investigating. It seems the instance created by Data pipelines cant see the mounts

The mount commands weren't throwing any timeout errors, so I spun up an EC2 instance with the same ami as Data pipeline uses. The mount command works the first time but no files appear on the share (they do on a "standard" EC2 Ami using the same mount command. If I unmount and run the command a second time, it hangs and doesn't time out (even after 20 mins)

Will do some more investigating but for now we just have the efs-backup.sh command running on a t2.micro as a cron (which works fine)

[dchymko]

Confirmed its the most recent version samples/EFSBackup/efs-backup.sh from c5876ef . For now in the data pipeline, I've changed myShellCmd to use this instead:

wget https://raw.githubusercontent.com/awslabs/data-pipeline-samples/2482fe032d01a31253da931d8582babec4ab1694/samples/EFSBackup/efs-backup.sh

Must be something to do with the params on the mount command. I'll investigate a bit further and update on the weekend.

[dchymko]

This turned out to be due to the newer command using NFS4.1 which isn't supported by the default data pipeline AMI (amzn-ami-pv-2013.03.1.x86_64-ebs (ami-05355a6c)).

To fix either:

1. In your data pipeline, specify a newer AMI (amzn-ami-pv-2016.03.2.x86_64-ebs - ami-0188776c)
2. Use the previous version that mount without specifying NFS version (less desirable)

[svscorp]

Faced the same. Is somebody is taking a look at this?

@jos4github @stmcpherson @mbeitchman @vinayaktAWS

[jagadeeshops]

Changing the AMI ID works.
Thanks @dchymko
This shows that we need to configure monitoring and auditing for all solutions...