Migrating a Legacy App to Cloud Native

Ship it!

This is part 8 in a series documenting my journey migrating my progressive web app, called SqAC, to AWS cloud native. If you haven’t been following it before now, here are the previous posts:

Now in part 8, I’ll add the final touches to the new AWS-hosted application and take it live! Also, my final conclusions.

Custom Domain Name

Hosting an application on S3 is easy — we did it back in part 3 with the amplify hosting add command. Easy, but not production worthy with a URL like http://sqac-amplify-20190817123020-hostingbucket-dev.s3-website-us-west-2.amazonaws.com

In part 7, CloudFront was added in order to have HTTPS, and we got a more succinct URL at d3l0j9nusq7n6r.cloudfront.net. That was better, but still not something I can ask someone to type into their browser. No, a custom domain is needed. In fact, my old application is running at sqac.fanello.net and it was time for this new cloud native version to take its place.

I first tried setting up a custom domain on my application via the AWS Amplify Console. The AWS Amplify Console though didn’t reder much new content upon clicking the Add Domain button and I found errors in the browser dev console. 😲 I suspect because this is feature centers around the idea of using AWS Amplify Console as a build-pipeline, which I don’t, and so without doing that domain management fails. Unfriendly UI, but it’s okay. This isn’t really an Amplify problem to solve; my app is really just hosted in an S3 bucket fronted by CloudFront and so I just need to find how to attach a custom domain to that. If this were just HTTP (no S), it would be a simple matter of adding a DNS CNAME record to forward my sub-domain to the AWS hosting domain. The security certificate provided by CloudFront complicates that because I can’t have a subdomain under fanello.net serve up a cloudfront.net certificate and expect a browser to accept it.

I tried Connecting to Third-Party Custom Domains, but that isn’t quite what I was trying to accomplish and so was unhelpful in itself. However, it led me to AWS Certificate Manager.

In AWS Console, I went to the AWS Certificate Manager and chose to create a new public certificate. (I did first check the pricing on such a certificate: it’s free!) I set up DNS Validation and then waited for it to happen… and nothing did.

After a couple tries, with a few days waiting each time, I eventually found that the CNAME name provided had a dot at the end that I had to remove when setting it in my DNS name server. 🤷‍♂️

When I tried to then attach the certificate to my CloudFront distribution, it wasn’t found. A quick Internet search turned up that CloudFront can only use certificates in us-east-1 (N. Virginia), something no tutorial bothered mentioning. 😞 It is on the CloudFront distribution settings page, so I would have known had read the small light grey text under the form field. Thus after a week finally getting my cert setup in us-west-2 (Oregon), I had to start over again. Fortunately I learned my lessons the first time. I removed the end dot, set the DNS TTL (Time To Live) values low, and had the new certificate setup in about five minutes.

Finally having the needed certificate, I could configure the CloudFront distribution by editing the settings. I added the certificate, set the “Alternative Domain Names” to my subdomain, turned on HTTP/2 (why is it not by default?) and lowered the price class to U.S., Canada, and Europe. Since the progressive web app is strongly cached by the browser, the slower access elsewhere in the world won’t rarely matter.

Refresh at https://sqac.fanello.net aaaand…. failure. Certificate still shows my old Let’s Encrypt issued certificate. With a delay of about a week playing around with the certificate, I forgot the initial goal! 🤦‍♂️ After setting the sqac.fanello.net subdomain as a CNAME to point to my CloudFront distribution…

I have a valid certificate from Amazon, the app loaded, but access to the S3 bucket managed by Amplify Storage is denied. Progress!

If you look back at the last blog post, after setting up CloudFront I then had to reconfigure authentication with the CloudFront URL. Now I have to change that to my new domain:

$ amplify update auth

Please note that certain attributes may not be overwritten if you choose to use defaults settings.

You have configured resources that might depend on this Cognito resource. Updating this Cognito resource could have unintended side effects.

 Using service: Cognito, provided by: awscloudformation  
 What do you want to do? Add/Edit signin and signout redirect URIs
 Which redirect signin URIs do you want to edit? https://d3l0j9nusq7n6r.cloudfront.net
 ? Update https://d3l0j9nusq7n6r.cloudfront.net/ https://sqac.fanello.net/  
 Do you want to add redirect signin URIs? No  
 Which redirect signout URIs do you want to edit? https://d3l0j9nusq7n6r.cloudfront.net/ 
? Update https://d3l0j9nusq7n6r.cloudfront.net/ https://sqac.fanello.net/ 
 Do you want to add redirect signout URIs? No  
TypeError: Cannot use 'in' operator to search for 'dev' in undefined  
    at keys.forEach.key (/Users/adamfanello/.nvm/versions/node/v10.16.3/lib/node\_modules/@aws-amplify/cli/lib/extensions/amplify-helpers/envResourceParams.js:33:19)  
    at Array.forEach (<anonymous>)  
    at getOrCreateSubObject (/Users/adamfanello/.nvm/versions/node/v10.16.3/lib/node\_modules/@aws-amplify/cli/lib/extensions/amplify-helpers/envResourceParams.js:32:10)  
    at AmplifyToolkit.saveEnvResourceParameters \[as \_saveEnvResourceParameters\] (/Users/adamfanello/.nvm/versions/node/v10.16.3/lib/node\_modules/@aws-amplify/cli/lib/extensions/amplify-helpers/envResourceParams.js:66:23)  
    at Object.saveResourceParameters (/Users/adamfanello/.nvm/versions/node/v10.16.3/lib/node\_modules/@aws-amplify/cli/node\_modules/amplify-provider-awscloudformation/src/resourceParams.js:26:19)  
    at saveResourceParameters (/Users/adamfanello/.nvm/versions/node/v10.16.3/lib/node\_modules/@aws-amplify/cli/node\_modules/amplify-category-auth/provider-utils/awscloudformation/index.js:110:12)  
    at serviceQuestions.then (/Users/adamfanello/.nvm/versions/node/v10.16.3/lib/node\_modules/@aws-amplify/cli/node\_modules/amplify-category-auth/provider-utils/awscloudformation/index.js:330:7)  
    at process.\_tickCallback (internal/process/next\_tick.js:68:7)  
There was an error adding the auth resource

Sigh. 😞 I issued Amplify CLI bug #3273. It turns out that something deleted my team-provider-info.json file, which is no longer in source control because it contained the Google Sign-In secret. I went to re-pull the environment from AWS, and thus recreate the file, but that too failed and I issued Amplify CLI bug #3274. 🙄 As a last straw, I found the old version of the file from source control, and then issued an amplify env pull. This time it worked, and prompted me for the Google authentication ID and secret, which I recovered from the Google Cloud Platform console. While in the GCP console, I realized I also needed to add the new subdomain in the list of Authorized redirect URIs, and so did so.

Then, Disaster

Everything was in place: custom domain name, SSL certificate, DNS redirection, missing file restore, and authentication reconfigure. Now was the time for the ultimate triumph! I issued an amplify push, and it froze. 😱 An hour later, the CloudFormation stack timed out and began a rollback to recover. After another hour, the rollback too had failed. The end explanation:

The following resource(s) failed to update: [OAuthCustomResourceInputs]

Huh? The top Google search hit for this error message was my own blog post from part 3 of this series! Unfortunately that problem was different. That time, there was a log in CloudWatch explaining what it didn’t like. This time there’s:

ERROR Uncaught Exception   
{  
    "errorType": "Runtime.ImportModuleError",  
    "errorMessage": "Error: Cannot find module 'cfn-response'",  
    "stack": \[  
        "Runtime.ImportModuleError: Error: Cannot find module 'cfn-response'",  
        "    at \_loadUserApp (/var/runtime/UserFunction.js:100:13)",  
        "    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)",  
        "    at Object.<anonymous> (/var/runtime/index.js:45:30)",  
        "    at Module.\_compile (internal/modules/cjs/loader.js:778:30)",  
        "    at Object.Module.\_extensions..js (internal/modules/cjs/loader.js:789:10)",  
        "    at Module.load (internal/modules/cjs/loader.js:653:32)",  
        "    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)",  
        "    at Function.Module.\_load (internal/modules/cjs/loader.js:585:3)",  
        "    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)",  
        "    at startup (internal/bootstrap/node.js:283:19)"  
    \]  
}

Searching for that turned up this issue with upgrading CloudFormation custom lambdas to Node.js 10 — something recently added by Amplify. I edited my authentication CloudFormation template to fix the relative path import as noted in the issue. The Amplify Node.js update instructions mention this change in behavior, but only in terms of our own custom functions, not the ones created by Amplify and hidden within CloudFormation templates. 😡

With that fixed, I still didn’t know how to recover. My “auth” CloudFormation stack was in the UPDATE_ROLLBACK_FAILED state. Multiple attempts to continue the rollback failed as well (after an hour each). I suspect that it was still trying to run the bad custom lambdas.

Failures like this can happen with any technology. This is why backups are important and when dealing with cloud services, we must always regard infrastructure as ephemeral. Data however, is not ephemeral. This stack contains the Cognito users. Also, as a nested stack for the entire application, it being in a failed state means the entire application stack is in a failed state.

I contacted someone on the AWS Amplify team asking for help. That contact, whom I met on LinkedIn and at re:Invent 2019, was quick to respond and pass me to a colleague on the Amplify CLI team. A couple email exchanges led simply to: “works for me”. 😢

Starting Over

Fortunately this was a development environment, and so the only data was my own. I had intended to simply roll it into production, but with the broken stack I was forced to start over with a brand new production environment:

$ amplify init
Scanning for plugins...
Plugin scan successful
Note: It is recommended to run this command from the root of your app directory
? Do you want to use an existing environment? No
? Enter a name for the environment prod
Using default provider awscloudformation

For more information on AWS Profiles, see:
docs.aws.amazon.com/cli/latest/userguide/cl..

? Do you want to use an AWS profile? Yes
? Please choose the profile you want to use sqac-amplify-cli
Adding backend environment prod to AWS Amplify Console app: d2g2lwfk7ok1zm
⠹ Initializing project in the cloud...

CREATE_IN_PROGRESS amplify-sqac-amplify-prod-161346 AWS::CloudFormation::Stack Sat Feb 01 2020 16:13:48 GMT-0800 (Pacific Standard Time) User Initiated
CREATE_IN_PROGRESS DeploymentBucket AWS::S3::Bucket Sat Feb 01 2020 16:13:51 GMT-0800 (Pacific Standard Time)
CREATE_IN_PROGRESS UnauthRole AWS::IAM::Role Sat Feb 01 2020 16:13:51 GMT-0800 (Pacific Standard Time)
CREATE_IN_PROGRESS AuthRole AWS::IAM::Role Sat Feb 01 2020 16:13:52 GMT-0800 (Pacific Standard Time)
CREATE_IN_PROGRESS UnauthRole AWS::IAM::Role Sat Feb 01 2020 16:13:52 GMT-0800 (Pacific Standard Time) Resource creation Initiated
CREATE_IN_PROGRESS DeploymentBucket AWS::S3::Bucket Sat Feb 01 2020 16:13:52 GMT-0800 (Pacific Standard Time) Resource creation Initiated
CREATE_IN_PROGRESS AuthRole AWS::IAM::Role Sat Feb 01 2020 16:13:52 GMT-0800 (Pacific Standard Time) Resource creation Initiated
⠹ Initializing project in the cloud...

CREATE_COMPLETE UnauthRole AWS::IAM::Role Sat Feb 01 2020 16:14:06 GMT-0800 (Pacific Standard Time)
CREATE_COMPLETE AuthRole AWS::IAM::Role Sat Feb 01 2020 16:14:07 GMT-0800 (Pacific Standard Time)
⠇ Initializing project in the cloud...

CREATE_COMPLETE DeploymentBucket AWS::S3::Bucket Sat Feb 01 2020 16:14:13 GMT-0800 (Pacific Standard Time)
⠸ Initializing project in the cloud...

CREATE_COMPLETE amplify-sqac-amplify-prod-161346 AWS::CloudFormation::Stack Sat Feb 01 2020 16:14:15 GMT-0800 (Pacific Standard Time)
✔ Successfully created initial AWS cloud resources for deployments.
✔ Initialized provider successfully.

You've opted to allow users to authenticate via Google. If you haven't already, you'll need to go to https://developers.google.com/identity and create
an App ID.

Enter your Google Web Client ID for your OAuth flow:
Enter your Google Web Client Secret for your OAuth flow:
? Do you want to configure Lambda Triggers for Cognito? No
Initialized your environment successfully.

Your project has been successfully initialized and connected to the cloud!

Some next steps:
"amplify status" will show you what you've added already and if it's locally configured or deployed
"amplify add " will allow you to add features like user login or a backend API
"amplify push" will build all your local backend resources and provision it in the cloud
“amplify console” to open the Amplify Console and view your project status
"amplify publish" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud

Pro tip:
Try "amplify add api" to create a backend API and then "amplify publish" to deploy everything

Then push up the new environment:

$ amplify push
✔ Successfully pulled backend environment prod from the cloud.

Current Environment: prod

Category Resource name Operation Provider plugin

Hosting S3AndCloudFront Create awscloudformation

Auth sqacauth Create awscloudformation

Storage storage Create awscloudformation

Function S3Trigger08755fbf Create awscloudformation

Api sqacamplify Create awscloudformation

? Are you sure you want to continue? Yes

GraphQL schema compiled successfully.

Edit your schema at /Users/adamfanello/dev/sqac/sqac-amplify/amplify/backend/api/sqacamplify/schema.graphql or place .graphql files in a directory at /Users/adamfanello/dev/sqac/sqac-amplify/amplify/backend/api/sqacamplify/schema
? Do you want to update code for your updated GraphQL API No
⠹ Updating resources in the cloud. This may take a few minutes...

... Lots of CloudFormation output as everything is built ...

Category	Resource name	Operation	Provider plugin
Hosting	S3AndCloudFront	Create	awscloudformation
Auth	sqacauth	Create	awscloudformation
Storage	storage	Create	awscloudformation
Function	S3Trigger08755fbf	Create	awscloudformation
Api	sqacamplify	Create	awscloudformation

See pull request part8/go-live.

With this in complete, I installed the seed data into S3 (with the S3 trigger automatically indexing it into DynamoDB) and the new SqAC is now live! One long (weekend warrior) journey migrating a legacy app to cloud-native AWS complete! 🎉

Conclusion

As you saw, doing a clean install to a new environment got past the failed stack. My celebration felt tainted though, with the dead stack still staring at me in bright red on the AWS Console. While creating a fresh production environment was probably the right thing to do anyway, it is not the solution to every problem. Developing and maintaining a software system over months and years involves updates. Updates that can’t lose users and user data. Being unable to find the root cause of the broken stack makes me very nervous about trusting Amplify with user data. There’s much good and much promise in Amplify — the new Amplify DataStore feature looks a-maz-ing — but my final experience leaves me hesitant to use or recommend this toolset. I waited several weeks to write this final post. Part of that delay was to give the Amplify team time to discover an explanation. Perhaps the bigger reason is that, after seven months on this exploration, I really did not want it to end this way. 😕

In the end, my final conclusion about using AWS Amplify is simple:

There are no shortcuts

Amplify comes across as a magic toolset to let frontend developers create applications without being backend developers or cloud engineers. This simply isn’t the case, and the members of the Amplify team that I have met tell me that this was never the intent. It is a toolset to help with the minutiae of managing and using cloud infrastructure. As my journey and (excruciatingly detailed) blog posts have shown, you still need to understand the AWS services behind everything Amplify is helping you with. A tool is only useful in the hands of someone who knows how to use it, and there is truth in the phrase: “I know enough to be dangerous.” My advice for app developers looking to leverage the cloud: learn the full stack or team up with a solutions architect. Find an AWS Meetup group in your area and do some networking. Find a consultancy that can #thinkcloudnative. AWS is still the most comprehensive platform on which to develop, so go forth and create!

(This story was original posted here in March 2020.)

Adam.Fanello<Building Apps in the Cloud>

Adam.Fanello<Building Apps in the Cloud>

Migrating a Legacy App to Cloud Native — Part 8

Custom Domain Name

Then, Disaster

Starting Over

Conclusion