Recently there's been a bit of controversy that microservices may be a mistake, and monoliths are (once again) the way to go. This began due to a blog post from the Amazon Prime Video team indicating the great cost reduction they realized by switching from various AWS serverless services into a containerized “monolith”. Fireworks lit up the AWS community! Is the Amazon Prime Video team really telling us to dump serverless and microservices?

A key line from the blog post comes at the beginning of the second paragraph:

Our Video Quality Analysis (VQA) team at Prime Video…

They did not turn all of Prime Video into a monolith and throw it all into a single ECS container! Rather, this is one team responsible for a relatively small functionality of the overall application. Isn’t that the exact target of a microservice?

Microservices are an architectural and organizational approach to software development where software is composed of small independent services that communicate over well-defined APIs. These services are owned by small, self-contained teams.

— https://aws.amazon.com/microservices/

What we have here is a classic growing pain of greenfield application development. The team broke their functionality down too far into deployed nanoservices. When they ran into trouble, they reorganized the way they deploy and brought their functionality back together into a single Video Quality Analysis microservice.

Finding the right balance between deploying a monolith, microservices, and nanoservices is almost impossible to predict upfront when developing new applications. If you are writing a new application, how do you mitigate this unknown?

How to find balance

The first key is to separate your software architecture from your solution architecture.

Software architecture is how developers organize code.

Solution architecture is how developers (or DevOps) deploy code.

Far too commonly, a stringent solution architecture is defined first. The solution is broken down into microservices from the beginning. Walls are put up between teams, and each team goes to their separate corners and write their software. This is a proven technique and comes from a need to manage large applications and groups of developers. Go ahead and do this for really obviously separate parts of your applications.

But when in doubt, don’t.

Putting up walls too early is like premature optimization. Just as you don’t know early on where the most bang-for-your-buck can be found in optimizations, you don’t know yet the best way to deploy your application.

The way to find balance is to separate the concerns of organizing code and communication, from deploying code and communication channels. This is where the software architecture comes in:

Write everything as nanoservices, but deploy them as a monolith

Now that I have your attention, here’s the more nuanced version:

Write everything you can as nanoservices, but deploy them initially as a monolith unless you have a clear reason not to. As needed, refactor what code is deployed where.

How to write software independent of deployment

The software architecture field is highly mature and based on principles that have evolved for roughly fifty years. In the euphoria of cloud solution architectures, containers, and serverless, the architecture of the software within these solutions has sometimes been overlooked. This approach outlined below tries to encapsulate a smorgasbord of software architecture principles:

• Information Hiding, least knowledge, single responsibility, high-cohesion and low-coupling, dependency inversion, interface segregation, Liskov substitution, open-closed, separation of concerns.

This list may be long, and some may recognize all five SOLID principles, but they all relate together such that every decision plays a part in satisfying multiple principles.

Everything Quietly Does One Thing Well

Hiding Implementation Detail

Right down to the class level (source file or module), write that class as though it were a nanoservice itself. The public API on that class is always written from the client’s point of view. (The client being any other class that uses it.)

For example, take the classic layered architecture repository classes. A repository manages a collection of data. The actual source of the data could be anything: a relational database, a NoSQL database, another microservice, or even hardcoded mock data. In true microservice thinking though, the details are hidden away from the clients of this class. As part of this information hiding, the class is named “WidgetRepository”, not “MySqlWidgetRepository” or “OtherServiceProxyService”. From the perspective of anything using this repository, it’s just the repository for a collection of data and the class and function names reflect that perspective.

Each Feature Is Independent

Every feature is a use case, and has its own independent code module. It's triggered by an event or request, executes one cohesive piece of business logic, and responds to a request. It can use repositories to manipulate data, without a care to where that data lives. If it does something noteworthy enough that something else might need to happen as a result, it published an event about it. Done. You might have all of these extend a base class, which will help for tying it all together later:

abstract class UseCase<generics Message, Response>
  abstract process(message: Message) return Response

This means “AddWidget” is a separate use case class module from “DeleteWidget”, etc.. A classic layered architecture would have these put together into a single all-knowing “WidgetService”. That approach though usually results in a huge “do everything” class that violates multiple design principles.

Once every feature is its own little module, these little use cases can be deployed in any way without them knowing or caring. Group them in folders by domains (i.e.: src/use-cases/widgets/), but initially KISS and deploy as a monolith. When you have a reason, deploy some separately, but don't move the code!

Use dependency injection to provide the external interfaces, such as “WidgetRepository”. (This also sets up easy unit testing on the all-important business logic in these use cases.) Depending on the capabilities of the programming language, it is best if the domain use cases define these dependencies as interfaces or abstract classes. Binding the interface to the actual implementation is a deployment consideration and key to making your use case nanoservice portable. You may initially inject a “WidgetRepository” implementation that accesses a database directly. Later, after deciding to break this use case out to another microservice, that deployment can bind a proxy implementation of “WidgetRepository” while other use cases that remain in the monolith continue to use the database implementation.

How you organize your domain code is independent of how you deploy the code! It's easiest to group related source code near each other, but that is not always the best way to run code. These are two different concerns, so don't let one force unnatural structure on the other.

Tying it Together

Message Brokers and Buses.

Use cases register themselves, or are registered, as handing a single specific message (event or request). In a monolith, this might be to an in-process bus. Events are published to the bus, and the bus routes to the use case. For a more distributed (microservice or serverless) system, the bus is replaced with a message broker. (Examples: ApacheMQ, RabbitMQ, Kafka, Amazon EventBridge, Amazon SNS, …)

What if you have a distributed system, but an event is published in the same process as the subscriber? Should optimize that? Probably not. If ultra-low latency is critical, then yes let the in-process bus recognize and optimize this. In most cases though, go ahead and send the event to the broker and let it come back. This lets the system work as intended, with queuing, failure and retry, archiving, process lifecycle, etc. all provided by your solution architecture. Also, this allows for other external subscriptions to also receive the event and maintained the option to split out that in-process subscriber without any other changes.

That brings us back to where we started:

Each use case can be its own nanoservice. Always code as if it is.

What is a Microservice?

Under this approach, a microservice is a deployment of a group of nanoservices (use cases). Along those lines, a monolith is a deployment of all of your nanoservices. By coding in nanoservices, you maintain full agility in how you deploy.

It Works For Us, And Can for You Too

I’ve used this approach on a couple of projects now. In one unusual case, the application was originally written to deploy as a serverless application on AWS (Lambdas, DynamoDB, SNS, API Gateway). Upon business request, it was also deployed as a pair of monoliths in a cloud virtual machine and a laptop. This was done without changing any of the use case code! That’s the power of separating software architecture from solution architecture.

This approach is built into an accelerator platform created and used by the Rackspace Professional Services Cloud Native Development team. Let us accelerate your next application.

Originally published on the Rackspace Technical Blog

First Timer?

Going this year, but have never attended in person before? Rackspace’s Chief Technology Evangelist Jeff DeVerter, Director of Cloud Native Development Matt Puccio, and I chatted about How to Survive AWS re:Invent. Watch for our collective tips:

Game Day!

Monday morning at re:Invent is Game Day! There are a few options, but you might want to check out this one - it looks really cool 🦄:

Knights of AI!

I'm excited to tell you that I've been part of a small team of engineers, architects, UI designers, and marketing folks to build for you a fun interactive game you can play at re:Invent. I'm really looking forward to seeing the response to this - it's going to be hot! 🔥 Prizes are awarded each day, but the real fun is how this shows off what we can do with Machine Learning and the type of software design I like to go on and on about: cloud native, serverless, event-driven, clean code, and automated testing!

Pictures say more than words, so what a wealth of knowledge these 54,000 pictures streamed into a video must be! (3 minutes @ 30fps)

Watch this behind the scenes video about Knights of AI:

Marketing put out a little clip of my personal invitation to you too:

Come Visit

I’ll be hanging out in the Venetian Expo for much of re:Invent - primarily at booth 244. I’d love to chat cloud native software architecture and engineering with you. I’ll do an official presentation about how we built Knights of AI at some time, in the booth. (Follow me on LinkedIn and I’ll post the day and time when I know it.)

In March, I made the choice to make the attempt at becoming an independent consultant. I knew it wasn’t the safest or easiest option and that I might not succeed, but had a drive to try and knew that I could easily pick up another employer if it didn’t work out.

Nearly six months later, I had generated content, got matters in place, chased the dream… and nothing. I went public with that dream, and had a great deal of encouragement but still no clients.

Since last March, the economy and job market has changed. A recession is on the horizon and some tech companies have hiring freezes and even layoffs. Uh oh. 😬

I considered a full gamut of options…

Independent

My first dream was going independent. I’d been working for a consultancy for years, and the idea of cutting out the middleman is highly appealing.

Throughout the year though, I discovered that the timing was bad:

• The economy is diving into recession, and we’re seeing layoffs and freezes in the tech field.
• Little contract demand right now; those hiring want full-time positions, often in old tech rather than the leading edge that I’m accustomed to.
• What contract jobs I found are commoditized to lower rates, often entry-level.

I thought recruiters would be helpful in finding contracts. One of my driving forces was a report by one of these recruiting companies showing high contract rates. That very company has proven useless, and I suspect the numbers in the report are “up to” numbers, not typical rates. I never saw a single solid contract lead from a recruiter.

My own leads have not lead to contracts, including the client that I had hoped to launch with.

Aside on Recruiters

Nearly all recruiters are clueless. I don’t wish to be mean, it’s just what I have experienced and should be expected. Almost none of these kind folks actually understand the jobs they are recruiting for. My experience is clearly in application development - I just happen to use the AWS ecosystem to create these applications. Recruiters see my AWS Solutions Architect certification though and say “Ah ha! DevOps!” or “Ah ha! Security expert.” They see my desire for “consultant”, and offer me full-time jobs. They see “24 years of professional experience”, and suggest mid-level software engineer positions. (At least they clue in to the software portion.)

I want to love recruiters. I want them to be my friends and allies. But they aren’t. Recruiters are exhausting. 😮‍💨

Other Consultancies

Some of my frustrations with my current employer, Rackspace, are typical frustrations when working with large companies. So I checked out smaller consultancies.

The smaller consultancies appeared to be more like Onica, the company I worked for that Rackspace swallowed up. Each of these though, were also DevOps shops just trying to get into the (cloud native) custom application business. This carries the risk that they may not have a sales pipeline, leading to being idle and the possibility of them giving up on this new offering. In one case, it looked like what they wanted was help in the “dev” part of DevOps. That isn’t very interesting to me.

Finally, I would need to rebuild my reputation within the new company.

All that risk, for the same compensation I have now.

Product Companies

Many of my former colleagues have left consultancies to work for product companies. I came from product companies.

Since working for a consultancy though, I have honed specific specialized skills. As a consultant, companies will hire specialists. When looking at full-time employment at product companies though, few wanted my specialization. The work is more adjacent - moving into tech stacks that I don’t care so much for. I could do them, but it would mean more generalization and giving up my subject matter expert status.

Then there’s compensation. Here’s how the jobs and compensation with product companies compare to what I’m earning now at Rackspace:

1. Slightly lower compensation for same work.
2. Same compensation to move up in management. (i.e.: V.P. of Engineering for small company.)
3. Higher compensation if I compromise my ethics. (i.e. planet harming industry)
4. Higher compensation if I compromise my health and family. (i.e. hello 60 hour work week)

The grass isn’t greener

There is a common phrase: The grass is greener on the other side. This phrase is actually a warning, that things appear better elsewhere. We are tempted by “greener pastures”. We are envious of anecdotes of others who have made the leap and found success. Often these success stories hide the downsides (see bullets 3 and 4 above).

I have taken a closer look at the neighbors’ lawns (employment options), and found that they too have flaws. Looking back on my own place, I find that maybe I can achieve my goals right here…

Stay at Rackspace… with change

I had written in my announcement about going independent:

I looked at moving out of management over to a Principal Software Architect role, but […] that new poorly defined role I suspected would be manager-lite.

and

I will keep learning and growing and, I hope, leave a wake of younger software engineers who have learned and grown as well.

and

Charting my Own Path

Solution: As a last act as manager, define the role as exactly what I’m looking for!

It would be hilarious to create the job description, open it, apply for it, and hire myself. The boss didn’t quite go for that, in part because the existing broad definition of the role was actually already a good fit for what I wanted, and I guess we aren’t allowed to hire ourselves. 🤷‍♂️ 😉

New role

So what will I do as a Principal Software Architect of Cloud Native Development? The overarching concept is “be a force multiplier.” Rather than spend all my time delivering for one customer, a principal architect has a team-wide impact. This includes:

• Evaluate tools and services, and spreading knowledge both within and outside the team.
• Create and evolve reusable solutions.
• Be a mentor for the entire team.
• Be an expert troubleshooter on any project, and ensure they stay on track.
• Work with pre-sales, helping set up projects for success before they even begin.

Did I backtrack?

You said you were doing something, and now you’re not! Flip flopper!

Fortunately I’m not in politics, and so I’m allowed to change my mind. 😜

I continually evaluate. Circumstances change. Knowledge grows. Course corrections are made. I may have “backtracked” on going independent, but not on charting a new course in my career.

What is changing for me?

I had been feeling idle and unfocused. This new role has reinvigorated me.

At the time I went into management, it was the only way to advance. Last April Rackspace opened up the new non-management career path. Even though I have grown competent in the manager role, it never fit into my self identity. It turns out that titles do matter; I never liked calling myself a Practice Manager, but am very happy to start calling myself a Principal Software Architect. Titles matter because self identity matters.

When not part of a delivery team, as a manager instead, I felt more isolated. The hope that this new role will offer more opportunity for human interaction. I had blamed the isolation on lost company culture, but this is just a reality of the pandemic and post-pandemic work-from-home era. I love working from home and never liked the distracting nature of the office, but there is a psychological impact. There’s a newly forming Rackspace Culture Team, and I have volunteered to join this team so that I can be part of the solution. 🤞

The great resignation has shaken us all, and resulted in brain-drain everywhere. I think this chaotic shuffling of jobs is easing now (I was almost part of it), so now it is time for recovery. As part of my new role, I hope again be part of the solution and make Rackspace a great place to work by being a resource for everyone. I hope to reverse the talent loss by spreading my reach and helping my fellow Rackers grow.

Finally, Rackspace itself is ever evolving changing. There’s a new CEO who sounds more cognizant what needs to be improved, and a global reorganization about to land. There’s also the global and local culture teams. I’m hopeful and energized again.

I ran across this blog post some time ago:

Four reasons why everyone except for Computer Scientists writes sloppy code by Ari Joury

It’s a really good write up, except that it starts right off in the title insulting a whole lot of developers. Joury doesn’t define who a Computer Scientist is. It is only someone like me with a degree that says “Bachelors of Computer Science”, or is it anyone who thinks like a computer scientist? I believe it is the latter: how someone thinks. However, in that case the very label makes no sense. This is exemplified in Joury’s Reason #1, which hits the nail on the head:

“Reason 1: For Computer Scientists, coding is an art. For everyone else, it’s a tool.”

• Ari Joury

I read this line and my head popped. 🤯 I’ve seen this sentiment before, but never so succinctly. But there’s a problem with it. If you study physics, chemistry, biology… you would be studying science. You would be pursuing knowledge of what is. As software developers though, we might study the problem space of what we are trying to solve with our solution, but then we go and create something new.

So I’ll modify this reason:

“Reason 1: For passionate software developers, coding is an art. For everyone else, it’s a tool”

• me

Passion = Art!

How does a passionate software developer differ from everyone else?

Everyone else will look at the requirements, perhaps with a user story, and dive in writing code. Tweak it. Stack Overflow it. In the end, the output is something that “works.” That’s the end for these “users of a tools” people.

“It works for that user story, therefore it is done.”

• those people

This code is usually easy to spot, because it will be a jumbled mess of spaghetti. That’s evolution. Does it work? Yes, at least for that one use case. Can it be maintained? Probably not.

A passionate software developer would be embarrassed by that. How can you be passionate about your work and deliver a mess? A code artist might go through the same evolution but then it will be refactored into a testable succinct piece of art.

More often though, a code artist won’t have just blindly started typing code. Writing is a form of art. Like any novelist, the artist will outline the overall plot first. The characters and interactions are identified and the story obtains some structure before a single sentence, a line of code, is written.

If you have a code artist doing your code reviews for a while, you will experience this passion for the art of programming. I hope you will come out of the experience as a matured artist yourself. We will challenge you. Either we will convince you that our artistic style is best, or you will rise to the challenge of standing up for your own artistic approaches. Or, you will remain a “it works therefore it is good” user of tools and hate all artists.

“You will get better every day. It is an art, not a science. It is a mixture of your creativity, your personality, your heart, your mind, your ethics and your values. It is you.”

• James Stanier

Stanier was talking about being a manager, but this is true of anything you care about.

Audience

Reason 2: Developers don’t always write with the reader in mind

• Ari Joury

This calls back to the point that a code artist is a writer, and we’re writing to two completely different readers of the same manuscript: the computer and our fellow developers. The computer doesn’t care one bit about English and is perfectly happy with this program:

static int e,n,j,o,y;int main(){for(++o;(n=-~getchar());e+=11==n,y++)o=n>0xe^012>n&&'`'^n^65?!n:!o?++j:o;printf("%8d%8d%8d\n",e^n,j+=!o&&y,y);}

Source: International Obfuscated C Code Contest

Our fellow developers care only a little about the programming language syntax; really it’s English that we read. What we name things is most important here, as well as how we organize things. This is what makes code readable, and not just functional.

Conclusion

So what’s my point? Have passion for your work! Go forth and create art!

(Watch out for stormy clouds ahead - photo by SpaceX)

The Outside Team

When asked that question, nearly all consultants jump into how to move your application. They’ll talk about lift & shift vs cloud native, microservices, serverless, domain-driven and event-driven designs. They are right to do so; these are important decisions to make when modernizing your applications.

An outside team will build or transform your product on their own and drop it in the lap of your in-house engineers, who are then lost. What is this cloud magic? How does this work?

Here’s the downside. Outsiders now know your product better than your in-house team. Without having developed that deep understanding, they won’t know how to maintain and enhance the work those experts built. Worse perhaps, they will break what was built by reverting to old ways and thus you loose the advantages that the outside team brought.

Big consultancies usually give little thought to your engineers; they are out to sell services and they are motivated to make you dependent on their services. They would have you keep your existing engineers only to maintain your old products as they attrition away. Then you have to keep coming back to the consultancy for more.

Having lead that outside expert team for several consultancy customers, I have seen time and again how ill prepared my client is at the end of the engagement. A few hours of knowledge transfer meetings can’t impart the skills that come from experience.

Use Your Existing Team

Your engineers need to learn how to architect and program the new cloud native way, that’s the only long term play.

Training courses will teach developers the new concepts, but often leave them unsure how to actually proceed with a real enterprise-level project. It then becomes trial and error, and stumbles, as they find their way to something that “works”. Before that first thing works, they will discover something that would have been better, but it’s too late to course correct.

Your developers aren’t bad, but they do learn by doing. As with anything we humans learn, we develop better skills and techniques as we practice. While learning, their lack of experience can lead to the same old problems of costly bills, difficulty scaling, and security holes. These problems can hobble your product for a long time to come, leading to another big redesign effort.

Embed an Expert

A third approach is to embed an individual cloud application development expert into your team. This expert can guide your in-house team and accelerate the project. An expert has gone through this trial and error many times already, so they skip the false starts and go right to a solution for you based on their vast experience.

Embed that expert as a member of your team, and build your new product together. Your engineers learn by doing, you have a product designed and guided by a cloud expert, and in the end your engineers understand the resulting work because they were involved in it every step of the way.

I have had only a couple of “build with” clients in my time at AWS consultancies, and always feel better about the handoff of these projects. I expect these applications to live on and continue to grow for years to come.

Inspired by: Laying Out a Road Map to Close the Cloud Skills Gap - InformationWeek

Throughout my career as a software engineer, certain nuggets of wisdom have come my way. I may pick some up from co-workers, from reading blogs, or often enough the old fashioned way - learning from my own errors.

I have tried to pass along what I have learned, to train newer software developers on the good things to do when writing applications, what not to do, and why. My key phrase I have latched onto is “maintainable code”. I’ve written my own guides on the subject; internal to companies, in public blog posts, and most often as comments in pull requests!

Over the decades, my expertise in my craft has grown and I’ve tried to give more back.

Then… something happened. I met Uncle Bob.

Not in person, unfortunately, but somehow I found myself to his web site, videos of his speeches, and finally to his book: Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin.

I have been wasting my time. 😉

This is the holy book of programming. All that I have discovered, is right here. In fact, this is the source of much of the advice I received over my many years.

I had heard of this book before, certainly, but actually read a book? On paper? For so long I have been learning online that this seemed quaint. Well I’m glad I finally gave in. I’m here to say: read this book! On Paper!

But why?

First off, after twenty-five years or so, there’s little in Clean Code that I haven’t heard before. So why read it now?

1. Uncle Bob doesn’t just say what to do, but convinces you why you should do it.
2. It’s all the advice on how to craft code in one referenceable manual.

That first point is where the biggest value has been for me, because even though I knew most of this advice, I wasn’t following all of it.

The second point make my goal of training up developers easier. Much of what I have written, and have been planning to write, is already in this book! When I see someone in need of learning, instead of writing a dissertation, I can just ask for their mailing address and send them a copy of Clean Code!

Here’s the crux of it: this is a shortcut to performing as a top-tier senior engineer!

• Buy this book (don’t borrow - you want to have it handy).
• Buy it on paper. (Flipping pages to reference code example isn’t so easy in a PDF, and e-books mutilate formatting.)
• Read it.
• Put what you learned into practice.

Could it be better?

Clean Code has one big flaw prevalent throughout: Java.

The author’s mindset is in Java. The examples are all in Java. Not just the code examples, but even the text assumes you know common enterprise Java concepts and tooling.

Back when Clean Code was written in the first decade of the 2000’s, Java was the language of enterprise application development, so it makes sense. Ten to twenty years later though, the book frequently refers to Java ecosystem concepts that won’t make any sense to someone who didn’t go through that era. It’d be great if new editions where made for other modern programming languages.

For the most part though, these Java-centric concepts can be glossed over by the reader. You could be writing in anything from C onwards and easily apply the lessons here, because most of them are timeless and most languages follow the same concepts. True functional languages may be the most distant from Java, but still I think even these programmers will find valuable lessons in Clean Code.

More Must Reads?

Are there other must-read books for software engineers and architects? Put it in a comment.

I come from an object-oriented programming (OOP) background - it was the hot stuff in my formative years of learning to program - driven by C++ and Java. Embracing unit testing meant embracing Inversion of Control (IoC) and Dependency Injection (DI). This was the way.

As my language of choice switched to TypeScript, and passing data as objects defined by interfaces passed through HTTP APIs, real OOP classes fell away but I continued to use classes for DI. Business logic was in service classes, which were really just collections of functions with shared DI in the constructor. Unit tests mocked out all those dependencies, and with pride my coverage often hit 100%. I tested my code against unit tests first, so that everything usually worked upon first deployment. I largely forgot how to use the IDE's debugger.

All was good.

I had heard the fans of Test Driven Development (TDD) exclaim the wonders of it, but figured my test before deploy approach basically did that. (Turns out, not really.)

I had heard the fans of functional programming (FP) exclaim their superiority over OOP. But hey, I'm just using classes to organize my functions for DI - so I was basically functional. (Yes, but no.)

I did not see the connection between these. I didn't fully understand either one, and I knew that. Occasionally I'd read a blog article about TDD or FP to try to understand the excitement. I wanted to find these techniques exciting! But everything I read was so basic that I couldn't see how to apply it to real problems, and they didn't show how to connect the concepts. Or worse perhaps, TDD was completely misrepresented and looked horrible because doing it wrong is horrible.

I'd think: functional is great for pure functions, but how do you test the orchestration that isn't pure?

What really is Test Driven Development?

Robert C. Martin (Uncle Bob) covers a lot of ground about software development practices in his talk about Expecting Professionalism and specifically TDD here. (Watch these.)

The analogy to the accounting practice of double-entry bookkeeping especially resonated for me. Double-entry bookkeeping is the discipline of keeping a transaction ledger for each account. When you subtract from one account, it must be added to another - immediately. Often this other "account" isn't one in the traditional sense, it is just a means of tracking and checking your work. It's a checksum. Subtract here; add there; compare the totals. If you make a mistake, the bookkeeper immediately knows.

We often write all the code, and then write all our tests to check it. Bookkeepers don't. They subtract an entry here, and add it there. It's an atomic operation. Why? Because if they did all one and then the other and make a mistake, the checksum at the end is simply "it doesn't match". Where is the mistake? It's somewhere in one of the many transactions they added. That's painful, and entering all transactions atomicly is less so.

TDD is like double-entry bookkeeping. The rules are put in two places, atomically. Write test code, then write some production code. Checksum. Write some test code, write some production code. Checksum. It may sound painful, but if you introduce a bug you will know it immediately. It's the ultimate linting tool. How painful is this compared to a bug in production? Once you get used to it, not at all. This is the #1 TDD promise: Bug free code! (Caveat: You have to understand the desired behavior, of course. Otherwise you'll write the wrong behavior twice.)

Here's the #2 TDD promise: Your code will be cleaner, easier to maintain, and self documenting. Watch the videos and find the arguments for this promise.

If Uncle Bob and I finally convinced you that Test Driven Development (TDD) is amazing, follow it up with Ian Cooper's dedicated talk on the topic: TDD, Where Did It All Go Wrong (Yes, this is actually pro TDD! He addresses where people get it wrong.)

Next in my research, I discovered this guide. No longer a simplistic example. TDD and FP rolled up in an example that is big enough to hit real challenges, and explained progressively rather than just an end result. But.... well... what's with all the factory functions? Everything is a factory now. 😬

That last article linked to this original guide to testing without mocks . Here's another concept that ties in neatly with TDD, but this one is using classes instead of functions. The classes though, still involve factories. Also, the examples are showing React code - reminding me that React components themselves use the factory pattern, whether functional or class-based. I don't know why he wants to go entirely without mocks (to the point of running a fake server in-process - just mock it already!), but the takeaway is that mocking (and faking) can be a final option rather than the first go-to choice.

What is Functional Programming?

Go full functional, or use classes? Either work in JavaScript, to an extent, but that's something standing out in these examples: they are both JavaScript not TypeScript. To some extent, I can see that powerful argument for strong typing is weakened when doing proper TDD: failures happen by default. That makes me think maybe I can allow implicit any (turn off that eslint rule) and not define types for everything, but there are still benefit to strong types. Are types more difficult with FP? There does appear to be a correlation between FP and going full dynamic typing. 🧐

When to use class vs function? This article about applying CleanCode to TypeScript helped me realize it is not an either-or choice. Looking at the patterns, I realize I can lean functional, but enjoy class and real OOP goodness where it makes sense. In short: classes are for object-oriented logic, modules are for grouping related functions.

But wait, you a FP practitioner shouts! That isn't what functional programming is about! Yes, Uncle Bob has set me straight in his talk about Functional Programming. Actual FP is about being stateless. A Pure Function is one with no side effects - given the same input it will always have the same output. Pure. Testable. Stateless. Often recursive. In fact, FP purists tell us that functional has no state, even local variables, and no loops; recursion takes the place of both and for this not to be a poor developer experience you need a language designed for this. (Lisp, Haskell, Clojure, ...) Simply using just functions instead of classes doesn't automatically mean FP. Really, applications can't be entirely FP; only functions can be pure. An application with no state is just a big function and has pretty limited use. Also, we're talking about JavaScript here, which not a real FP language, but rather a hybrid. Functions don't have to be pure. We even have classes (functions plus state).

Can we do FP in JavaScript? Yes, the Array functions like map and filter are functional. The popular RxJS library is also functional, and if you have worked with it you have an idea of how complicated functional can get. The big dog in TypeScript functional is the fp-ts. Even more than RxJS, it requires massive buy-in by the entire application team to learn and use this approach.

So am I looking at real FP, or just not using the class keyword?

So... TDD?

I'm buying what TDD is selling. It sounds doable, and individuals can do it even on existing projects and without forcing every other developer of an application, now and in the future, to understand and use it. (Indeed, I have started doing this!)

Not sure how to get actually get going?

Here's the Three Rules of TDD:

1. Write production code only to pass a failing unit test.
2. Write no more of a unit test than sufficient to fail (compilation failures are failures).
3. Write no more production code than necessary to pass the one failing unit test.

Here's some Live Stream Examples doing TDD in JavaScript.

So... FP?

For me, using TypeScript, no.

Kent C. Dodd's arguments for a functional approach is worth a read. I don't buy into the "this is too complicated" argument, I think he's just allergic to to the class keyword, but stateless programming with pure functions instead of OOP has some merit. Not only does it make testing easy, but it plays real nice with using objects defined as interfaces, instead of classes. One might implement the builder pattern with a class though, for some logical transformations.

Should we scope functions at the module (file) level instead of class? Here's how we might do Inversion of Control (IoC) without any fancy Dependency Injection (DI) magic:

function a() {
}

function fakeA() {
}

function b() {
}

function fakeB() {
}

export function moduleFactory(
    otherModuleDependency = otherModuleFactory(),
) {
    return {a, b};
}

export function fakeModuleFactory() {
    return {a: fakeA, b: fakeB};
}

Much FP code is actually allergic not only to the class keyword, but even function! So we might export our factory function as the module default like this:

import UserRepository from "user.repository";

export default (repo = UserRepository()) => {
    return {
        a: () => {
        },
        b: () => {
        },
    };
}

To me, this is getting less readable than a nice export class UserLogic with constructor DI.

dvlsg on Reddit perhaps sums up the functions (closure) vs class debate best:

Some people just prefer using closures to using state on classes, and don't like using 'this'. That's really it. It's a valid opinion, but it is just an opinion.

Then there's Dax Raad on Twitter:

a fundamental tradeoff that no one talks about with functional programming is the more you use it, the more annoying you become

FP is a big shift. Going all the way with it will make your code unmaintainable by most of the programming community, and so requires total buy-in. Choosing to do FP in a language it isn't native to feels even worse than trying convincing folks to switch to a proper FP language. Arguments can be made to do this, but it is a huge leap.

So... OOP?

Functional and object-oriented aren't black-and-white polar opposite options. Let's explore a little.

We can certainly borrow much from FP, just as JavaScript ES2015 did when it introduced the new Array functions like map and reduce. Tucking business logic into pure functions and immutable classes adopts some FP concepts without going too deep.

The function-only and "no mock" (which turns out to just be hand-crafted fakes) approach for DI and testing just doesn't appear to buy anything. Class constructors are replaced with factories. Here is a good example. Whether highly related functions are grouped together as returned by one factory, or grouped together as one class, doesn't make any difference. I have certainly had trouble with classes growing large, with all the logic and orchestration for a certain thing crammed together, but that can happen with functions and modules too.

The solution is not to group functions by the thing they operate on, but by dependencies they share (high cohesion) and what they do (single responsibility). Instead of having a UserService class that grows huge, have a class per use-case that operates on a user. That keeps the classes small.

Below I'm using interfaces and a (pure) guard function from "domain/models", pure functions from "domain/logic" and "utils/assertions", and coordinating it all as a use case in a clean cohesive class.

import { isUserSignUpRequest, UserSignUpRequest, UserSignUpResponse } from "domain/models";
import { newUserFactory } from "domain/logic/user";
import { assertValidInput } from "utils/assertions";

// Optionally use DI magic to gain performance of singletons:
// @Injectable()
export class UserSignUpUseCase {
    // Dependencies defaulted - tests can provide mocks or fakes
    constructor(
        auth = new AuthService(),
        userRepo = new UserRepository(),
    ) {
    }

    async process(request: UserSignUpRequest): Promise<UserSignUpResponse> {
        this.validateRequest(request);
        const user = newUserFactory(request);
        await this.userRepo.put(user);
        return {user};
    }

    private validateRequest(request: UserSignUpRequest): void {
        this.auth.assertIsNotAuthenticated();
        assertValidInput(request, isUserSignUpRequest);
    }
}

These use-case classes can also be called "Controllers" in the traditional sense. That name is often used now for RESTful path handling though, so I find UseCase is clearer.

Could this be done with module scope exporting a factory function? Absolutely. Would it be as readable? I say no.

Rules to Follow?

This is my personal take and how I intend to proceed. Your research and background may lead you to a different conclusions.

1. Use TDD, with circumstantial flexibility. (Don't be obsessive.)
2. Use types and interfaces for APIs.
3. Favor pure functions for business transformation logic - anything that takes a parameter or two, and returns a result or two without mutation or outside state.
4. Use OOP where it has value - but the object must be self-contained so that state changes only impact that object instance.
5. Use Dependency Injection (DI) with small classes that have a single responsibility and high cohesion.
   • These classes aren't OOP, just collections of functions and dependencies.
   • This can be done with modules and factory functions, but this is more effort for less readable syntax than a nice clean class; this syntactic sugar was added for good reason. Also, separating the function grouping name (class) from the file name feels more flexible.
6. If a test fake is needed, export it along-side the real code so that it can be reused by any test that depend on it.

File structure

Wrapping all this up, how might I organize the source code?

.
└── src
    ├── domain
    │   ├── logic  # domain logic functions & object-oriented classes
    │   │   └── user.logic.ts
    │   └── models
    │       ├── index.ts
    │       └── user.ts
    ├── handlers
    │   └── user-sign-up.lambda.ts
    ├── external
    │   ├── dynamodb.service.ts
    │   └── repositories
    │       └── user.repository.ts
    └── use-cases
        └── user
            ├── user-sign-up.test.ts
            └── user-sign-up.ts

The entire domain, or just domain/models, might go into a separate package to be shared with client code - these models are your API contracts.

The handlers are the entry points into your code. You might have separate handlers for different environments such as one for AWS Lambda and another for a containerized cloud. This puts you on the road to Hexagonal Architecture. Here, user-sign-up.lambda.ts just deals with unpacking the request from API Gateway and Lambda, calling the use-case, and formatting the proper response back to Lambda and API Gateway.

The external directory is the right-side of your Hexagonal Architecture - the interfaces out to external sources and targets of state.

Finally, use-cases coordinate processing of individual requests and commands.

Conclusion

These explorations of techniques are fun, but sure do make for long blog posts! The relation between TDD and FP isn't as strong as I started off thinking, but figuring out how to make unit testing work with FP was highly relevant.

Let's sum this up:

1. TDD gives you superpowers; embrace it!
2. Real Functional Programming is only practical in a language made for it, but we can take lessons from FP and apply them in other languages.
3. The class keyword is not poison, and is useful syntactic sugar beyond OOP.

Serverless Stack (SST) is the latest in tooling to make it simpler to develop serverless applications in AWS. Serverless Framework is the biggest player in this field, because it was the first and did a pretty good job. I've used it for a few years. However, when working with anything more than a few Lambdas, an API, and a sprinkling of other serverless services like SQS, pain points appear and Infrastructure as Code (IaC) becomes just as much workaround as code.

At today's SST 1.0 Conference, I gave a brief presentation about these pain points and workarounds, and how SST simplified my deployment tooling. You can watch my talk at SST 1.0 Conf here. I strongly encourage you to watch the entire conference There is amazing content in this short conference, from employees at SST and volunteer presenters such as myself.

Compare to AWS Amplify

Frequently folks pop onto the Serverless Stack Slack (where there is amazing support!) and ask how SST compares with AWS Amplify. You might know that I did an entire series about AWS Amplify and even though I started it with high hopes, and completed my effort, the series is a painful journey to behold. It is filled with road blocks, workarounds, and bugs.

Both AWS Amplify and Serverless Stack attempt to make it easier to author serverless applications. Both are still evolving and improving. Unfortunately, my conclusion is that the AWS Amplify approach is fatally flawed; it missed the mark and often makes it harder to work with AWS rather than easier. Serverless Stack ❤️ has succeeded where Amplify 😔 failed.

Can you identify some of the most common testing strategies in this industry and define what those are?

That's a huge question! Testing strategies can be broken down into a matrix of categories. First, there is manual vs automated. That's easy to understand - automated tests are written as code and can run automatically. Manual tests are often viewed as a procedure, in a human sense, in that a person performs each step of the procedure.

From those two broad categories, we then have to look at scope. What is included in the test, and what is not. Software is built on layers. Some of these are easy to see: the app running on your iPhone, vs the custom business logic running in the cloud, vs third party services like what AWS provides. With IoT, there's the actual hardware devices as well - that's another layer. When you decide how many of these layers you want to test together, you are defining the scope of the test.

When we only test one layer in isolation, that's a unit test - one unit of code. If two or more layers are tested together, that's an integration test. If all layers are tested together, that's called end-to-end testing.

We also have regression testing. Regression testing refers to automated tests that are designed to catch when something changes unexpectedly. They are often written after fixing a bug that existing tests failed to catch. Once written though, they’re just another automated test.

There are more special adjectives for different scopes of integration tests. UI tests are scoped around the user interface. API tests are scoped to start with the application program interfaces - server endpoints - down through all layers of the server.

Can you dive deeper into unit tests and explain how those are typically setup?

Unit testing allows us to focus on the smallest scope of code, and ensure that it is working correctly. This requires isolating it, by replacing the layers around it with fake or "mock" implementations. Those surrounding layers are the input into the unit being tested. We code up a variety of inputs to feed into that unit of code, and verify that the resulting output is what we want. If it is, the test passed.

This sort of isolation is only really possible for coded tests, which means automated. Because these are entirely automated and isolated, they are fast and easy to run in the various automated continuous integration pipelines like Bitbucket Pipeline, AWS CodePipeline, Jenkins, etc.

What about UI tests, can you speak to when these might be the most useful and what type of tools would best accomplish this?

UI tests are typically automated integration tests that focus on the user interface. The automated tests, which is code, interacts with the user interface like a user would and checks that the results on the screen are as expected. These work well to test entire user stories, although they aren’t limited to that.

The most popular tools for UI testing for a long time has been Selenium and some other tools that build on top of it. They’re rather difficult to work with though. For webapp testing, a relative newcomer called Cypress is far more pleasant to work with.

What about testing with hardware? Is there a lot more to define there?

Firmware and device testing has all the same kinds of considerations as software. The manufacture of hardware will be concerned about physical testing - that's hardware testing - but that is outside my area of expertise. These devices are however, programmed. The programming on hardware devices is often called firmware, but that's more of a historical artifact than a real distinction these days. Devices now have an operating system and run application software. No surprise then that the same concepts of testing still apply. The distinction though is how we isolate all the physical inputs to perform automated testing. That sometimes involves another piece of hardware to electronically push buttons and flip switches, or it may connect via a communication port and simulate inputs that way.

What experience do you have with load testing?

The most common approach: throw it into production, monitor as use increases, and react. When you're dealing with a new product bringing customers in very gradually, that isn't as terrible as it sounds. With a hyperscaler like AWS, a well architected solution will be fine.

Better of course is a proper load test, and I led this effort for an IoT customer that asked for it. In this case, they were planning to very quickly migrate tens of thousands of devices and thousands of users from a legacy platform to a new one that we had built, so we have to be ready.

Load testing is just a form of integration test, usually scoped to just your cloud-side layers. A pretty basic load test will just hit one application endpoint hard and see if or when it breaks. More sophisticated is to script an automated integration test that acts like real users. Then you can use something like AWS Fargate to scale out our simulated users to produce a heavy load. For an application with input coming from somewhere other than end users, such as IoT devices, the load test is modeled very similar to a unit test - mocking the inputs from any direction into the cloud. With a single automated test simulating both thousands of IoT devices and the thousands of users interacting with them in realistic ways, you have a real cloud end-to-end test under load.

Cloud monitoring tools can monitor the system health during a load test. The testing scripts themselves can also track latency and failures from their end and report results.

Can you talk to us a bit about automated testing, which I think might tie into test driven development, and if not, can you explain the differences?

Automated tests are written so that they can be run again and again without human involvement. That is in contrast to manual tests.

Test Driven Development is a discipline of testing whereby the programmer writes the automated unit tests at the same time as implementing the logic. We start by writing a test the way we want to use the logic, and this initially fails because the logic code doesn't yet exist. We then write just enough logic until the test passes, perhaps be writing a function that does nothing. Then we write some more test until it fails again, and write more code until is passes again. This continues back and forth until the solution is complete. It sounds absolutely crazy, but can expose nice clean solutions that we might not have thought of without writing a fake consumer of the logic right from the start. You end up with just the right amount of solution and unit tests as you need, and no more. Integration tests can be created this way as well.

While this approach takes a little extra upfront effort, when done right bugs become rare.

What can you tell me about working with external QA teams? In your experience, have you seen an efficient workflow from external QA teams?

Oh yes. Quality quality assurance members are there to find problems before the end users do. That's a valuable service! Developers can get tunnel vision; we know what we expect the user to do and code for that. You need a separate person to do the unexpected to find bugs, and try the same thing in two versions each of three different browsers, for instance.

On the other side of that, what have you seen that simply just doesn’t work?

The key to successful QA is a respectful relationship. Not only do the developers and testers need to recognize that we are there to support each other, but the customer must not poison that. When a customer is willing to pay for QA, and then is shocked and assigns blame when QA actually finds bugs, that destroys the relationship. It can't be competition.

What are the best testing tools you would recommend?

I already mentioned Cypress for UI testing of web applications. Other than that, the tools vary by programming language. Unit tests must be written in the same language as the code it tests. That isn’t required for integration tests, but often using the same tool for integration testing as the unit tests reduces the cognitive load and allows for reuse of some code.

Can you describe what the ideal test coverage looks like?

Testing should be targeted to where the smallest amount of it can have the biggest impact. If you unit test, and API integration test, and end-to-end test the same code - you’re testing the same logic three times. There are always some added benefits to more test coverage, but the returns diminish.

That said, my default for unit tests is 100% function and line coverage. That is often reached, but I’m not going to spend much time trying to hit one hard to reach branch that is not even expected to ever happen. For APIs, with the code behind it already unit tests, the target is every API but with just a representative sample of possible inputs. Maybe one success and one failure.

Therefore, half of the population is inexperienced; junior to middle-level. With five years, you're the old guy! (We frequently call them senior software engineers.)

Extend this out:

• 75% of software developers have fewer than 10 years of experience
• 87% of software developers have fewer than 15 years of experience
• 94% of software developers have fewer than 20 years of experience

I've been writing software since 1987, earning my Computer Science degree and starting my first professional job in 1998.

Young developers often ask where all the old programmers went. They wishfully think we all got rich and retired early. A few did, but that's pretty rare. Many moved to management. Some moved to project management, UI/UX design, and other related fields. The rest of us are still here, but with exponential growth in jobs we are vastly outnumbered!

Those of us with a passion for the work stick around. We continue to perfect our craft. We learn from mentors like "Uncle Bob" Martin. We mentor others, trying to level-up our younger colleagues, so that they too can become "seniors". 😉

Here's a link to the recording on LinkedIn.

(Unfortunately my video quality was terrible. Thanks Cox Communications!)

Note to authors and editors: If you use any of these phrases, I will not click!

Why did you?

Science says…

“Science” doesn’t talk. A scientific study may indicate some interesting discovery. Real science employes the scientific method of hypothesis, experimentation, and peer review. If an article claims that “science” says something, and it is missing these components or a source with them… it isn’t science. It’s either marketing or political manipulation.

One Weird Trick

This is just pure clickbait by uncreative marketing people. Anybody still using this isn’t even trying anymore.

Emotional Intelligence

This is the latest catch phrase for being in control of your emotions, and having empathy for others. Long ago we’d call it “being in touch with your feminine side”, because somehow we didn’t realize how ridiculously sexist that was. (Or just didn’t care, and thus weren’t? 🤔) Anyway, this is a new headline fad. I don’t even know what sort of articles are behind these headlines, because I refuse to click on them.

I was reading about the new aws-sdk for Javascript version 3 (aws-sdk-js-v3), and part of the advancement is the modularization of the SDK to reduce the resulting deployment package size — and by extension cold-start time.

You can read about the new v3 SDK, including the modular architecture, here.

What if you are working on a project though that is stabilizing for release, or you just aren’t ready to make the jump to v3 yet? That’s where I was, but out of curiosity I read the migration guide and learned that the first step is to convert the existing aws-sdk-js v2 imports away from things like:

import AWS from “aws-sdk”;
const dynamoDB = new AWS.DynamoDB();

into:

import DynamoDB from "aws-sdk/clients/dynamodb";
const dynamoDB = new DynamoDB();

I’m already using Webpack and Tree Shaking!

This would supposedly reduce the amount of code imported, even with v2. So maybe I could make this simple change, preparing the code base for v3 later and safely improve performance now? I wasn’t sure how much such a change would help, since the project was already tree-shaking with webpack. I was aware though that webpack doesn’t work well on aws-sdk-js… so maybe?

To try it out, I first added a new rule to tslint.json (I’m using Typescript):

"import-blacklist": [true, "aws-sdk"]

With that, tslint highlights all the places that need to be changed and bans anyone from unwittingly doing a top-level import later and undoing any benefits. Cool. 😎

Because the code base uses hexagonal architecture, tslint only found twelve files that needed to be updated. 👍 I made a few changes such as:

- import {ApiGatewayManagementApi} from "aws-sdk";

+ import ApiGatewayManagementApi from "aws-sdk/clients/apigatewaymanagementapi";

and

- import * as AWS from 'aws-sdk';

+ import Iot from 'aws-sdk/clients/iot';
+ import IotData from 'aws-sdk/clients/iotdata';
+ import {AWSError} from 'aws-sdk/lib/error';

A quick deploy and into the AWS Console I go to look in the stack deployment buckets in AWS S3 and compare the new and previous packages. What I found sealed the deal! Here area few of the results:

╔════════════╦═══════════════╦══════════╦═══════════╗
║ Stack Name ║ Previous Size ║ New Size ║ Reduction ║
╠════════════╬═══════════════╬══════════╬═══════════╣
║ ingest ║ 9.0 MB ║ 7.4 MB ║ 18% ║
║ provision ║ 4.2 MB ║ 1.8 MB ║ 57% ║
║ device-api ║ 14.3 MB ║ 6.5 MB ║ 55% ║
║ general-api║ 1.4 MB ║ 0.6 MB ║ 57% ║
║ account-api║ 5.6 MB ║ 2.4 MB ║ 57% ║
╚════════════╩═══════════════╩══════════╩═══════════╝

Overall average reduction in size: 49%

It was an easy change that took mere minutes to forever speed up cold starts.

Yes, bundle your AWS SDK!

One last note, as I’m sure someone will point out that you can reduce sizes even more by using the SDK already within the Lambda execution environment instead of bundling it with your code.

Don’t fall for that trap!

That goes against Best practices for working with AWS Lambda functions, which includes controlling your dependencies. The SDK in the execution environment is there for quick little functions with no other dependencies — often made from within the AWS Console. Relying on this ever-changing version of the SDK in your production environment means that you never know when your system will change from under you and mysteriously break. It isn’t worth it! Bundle your dependencies, but only what you need.

The code and text of this presentation are in GitHub.

Here's the details from the Meetup event:

Onica's Lead Software Architect, Adam Fanello, will introduce how to use hexagonal architecture to organize your code so that it is focused on what makes your code unique while being scalable, testable, and flexible. Theory will be put into practice by showing how an unorganized feature can be transformed

About the Presenter:

Adam Fanello is a Lead Software Architect at Onica, a Rackspace Technology Company, focused on cloud native development in AWS as well as the web app clients that use the cloud. He's been creating full stack web applications since the days of JSP.

This is part 8 in a series documenting my journey migrating my progressive web app, called SqAC, to AWS cloud native. If you haven’t been following it before now, here are the previous posts:

• Part 1: Background
• Part 2: Requirements & Architecture
• Part 3: Authentication
• Part 4: Add Cloud Storage
• Part 5: Use Cloud Storage
• Part 6: AppSync API and S3 Trigger
• Part 7: Federation & Guest Access

Now in part 8, I’ll add the final touches to the new AWS-hosted application and take it live! Also, my final conclusions.

Custom Domain Name

Hosting an application on S3 is easy — we did it back in part 3 with the amplify hosting add command. Easy, but not production worthy with a URL like http://sqac-amplify-20190817123020-hostingbucket-dev.s3-website-us-west-2.amazonaws.com

In part 7, CloudFront was added in order to have HTTPS, and we got a more succinct URL at https://d3l0j9nusq7n6r.cloudfront.net. That was better, but still not something I can ask someone to type into their browser. No, a custom domain is needed. In fact, my old application is running at sqac.fanello.net and it was time for this new cloud native version to take its place.

I first tried setting up a custom domain on my application via the AWS Amplify Console. The AWS Amplify Console though didn’t reder much new content upon clicking the Add Domain button and I found errors in the browser dev console. 😲 I suspect because this is feature centers around the idea of using AWS Amplify Console as a build-pipeline, which I don’t, and so without doing that domain management fails. Unfriendly UI, but it’s okay. This isn’t really an Amplify problem to solve; my app is really just hosted in an S3 bucket fronted by CloudFront and so I just need to find how to attach a custom domain to that. If this were just HTTP (no S), it would be a simple matter of adding a DNS CNAME record to forward my sub-domain to the AWS hosting domain. The security certificate provided by CloudFront complicates that because I can’t have a subdomain under fanello.net serve up a cloudfront.net certificate and expect a browser to accept it.

I tried Connecting to Third-Party Custom Domains, but that isn’t quite what I was trying to accomplish and so was unhelpful in itself. However, it led me to AWS Certificate Manager.

In AWS Console, I went to the AWS Certificate Manager and chose to create a new public certificate. (I did first check the pricing on such a certificate: it’s free!) I set up DNS Validation and then waited for it to happen… and nothing did.

After a couple tries, with a few days waiting each time, I eventually found that the CNAME name provided had a dot at the end that I had to remove when setting it in my DNS name server. 🤷‍♂️

When I tried to then attach the certificate to my CloudFront distribution, it wasn’t found. A quick Internet search turned up that CloudFront can only use certificates in us-east-1 (N. Virginia), something no tutorial bothered mentioning. 😞 It is on the CloudFront distribution settings page, so I would have known had read the small light grey text under the form field. Thus after a week finally getting my cert setup in us-west-2 (Oregon), I had to start over again. Fortunately I learned my lessons the first time. I removed the end dot, set the DNS TTL (Time To Live) values low, and had the new certificate setup in about five minutes.

Finally having the needed certificate, I could configure the CloudFront distribution by editing the settings. I added the certificate, set the “Alternative Domain Names” to my subdomain, turned on HTTP/2 (why is it not by default?) and lowered the price class to U.S., Canada, and Europe. Since the progressive web app is strongly cached by the browser, the slower access elsewhere in the world won’t rarely matter.

Refresh at https://sqac.fanello.net aaaand…. failure. Certificate still shows my old Let’s Encrypt issued certificate. With a delay of about a week playing around with the certificate, I forgot the initial goal! 🤦‍♂️ After setting the sqac.fanello.net subdomain as a CNAME to point to my CloudFront distribution…

I have a valid certificate from Amazon, the app loaded, but access to the S3 bucket managed by Amplify Storage is denied. Progress!

If you look back at the last blog post, after setting up CloudFront I then had to reconfigure authentication with the CloudFront URL. Now I have to change that to my new domain:

$ amplify update auth

Please note that certain attributes may not be overwritten if you choose to use defaults settings.

You have configured resources that might depend on this Cognito resource. Updating this Cognito resource could have unintended side effects.

 Using service: Cognito, provided by: awscloudformation  
 What do you want to do? Add/Edit signin and signout redirect URIs
 Which redirect signin URIs do you want to edit? https://d3l0j9nusq7n6r.cloudfront.net
 ? Update https://d3l0j9nusq7n6r.cloudfront.net/ https://sqac.fanello.net/  
 Do you want to add redirect signin URIs? No  
 Which redirect signout URIs do you want to edit? https://d3l0j9nusq7n6r.cloudfront.net/ 
? Update https://d3l0j9nusq7n6r.cloudfront.net/ https://sqac.fanello.net/ 
 Do you want to add redirect signout URIs? No  
TypeError: Cannot use 'in' operator to search for 'dev' in undefined  
    at keys.forEach.key (/Users/adamfanello/.nvm/versions/node/v10.16.3/lib/node\_modules/@aws-amplify/cli/lib/extensions/amplify-helpers/envResourceParams.js:33:19)  
    at Array.forEach (<anonymous>)  
    at getOrCreateSubObject (/Users/adamfanello/.nvm/versions/node/v10.16.3/lib/node\_modules/@aws-amplify/cli/lib/extensions/amplify-helpers/envResourceParams.js:32:10)  
    at AmplifyToolkit.saveEnvResourceParameters \[as \_saveEnvResourceParameters\] (/Users/adamfanello/.nvm/versions/node/v10.16.3/lib/node\_modules/@aws-amplify/cli/lib/extensions/amplify-helpers/envResourceParams.js:66:23)  
    at Object.saveResourceParameters (/Users/adamfanello/.nvm/versions/node/v10.16.3/lib/node\_modules/@aws-amplify/cli/node\_modules/amplify-provider-awscloudformation/src/resourceParams.js:26:19)  
    at saveResourceParameters (/Users/adamfanello/.nvm/versions/node/v10.16.3/lib/node\_modules/@aws-amplify/cli/node\_modules/amplify-category-auth/provider-utils/awscloudformation/index.js:110:12)  
    at serviceQuestions.then (/Users/adamfanello/.nvm/versions/node/v10.16.3/lib/node\_modules/@aws-amplify/cli/node\_modules/amplify-category-auth/provider-utils/awscloudformation/index.js:330:7)  
    at process.\_tickCallback (internal/process/next\_tick.js:68:7)  
There was an error adding the auth resource

Sigh. 😞 I issued Amplify CLI bug #3273. It turns out that something deleted my team-provider-info.json file, which is no longer in source control because it contained the Google Sign-In secret. I went to re-pull the environment from AWS, and thus recreate the file, but that too failed and I issued Amplify CLI bug #3274. 🙄 As a last straw, I found the old version of the file from source control, and then issued an amplify env pull. This time it worked, and prompted me for the Google authentication ID and secret, which I recovered from the Google Cloud Platform console. While in the GCP console, I realized I also needed to add the new subdomain in the list of Authorized redirect URIs, and so did so.

Then, Disaster

Everything was in place: custom domain name, SSL certificate, DNS redirection, missing file restore, and authentication reconfigure. Now was the time for the ultimate triumph! I issued an amplify push, and it froze. 😱 An hour later, the CloudFormation stack timed out and began a rollback to recover. After another hour, the rollback too had failed. The end explanation:

The following resource(s) failed to update: [OAuthCustomResourceInputs]

Huh? The top Google search hit for this error message was my own blog post from part 3 of this series! Unfortunately that problem was different. That time, there was a log in CloudWatch explaining what it didn’t like. This time there’s:

ERROR Uncaught Exception   
{  
    "errorType": "Runtime.ImportModuleError",  
    "errorMessage": "Error: Cannot find module 'cfn-response'",  
    "stack": \[  
        "Runtime.ImportModuleError: Error: Cannot find module 'cfn-response'",  
        "    at \_loadUserApp (/var/runtime/UserFunction.js:100:13)",  
        "    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)",  
        "    at Object.<anonymous> (/var/runtime/index.js:45:30)",  
        "    at Module.\_compile (internal/modules/cjs/loader.js:778:30)",  
        "    at Object.Module.\_extensions..js (internal/modules/cjs/loader.js:789:10)",  
        "    at Module.load (internal/modules/cjs/loader.js:653:32)",  
        "    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)",  
        "    at Function.Module.\_load (internal/modules/cjs/loader.js:585:3)",  
        "    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)",  
        "    at startup (internal/bootstrap/node.js:283:19)"  
    \]  
}

Searching for that turned up this issue with upgrading CloudFormation custom lambdas to Node.js 10 — something recently added by Amplify. I edited my authentication CloudFormation template to fix the relative path import as noted in the issue. The Amplify Node.js update instructions mention this change in behavior, but only in terms of our own custom functions, not the ones created by Amplify and hidden within CloudFormation templates. 😡

With that fixed, I still didn’t know how to recover. My “auth” CloudFormation stack was in the UPDATE_ROLLBACK_FAILED state. Multiple attempts to continue the rollback failed as well (after an hour each). I suspect that it was still trying to run the bad custom lambdas.

Failures like this can happen with any technology. This is why backups are important and when dealing with cloud services, we must always regard infrastructure as ephemeral. Data however, is not ephemeral. This stack contains the Cognito users. Also, as a nested stack for the entire application, it being in a failed state means the entire application stack is in a failed state.

I contacted someone on the AWS Amplify team asking for help. That contact, whom I met on LinkedIn and at re:Invent 2019, was quick to respond and pass me to a colleague on the Amplify CLI team. A couple email exchanges led simply to: “works for me”. 😢

Starting Over

Fortunately this was a development environment, and so the only data was my own. I had intended to simply roll it into production, but with the broken stack I was forced to start over with a brand new production environment:

$ amplify init
Scanning for plugins...
Plugin scan successful
Note: It is recommended to run this command from the root of your app directory
? Do you want to use an existing environment? No
? Enter a name for the environment prod
Using default provider awscloudformation

For more information on AWS Profiles, see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html

? Do you want to use an AWS profile? Yes
? Please choose the profile you want to use sqac-amplify-cli
Adding backend environment prod to AWS Amplify Console app: d2g2lwfk7ok1zm
⠹ Initializing project in the cloud...

CREATE_IN_PROGRESS amplify-sqac-amplify-prod-161346 AWS::CloudFormation::Stack Sat Feb 01 2020 16:13:48 GMT-0800 (Pacific Standard Time) User Initiated
CREATE_IN_PROGRESS DeploymentBucket AWS::S3::Bucket Sat Feb 01 2020 16:13:51 GMT-0800 (Pacific Standard Time)
CREATE_IN_PROGRESS UnauthRole AWS::IAM::Role Sat Feb 01 2020 16:13:51 GMT-0800 (Pacific Standard Time)
CREATE_IN_PROGRESS AuthRole AWS::IAM::Role Sat Feb 01 2020 16:13:52 GMT-0800 (Pacific Standard Time)
CREATE_IN_PROGRESS UnauthRole AWS::IAM::Role Sat Feb 01 2020 16:13:52 GMT-0800 (Pacific Standard Time) Resource creation Initiated
CREATE_IN_PROGRESS DeploymentBucket AWS::S3::Bucket Sat Feb 01 2020 16:13:52 GMT-0800 (Pacific Standard Time) Resource creation Initiated
CREATE_IN_PROGRESS AuthRole AWS::IAM::Role Sat Feb 01 2020 16:13:52 GMT-0800 (Pacific Standard Time) Resource creation Initiated
⠹ Initializing project in the cloud...

CREATE_COMPLETE UnauthRole AWS::IAM::Role Sat Feb 01 2020 16:14:06 GMT-0800 (Pacific Standard Time)
CREATE_COMPLETE AuthRole AWS::IAM::Role Sat Feb 01 2020 16:14:07 GMT-0800 (Pacific Standard Time)
⠇ Initializing project in the cloud...

CREATE_COMPLETE DeploymentBucket AWS::S3::Bucket Sat Feb 01 2020 16:14:13 GMT-0800 (Pacific Standard Time)
⠸ Initializing project in the cloud...

CREATE_COMPLETE amplify-sqac-amplify-prod-161346 AWS::CloudFormation::Stack Sat Feb 01 2020 16:14:15 GMT-0800 (Pacific Standard Time)
✔ Successfully created initial AWS cloud resources for deployments.
✔ Initialized provider successfully.

You've opted to allow users to authenticate via Google. If you haven't already, you'll need to go to https://developers.google.com/identity and create
an App ID.

Enter your Google Web Client ID for your OAuth flow:
Enter your Google Web Client Secret for your OAuth flow:
? Do you want to configure Lambda Triggers for Cognito? No
Initialized your environment successfully.

Your project has been successfully initialized and connected to the cloud!

Some next steps:
"amplify status" will show you what you've added already and if it's locally configured or deployed
"amplify add " will allow you to add features like user login or a backend API
"amplify push" will build all your local backend resources and provision it in the cloud
“amplify console” to open the Amplify Console and view your project status
"amplify publish" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud

Pro tip:
Try "amplify add api" to create a backend API and then "amplify publish" to deploy everything

Then push up the new environment:

$ amplify push
✔ Successfully pulled backend environment prod from the cloud.

Current Environment: prod

Category	Resource name	Operation	Provider plugin
Hosting	S3AndCloudFront	Create	awscloudformation
Auth	sqacauth	Create	awscloudformation
Storage	storage	Create	awscloudformation
Function	S3Trigger08755fbf	Create	awscloudformation
Api	sqacamplify	Create	awscloudformation

? Are you sure you want to continue? Yes

GraphQL schema compiled successfully.

Edit your schema at /Users/adamfanello/dev/sqac/sqac-amplify/amplify/backend/api/sqacamplify/schema.graphql or place .graphql files in a directory at /Users/adamfanello/dev/sqac/sqac-amplify/amplify/backend/api/sqacamplify/schema
? Do you want to update code for your updated GraphQL API No
⠹ Updating resources in the cloud. This may take a few minutes...

... Lots of CloudFormation output as everything is built ...

See pull request part8/go-live.

With this in complete, I installed the seed data into S3 (with the S3 trigger automatically indexing it into DynamoDB) and the new SqAC is now live! One long (weekend warrior) journey migrating a legacy app to cloud-native AWS complete! 🎉

Conclusion

As you saw, doing a clean install to a new environment got past the failed stack. My celebration felt tainted though, with the dead stack still staring at me in bright red on the AWS Console. While creating a fresh production environment was probably the right thing to do anyway, it is not the solution to every problem. Developing and maintaining a software system over months and years involves updates. Updates that can’t lose users and user data. Being unable to find the root cause of the broken stack makes me very nervous about trusting Amplify with user data. There’s much good and much promise in Amplify — the new Amplify DataStore feature looks a-maz-ing — but my final experience leaves me hesitant to use or recommend this toolset. I waited several weeks to write this final post. Part of that delay was to give the Amplify team time to discover an explanation. Perhaps the bigger reason is that, after seven months on this exploration, I really did not want it to end this way. 😕

In the end, my final conclusion about using AWS Amplify is simple:

There are no shortcuts

Amplify comes across as a magic toolset to let frontend developers create applications without being backend developers or cloud engineers. This simply isn’t the case, and the members of the Amplify team that I have met tell me that this was never the intent. It is a toolset to help with the minutiae of managing and using cloud infrastructure. As my journey and (excruciatingly detailed) blog posts have shown, you still need to understand the AWS services behind everything Amplify is helping you with. A tool is only useful in the hands of someone who knows how to use it, and there is truth in the phrase: “I know enough to be dangerous.” My advice for app developers looking to leverage the cloud: learn the full stack or team up with a solutions architect. Find an AWS Meetup group in your area and do some networking. Find a consultancy that can #thinkcloudnative. AWS is still the most comprehensive platform on which to develop, so go forth and create!

(This story was original posted here in March 2020.)

This is part 7 in a series documenting my journey migrating my progressive web app, called SqAC, to AWS cloud native. If you haven’t been following it before now, here are the previous posts:

• Part 1: Background
• Part 2: Requirements & Architecture
• Part 3: Authentication
• Part 4: Add Cloud Storage
• Part 5: Use Cloud Storage
• Part 6: AppSync API and S3 Trigger

Now in part 7, I’ll enhance user accounts first with Google Sign-in (federation), and then with local (guest) accounts for users to try out the app first.

But first…

Updated Dependencies

This series isn’t about Angular, so I won’t go into details. However I found that to update to the latest Amplify libraries, I had to upgrade Angular from 6.0 to 8.0. At the same time, I updated various other dependencies.

Just about a week before AWS deprecated Node.js v8 in Lambdas, Amplify finally added support for Node.js v10 and so that update has been performed as well.

Pull Requests of code changes:

• Part 7 — Update Dependencies
• Part 7 — Amplify Update for Node.js 10

Add Google Sign-In

The new cloud native version of SqAC gained user authentication via Cognito back in part 3. At the time, I didn’t add any federated sign-ins. The legacy app supported Google and Facebook sign-in, but I found that Facebook sign-in wasn’t popular. With Cognito accounts available in this new app, there’s no true need for any federated accounts, but personally I like using my Google account for authentication because it’s easier than tracking another account. Besides, this is another feature of Amplify to explore!

Amplify’s documentation to setup Google is here. Yes, I actually used a bit of Google Cloud Platform for my AWS native app. Breath. It’s okay.

$ amplify auth update
Scanning for plugins...
Plugin scan successful
Please note that certain attributes may not be overwritten if you choose to use defaults settings.

You have configured resources that might depend on this Cognito resource. Updating this Cognito resource could have unintended side effects.

Using service: Cognito, provided by: awscloudformation
What do you want to do? Update OAuth social providers
Select the identity providers you want to configure for your user pool: Google

You've opted to allow users to authenticate via Google. If you haven't already, you'll need to go to https://developers.google.com/identity and create
an App ID.

Enter your Google Web Client ID for your OAuth flow:
Enter your Google Web Client Secret for your OAuth flow:
Successfully updated resource sqacauth locally

Some next steps:
"amplify push" will build all your local backend resources and provision it in the cloud
"amplify publish" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud

$ amplify push
✔ Successfully pulled backend environment dev from the cloud.

An amplify push later, and Cognito is setup to use Google Sign-In. Once again, Amplify makes authentication easy! 🥳

In my Angular application’s account page, I inject AmplifyService as amplifySvc and added the function:

signInWithGoogle() {  
   this.amplifySvc.auth().federatedSignIn({provider: 'Google'});  
}

A button on the page calls this function. This takes me to the familiar Google account selection page, and then navigates back to my application. There were a few problems though:

1. It doesn’t recognize that I’m signed in until I navigated to the account page. Simply rendering the <amplify-authenticator> component triggered the sign-in to be recognized, but I don't want that on the app landing page.
2. My name is “Unknown”.
3. No photograph of myself was received.

We’ll get to problem #1 in a moment. The second two problems caught me by surprise because the legacy SqAC application also supported Google sign-in, through Passport. In the Cognito AWS Console, I was able to select attribute mapping for name and picture (email and sub were already checked). This resolved receiving my name and photograph. This was easy to do, but I already knew where to look because I have experience with OAuth and Cognito. Someone looking to use Amplify without AWS experience would be lost.

Problem #1 remains: sign-in isn’t recognized right away. Upon sign-in with Google, the app reloads and I see the Cognito user information in browser local storage. Yet, Auth.authStateChange$ reports a state of “signedOut”. Explicitly requesting the current auth user also returns “No current user”. If I reload the page, then the state becomes “signedIn” and all is well. This seems like a bug. 🐞 I added information to the existing Amplify-js Issue #4621. After some messages exchanged and some experimentation, here’s a copy of my comment after I resolved this issue:

Okay, found two fixes based on your comments @Amplifiyer, both involving using the Hub to listen for signIn. I first used this as a direct analogue to monitoring Auth.authStateChange$. The Hub told me when a federated user signed in/out, and authStateChange$ for a user pool user. I still wanted to land on my accounts page though. So I changed it to navigate to the account page on Hub signIn event. Doing that caused Auth.authStateChange$ to trigger when the auth component rendered, so I just rely on that. Here's the code change: kernwig/sqac-amplify@8de758e

Note: The Hub doesn't play nice with Angular, thus the need to use NgZone directly.

Amplify Hub is an in-browser publish-subscribe capability that is used by Amplify and available for us to use too. In Angular, services and RxJS provide for similar behavior and so I hadn’t used Hub before this.

Auth Secrets

Upon going to commit these changes to github, I noticed that my Google Web Client Secret was stored in amplify/team-provider-info.json. Whether or not to commit this file is discussed in this new section of the Amplify documentation. I ended up removing the file from source control.

Secure Hosting & Authentication

To take federated login live, I needed HTTPS hosting. (OAuth doesn’t allow http callback URLs.) Thus I had to update my hosting configuration to include CloudFront, the AWS Content Delivery Network:

$ amplify update hosting
? Specify the section to configure CloudFront
CloudFront is NOT in the current hosting
? Add CloudFront to hosting Yes
? default object to return from origin index.html
? Default TTL for the default cache behavior 86400
? Max TTL for the default cache behavior 31536000
? Min TTL for the default cache behavior 60
? Configure Custom Error Responses No
? Specify the section to configure Publish
You can configure the publish command to ignore certain directories or files.
Use glob patterns as in the .gitignore file.
? Please select the configuration action on the publish ignore. exit
? Specify the section to configure exit
$ amplify publish

This deployed SqAC to https://d3l0j9nusq7n6r.cloudfront.net. I had to add this to the app’s credentials at Google, and then to my app:

$ amplify update auth
Please note that certain attributes may not be overwritten if you choose to use defaults settings.

You have configured resources that might depend on this Cognito resource. Updating this Cognito resource could have unintended side effects.

Using service: Cognito, provided by: awscloudformation
What do you want to do? Add/Edit signin and signout redirect URIs
Which redirect signin URIs do you want to edit? (Press to select, to toggle all, to invert selection)
Do you want to add redirect signin URIs? Yes
Enter your new redirect signin URI: https://d3l0j9nusq7n6r.cloudfront.net/
? Do you want to add another redirect signin URI No
Which redirect signout URIs do you want to edit? (Press to select, to toggle all, to invert selection)
Do you want to add redirect signout URIs? Yes
Enter your new redirect signout URI: https://d3l0j9nusq7n6r.cloudfront.net/
? Do you want to add another redirect signout URI No
Successfully updated resource sqacauth locally

$ amplify push

But wait, this doesn’t work! 😱The Javascript library isn’t smart enough to choose between this CloudFront URL or the localhost one I already had. The aws-exports.js file contains both URLs comma-separated, but at runtime just the first one (localhost, since it was there first) is used. This utterly fails when running the app from CloudFront. See the issue report here. How this gap continues to exist, I do not understand. Surely nearly all front-end developers work first locally (localhost) before deploying to the cloud (CloudFront)? I added this hack to the Angular main.ts to handle the problem:

import awsconfig from './aws-exports';

// Choose an OAuth config based on environment
const redirectSignInOptions = awsconfig.oauth.redirectSignIn.split(',');
const redirect = environment.production
   ? redirectSignInOptions.find(s => s.startsWith('https'))  
   : redirectSignInOptions.find(s => s.includes('localhost'));  
awsconfig.oauth.redirectSignIn = redirect;  
awsconfig.oauth.redirectSignOut = redirect;

Here’s another surprise: by default, amplify publish does not invalidate the CloudFront cache. 🙄 That means after we publish and reload in browser, we will not see our changes! (Unless we wait a day - not even clearing the browser cache will help.) To actually publish and test try out the changes, we must explicitly pass an option: amplify publish --invalidateCloudFront. Yes, this is documented here and of course I won’t remember this next time I publish and so I added a script to my package.json for it.

Pull Request: Part 7 — Add Google Auth Provider

Guest Access

One of my requirements when migrating to AWS was to add guest access. I’ve had some complaints from folks who wanted to try out my app, but were turned off by having to authenticate with Google or Facebook in order to do anything. One way of appeasing this concern was to use a Cognito User Pool, so that users don’t need to link a social account. More impactful though is to allow basic functionality without creating an account at all. In SqAC, I called this a “local account” and provided some warnings about data loss without cloud backup. However, this provides a means for someone to play with the app and decide if it is something they want to go farther with before providing an email address.

Most of my work consisted of changes to my client application itself. I was able to load public content from storage (AWS S3) without difficulty thanks to the work I did in part 4. User data is stored in the browser’s IndexedDB. When there’s a cloud account, this data is copied up to S3 as well; for local accounts this step is skipped. Everything was going great until I hit the last feature to test: search via AppSync. This failed with an exception thrown saying “no current user”. 🤔 This surprised me — I figured not including an @auth on my GraphQL type meant “no auth” and thus no need for a user. 🤷‍♂️ Nope. A search brought me to some background and ultimately I stumbled right to the mass of Amplify documentation that tells us how to do public authentication. It says...

👉Note: Don’t do this! Read on! 👈
First, use API KEY authentication type:

$ amplify update api
? Please select from one of the below mentioned services: GraphQL
? Choose the default authorization type for the API API key
? Enter a description for the API key: Public access
? After how many days from now the API key should expire (1-365): 7
? Do you want to configure advanced settings for the GraphQL API No, I am done.

The following types do not have '@auth' enabled. Consider using @auth with @model

• Collection
  Learn more about @auth here: https://aws-amplify.github.io/docs/cli-toolchain/graphql#auth

GraphQL schema compiled successfully.

Second, use auth mode API KEY and @auth(rules: [{allow: public}]) on the GraphQL type.

I was nervous about the 7 day (default) expiration. Upon an amplify push, the CLI showed me a value for the “GraphQL API KEY” and it is set in my aws-export.js. My app was able to successfully perform the GraphQL query without creating a Cognito user, but it was clear that this would stop working in a week. I could increase this to 365, but that only delays the problem. Amplify CLI issue 1450 has conversation of people struggling with this.

Back in that Amplify documentation, it first says that API KEY must be used for public access, then gives a second example with IAM authentication for public access. Why not try that? 🤷‍♂️

I updated my GraphQL with @auth(rules: [{allow: public, provider: iam}]) and then on the CLI:

$ amplify update api
? Please select from one of the below mentioned services: GraphQL
? Choose the default authorization type for the API IAM
? Do you want to configure advanced settings for the GraphQL API No, I am done.

GraphQL schema compiled successfully.

$ amplify api gql-compile
$ amplify push

A bit of testing with a cloud account and a local (guest) account — and it’s working! 🎉

Call out to the Amplify team: This bit of documentation can be made clearer that there are two approaches, and how to choose between them.

Conclusion

As with everything Amplify (and honestly, doing new things in general) adding Google Sign-In and guest access was harder than it looked. It took some web searching and experimentation, but ultimately was easier than it would have been without Amplify.

Coming next time…

Cut over! The legacy app is still at https://sqac.fanello.net/, and this new one is at https://d3l0j9nusq7n6r.cloudfront.net. One has a slightly friendly name than the other. 😉

I just paid for another month of DigitalOcean hosting for the legacy app, and hope for it to be the last.

Until next time. 😎

(This story was originally published here in January 2020.)

Every year, re:Invent is a Las Vegas all-you-can-eat buffet of new AWS capabilities being announced. Earlier Onica blog posts have covered the 2019 re:Invent keynote announcements from Andy Jassy and Dr. Werner Vogels in near real time. There is so much going on though, that many other announcements fly under the radar. This humble serverless and web application developer attended his first re:Invent this year and noticed a few new capabilities that were not mentioned in the keynotes, but caught my attention. One may be the answer to a popular question of our times: what comes after serverless?

Read the full post at https://onica.com/blog/aws-announcements/key-serverless-announcements-at-reinvent-2019/

• Part 1: Background
• Part 2: Requirements & Architecture
• Part 3: Authentication
• Part 4: Add Cloud Storage
• Part 5: Use Cloud Storage

Review Our Goal

User settings and collections can be saved to and retrieved from the cloud. Now I need to finish off publishing “public” collections and add in the ability for other users to find these. Let’s refer back to the architecture diagram:

In this series installment, I will implement the row down the middle: AppSync, DynamoDB, and the Validate & Index lambda.

The last time I showed this diagram, the two lambdas were outside of the “Managed by AWS Amplify” box. I have since found how to include those within Amplify…

Amplify API is AppSync

Amplify has two options for implementing APIs. One is a traditional API Gateway to Lambda approach, that Amplify refers to as REST. (Nothing about it actually enforces RESTfulness, it simply can be RESTful.) Beyond the Amplify documentation, I have seen no chatter about this option. When someone talks about Amplify API, they’re talking GraphQL. On the AWS cloud end, this is done using the AppSync service.
I’ll add an API to SqAC via the amplify add api command:

$ amplify add api
? Please select from one of the below mentioned services GraphQL
? Provide API name: sqacamplify
? Choose an authorization type for the API Amazon Cognito User Pool
Use a Cognito user pool configured as a part of this project
? Do you have an annotated GraphQL schema? No
? Do you want a guided schema creation? Yes
? What best describes your project: Single object with fields (e.g., “Todo” with ID, name, description)
? Do you want to edit the schema now? Yes
Please edit the file in your editor: /Users/adamfanello/dev/sqac/sqac-amplify/amplify/backend/api/sqacamplify/schema.graphql
? Press enter to continue

GraphQL schema compiled successfully.

For authorization I chose Amazon Cognito User Pool, which is what you want for user-facing clients. I’m actually building a public read-only API and thus won’t enforce any authentication, but there’s no “none” option.

My GraphQL API schema is extremely simple. Really, I could easily do this by other means with a single DynamoDB table and the REST API option — or even with the client accessing DynamoDB directly! The point of this journey is to try new things though, thus GraphQL and AppSync with Amplify. Here’s the GraphQL schema:

type Collection  
 # Store in DynamoDB, disable mutations and subscriptions  
 @model( mutations: null, subscriptions: null)  
{  
 id: ID!  
 created: String!  
 modified: String!  
 revision: Int!

 name: String!  
 author: String!  
 authorUserId: String!  
 description: String!  
 difficulty: Int!  
 level: String!  
 formations: Int  
 families: Int  
 calls: Int  
 modules: Int!  
 license: String!  
}

This is a public read-only API, thus in the model I’m disabling all mutations and there’s no @auth directive. This simply lets users of the app search for collections that the individual authors have released to the public. The @model directive tells Amplify to expand this type with a GetCollection query and a ListCollections query, and all the GraphQL input and output types needed to drive those. (It would have done more had I not disabled mutations and subscriptions.) Best of all, since Amplify knows I’m using the Angular framework, it also generated Typescript types and an Angular service for me to use. The result: I don’t actually have to touch GraphQL in my client; it’s simply a function call to do what I need. 😎

The DynamoDB table created is by default PAY_PER_REQUEST, which makes it truly serverless. However, this on-demand option does not fall under the AWS free tier. So I switched DynamoDBBillingMode to PROVISIONED with the default of 5 RCU and 5 WCU - which fits within free tier and is plenty for my app’s current needs.

S3 Trigger Lambda

Since the API is read-only, you may wonder where the data comes from. That’s the job of the Validate & Index lambda — an S3 trigger. Whenever a collection is uploaded, it’ll trigger this Lambda that will manage public collections, including updating the DynamoDB table that this API is reading from. Adding the trigger:

$ amplify update storage
? Please select from one of the below mentioned services Content (Images, audio, video, etc.)
? Who should have access: Auth and guest users
? What kind of access do you want for Authenticated users? (Press to select, to toggle all, to invert selection)create/update, read, dele
te
? What kind of access do you want for Guest users? (Press to select, to toggle all, to invert selection)read
? Do you want to add a Lambda Trigger for your S3 Bucket? Yes
? Select from the following options Create a new function
Successfully added resource S3Trigger08755fbf locally
? Do you want to edit the local S3Trigger08755fbf lambda function now? Yes
Please edit the file in your editor: /Users/adamfanello/dev/sqac/sqac-amplify/amplify/backend/function/S3Trigger08755fbf/src/index.js
? Press enter to continue
Successfully updated resource

This wiped out all my customization in storage/parameters.json as described in part 5. 😑 Fortunately, I have the good sense to review all the changes from a CLI tool before committing them, and so was able to revert the damage leaving only the “triggerFunction” parameter changed.

I also edited the CloudFormation template for this function to use Node.js 10 and limit to a single concurrent execution. (Amplify is still setting Lambdas to the Node.js 8 runtime, even though its deprecation is only five weeks from when I write this.)

Now I need this trigger to be able to write to the Collection table in DynamoDB. For this, two things are needed:

1. The lambda needs the name of the table.
2. The lambda needs permission to write to the table.

Finding the table

This required some experimentation and mild hackery. Amplify doesn’t provide any information other than the environment name (“dev”) to the trigger Lambda. Looking in AWS Console, I see that the table name is a concatenation of my type name “Collection”, the generated AppSync API identifier, and the environment name. “Collection” is a static name and so safe to hard code. The environment is provided to the Lambda via environment variable (overloading the word “environment” in different contexts), and the AppSync ID I found in the amplify-meta.json file. 🤔 I tried doing a Javascript require(‘../../../amplify-meta’) to fetch this content, but the file wasn’t delivered as part of the Lambda source and thus failed at runtime. (I expected as much, but it was worth a try.) With some searching, I found that Amplify has a pre-deployment hook capability, as described in the documentation. 🎉 I used this package.json script to copy amplify-meta.json to the function src directory, and added Typescript compilation while at it!

In the Lambda’s package.json at amplify/backend/function/S3Trigger08755fbf/src/

"scripts": {  
  "build": "cp ../../../amplify-meta.json . && ../../../../../node\_modules/.bin/tsc"  
}

In the Lambda src directory, I also ran tsc --init to generate a tsconfig.json and thus ready for Typescript.

Then in my top-level package.json:

"scripts": {  
  "amplify:S3Trigger08755fbf": "cd amplify/backend/function/S3Trigger08755fbf/src/ && npm run build"  
}

It could have all been put in the top-level, but this way I can do trial builds from the lambda src directory. Now whenever doing an amplify push, the latest meta data will be copied and Typescript compiled. 🎉
(Of course I added the copied amplify-meta.json and compiled .js output to my .gitignore file.)

Put all together, my Lambda source code now includes:

const amplifyMeta = require('./amplify-meta');  
const ddbTableName = 'Collection-'
  + amplifyMeta.api.sqacamplify.output.GraphQLAPIIdOutput   
  + '-' + process.env.ENV;

I did attempt another approach as mentioned here, where @mikeparisstuff states that if I just define an input parameter for a CloudFormation stack (say, the S3 trigger lambda), then Amplify will provide the value. I tried this with AppSyncApiId, but it failed as unresolvable. Perhaps this works only within the API category.

Adding permissions

There may be elegant solutions for adding permission for the S3 trigger lambda to access the DynamoDB table, but I couldn’t find any. I simply added the new statement to the policy given to the lambda’s execution role, right in the storage category’s CloudFormation. I even gave it access to all resources, which I feel is fine because I’m following the practice of one environment to one AWS account. As such the wildcard resource only gives access to the one DynamoDB table in the account. Specifically, I modified the S3TriggerBucketPolicy in amplify/backend/storage/storage/s3-cloudformation-template.json to add this to the Statement array:

{  
   "Effect": "Allow",  
   "Action": [  
       "dynamodb:PutItem",  
       "dynamodb:DeleteItem",  
       "dynamodb:GetItem",  
       "dynamodb:Query",  
       "dynamodb:UpdateItem"  
   ],  
   "Resource": "\*"  
}

The S3 Trigger Logic

Amplify created a bare bones example Lambda, and I have now given it permission to access the DynamoDB table created by the Amplify API category. Now I have to write up the exact logic. 🙌 This is the easy part for me. Infrastructure is meh 🤷‍♂. Fighting with tools is argh! 😱 Getting back to just writing some code with familiar tools, language, and services is like comfort food; I can just relax and enjoy. 🤗

After some iterative experimentation and coding, the Lambda was written. You can find it in PR #7. (Notice the actual handler is just eight lines of loop and error handling — everything else is in small helper functions. Access to S3 and DynamoDB are abstracted into the storage.ts and database.ts files.)

The resulting logical flow is described below. Note that there is only one S3 bucket and one Lambda (with multiple executions). They are shown repeatedly here to break it down by steps.

The “Triggered but no action” lambda executions are unfortunate side effects because everything is in one bucket and without pre-execution filtering. (Hand-rolled S3 allows both of these approaches, but not Amplify.) Since the Lambda will always be warm at those two points (I set concurrency to one), it’s just a quick sub-100 millisecond execution to detect state and exit.

Implement Search in Client

To start using the Amplify API category, the first step is to add API to the list of Amplify modules registered in app.module.ts, as described here. The exact code to add for this is not specified; the linked documentation just has examples for three other categories. The key is import API from "@aws-amplify/api"; and then add API to the object passed to AmplifyModules.

My search capability allows the user to enter a bit of text, and then searches for that text in the collection’s name, description, and author’s name. The challenge was figuring out just how to do that with the generated ListComponents function. The input type allows for criteria to be given for each property, but does not specify whether multiple criteria is an and operation or an or operation. Experimentation found that it’s anAND, but what I really wanted was a mix of ORs (text in any of three properties) and ANDs for the other search criteria (difficulty and level properties). I finally noticed that the generated ModelCollectionFilterInput has three extra properties at the end that are not part of my data model: and, or, and not. That could work. By the time I noticed these I had also realized through my experimentation that the text searches are case-sensitive. This is how DynamoDB behaves, and thus so does AppSync talking to DynamoDB. In the end I added a searchText property to the GraphQL schema and added a bit to the Lambda to set this value when writing to DynamoDB. With this new property containing the three text fields concatenated and in all lower case, the input to ListComponents in the client can now simply set the relevant criteria using the lower-cased search text and default and operation behavior.

I explain all that dry detail because you won’t find it explained anywhere else. Even digging into the resolver for ListComponents lead to a dead end - a Velocity Template utility function that translates the input to a DynamoDB filter expression, but without the details of how. (Find toDynamoDBFilterExpression on this page if you want to see it.) Are you tired of me complaining about poor documentation? Me too. Moving on.

Here’s the good news: after this, the Amplify → Angular → GraphQL → AppSync → DynamoDB integration just worked! Once some details are cleared up, it’s really easy. 🎉 I linked to all the pertinent little bits of code in the paragraphs above.

Coming next time…

Upon completing this section, I was excited to realize that this migrated version of SqAC has reached feature parity with the existing production version! 🎉 🥳

There is one more feature on my to do list: guest accounts. Additionally, there are finishing touches such as S3 policy, revision purging by age, CloudWatch monitoring, and email notifications.

Until next time. 🙇‍♂️

(Story originally published here in November 2019.)

In part 4 of this series, I add Amplify Storage and explore its security model and how to customize it…

• Part 1: Background
• Part 2: Requirements & Architecture
• Part 3: Authentication

Amplify has a storage module which may be backed in AWS by either S3 or DynamoDB. Back in part 2 when exploring application requirements, I noted that S3 would be used for storage of user settings and data collections. In short, DynamoDB could not be used because:

1. Records are limited to 400 KB and I don’t want to limit a collection to that size.
2. Amplify’s storage API may use S3 or DynamoDB; we can’t use both for different data. Thus user settings, while small, will also go into S3.

Note: I later discovered that while the CLI provides an option for DynamoDB storage, none of the Amplify libraries, in any language, support it.

We will use DynamoDB for search (see requirements), but that will come later. Right now, we just want to save and read back user settings and collections.

If it hasn’t been clear yet in this series, I’m writing this as I attempt to use Amplify. This is a series about a journey, not a straight-up how-to guide.

Add Storage

Where did we leave off?

$ amplify status

Current Environment: dev

Category	Resource name	Operation	Provider plugin
Hosting	S3AndCloudFront	No Change	awscloudformation
Auth	sqacauth	No Change	awscloudformation

Hosting endpoint: http://sqac-amplify-20190817123020-hostingbucket-dev.s3-website-us-west-2.amazonaws.com
Hosted UI Endpoint: https://sqac-dev.auth.us-west-2.amazoncognito.com/

I have only hosting and auth. So let’s add storage:

$ amplify add storage
? Please select from one of the below mentioned services Content (Images, audio, video, etc.)
? Please provide a friendly name for your resource that will be used to label this category in the project: storage
? Please provide bucket name: sqac-amplify-user-data
? Who should have access: Auth and guest users
? What kind of access do you want for Authenticated users? create/update, read, delete
? What kind of access do you want for Guest users? read
? Do you want to add a Lambda Trigger for your S3 Bucket? No
Successfully added resource storage locally

Some next steps:
"amplify push" builds all of your local backend resources and provisions them in the cloud
"amplify publish" builds all of your local backend and front-end resources (if you added hosting category) and provisions them in the cloud

I choose content as “Images, audio, video, etc” vs the NoSQL option. This is how you get S3 instead of DynamoDB. I then gave things friendly names, rather than the randomly generated defaults.

When exploring the requirements, I stated that I wanted first-time visitors to be able to play with my application without first creating an account, thus I selected to provide guest read-only access. Authenticated users have full read-write.

While we will eventually add a Lambda Trigger to the S3 bucket, this is a big task for later and so I answered No for now. Amplify allows us to use the amplify storage update command to change things later.

Moving on, let’s deploy these changes…

$ amplify push

Current Environment: dev

Category	Resource name	Operation	Provider plugin
Storage	storage	Create	awscloudformation
Hosting	S3AndCloudFront	No Change	awscloudformation
Auth	sqacauth	No Change	awscloudformation

? Are you sure you want to continue? Yes
⠴ Updating resources in the cloud. This may take a few minutes...

… A few dozen lines of CloudFormation output over a few minutes …

✔ All resources are updated in the cloud

Nothing to it. In the AWS Console (web site), I see a new sqac-amplify-user-data-dev empty bucket. (The -dev is my environment name; it gets appended to everything to support multiple environments. 🥳)

So we’re done, right? Well… that depends.

Security Policies and Parameters

Warning: I’m going to dive deep into AWS policies and CloudFormation here. You can follow along with the files in the part 4 pull request, or just let your eyes glaze over. 😳

I poked my nose into the new CloudFormation stack that defines the storage, including policies, and noticed it isn’t exactly what I want. I want users to keep their private data at the private access level, and shared data in the protected access level, and nothing else. The policies though are allowing users write access in the public area too, which is not very useful as anyone could put anything here and any one else can modify or delete it. I really don’t want that. Uh oh? But then I realized I was looking at the stack’s “Parameters” section and there is a parameters.json file in the amplify folder. 💡Here is parameters.json:

{  
   "bucketName": "sqac-amplify-user-data",  
   "authPolicyName": "s3_amplify_7405df3b",  
   "unauthPolicyName": "s3_amplify_7405df3b",  
   "authRoleName": {  
       "Ref": "AuthRoleName"  
   },  
   "unauthRoleName": {  
       "Ref": "UnauthRoleName"  
   },  
   "selectedGuestPermissions": [  
       "s3:GetObject",  
       "s3:ListBucket"  
   ],  
   "selectedAuthenticatedPermissions": [  
       "s3:PutObject",  
       "s3:GetObject",  
       "s3:ListBucket",  
       "s3:DeleteObject"  
   ],  
   "s3PermissionsAuthenticatedPublic": "s3:PutObject,s3:GetObject,s3:DeleteObject",  
   "s3PublicPolicy": "Public_policy_9efc80af",  
   "s3PermissionsAuthenticatedUploads": "s3:PutObject",  
   "s3UploadsPolicy": "Uploads_policy_9efc80af",  
   "s3PermissionsAuthenticatedProtected": "s3:PutObject,s3:GetObject,s3:DeleteObject",  
   "s3ProtectedPolicy": "Protected_policy_7b753c06",  
   "s3PermissionsAuthenticatedPrivate": "s3:PutObject,s3:GetObject,s3:DeleteObject",  
   "s3PrivatePolicy": "Private_policy_7b753c06",  
   "AuthenticatedAllowList": "ALLOW",  
   "s3ReadPolicy": "read_policy_9efc80af",  
   "s3PermissionsGuestPublic": "s3:GetObject",  
   "s3PermissionsGuestUploads": "DISALLOW",  
   "GuestAllowList": "ALLOW",  
   "triggerFunction": "NONE"  
}

Cool. Let’s see what I want here…

"s3PermissionsAuthenticatedPublic": "s3:GetObject",

I think that’ll do it. All I did was remove permission to write to the public area. Anyone, including guests, can still read from it. Only data that I manually put in there can exist, so I can use this for the guest “demo mode” content. (I won’t though, read on.) Nothing in here makes it entirely clear how the protected functionality works, but these are just configuration options not the actual policies. For that, I dig around in the s3-cloudformation-template.json file that Amplify created. The JSON format is a bit verbose though, so I pop open the CloudFormation Designer (GUI) tool in AWS Console and use that to select policies to look at in more concise resulting YAML.

First interesting bit I found:

S3GuestReadPolicy:  
    DependsOn:  
      - S3Bucket  
    Condition: GuestReadAndList  
    Type: 'AWS::IAM::Policy'  
    Properties:  
      PolicyName: !Ref s3ReadPolicy  
      Roles:  
        - !Ref unauthRoleName  
      PolicyDocument:  
        Version: 2012-10-17  
        Statement:  
          - Effect: Allow  
            Action:  
              - 's3:GetObject'  
            Resource:  
              - !Join   
                - ''  
                - - 'arn:aws:s3:::'  
                  - !Ref S3Bucket  
                  - /protected/*  
          - Effect: Allow  
            Action:  
              - 's3:ListBucket'  
            Resource:  
              - !Join   
                - ''  
                - - 'arn:aws:s3:::'  
                  - !Ref S3Bucket  
            Condition:  
              StringLike:  
                's3:prefix':  
                  - public/  
                  - public/*  
                  - protected/  
                  - protected/*

If you haven’t noticed yet, Amplify manages access levels by prefixing the S3 key with public, protected, or private, followed by the authenticated user’s ID. (You can think of these as file system folders, even though technically they are not.) Thus a hypothetical user with Cognito ID 123 will store private data in private/123/ and protected data in protected/123/. Notice here that there is a public/* prefix matching condition. Does that mean that users can only write to their own public area? If so, what is the difference between public and protected? I must dig some more!

This policy does answer a question for me though: a guest user can read protected content, not just public. I may have no need for public at all then.

The S3AuthReadPolicy (for authenticated users, not guests) is similar, but has two more prefixes to allow reads of the user’s private data. Thus authenticated users can read anything except other user’s private sections:

Condition:  
      StringLike:  
        's3:prefix':  
          - public/  
          - public/*  
          - protected/  
          - protected/*  
          - 'private/${cognito-identity.amazonaws.com:sub}/'  
          - 'private/${cognito-identity.amazonaws.com:sub}/*'

There’s an S3GuestUploadPolicy which would allow guests to upload to an uploads/ prefix, but it is nullified by the default parameter of s3PermissionsGuestUploads being set to DISALLOW. 👍This leads to the S3AuthUploadPolicy for authenticated users, which is allowing users to dump files into the uploads/ prefix with no way of reading it back. I figure this must be a feature to allow for uploads that are then processed by a triggered Lambda. I have no use for such uploads, and so in parameters.json I set s3PermissionsAuthenticatedUploads to DISALLOW, just like the parameter for guests.

Moving on, I see an S3AuthPublicPolicy which ties to the s3PermissionsAuthenticatedPublic parameter that I already changed to S3:GetObject; thus authenticated users can only read, not write to the public area. I see that this applies to anywhere in the public prefix, with no restriction around the user’s ID:

Resource:  
      - !Join   
        - ''  
        - - 'arn:aws:s3:::'  
          - !Ref S3Bucket  
          - /public/*

Contrast that to S3AuthProtectedPolicy, which does restrict write activity to the user’s ID:

Resource:  
       - !Join   
          - ''  
          - - 'arn:aws:s3:::'  
            - !Ref S3Bucket  
            - '/protected/${cognito-identity.amazonaws.com:sub}/*'

The S3AuthPrivatePolicy looks nearly the same, protecting the private folder:

Resource:  
       - !Join   
          - ''  
          - - 'arn:aws:s3:::'  
            - !Ref S3Bucket  
            - '/private/${cognito-identity.amazonaws.com:sub}/*'

So what’s the difference? Scroll back up to S3GuestReadPolicy and S3AuthReadPolicy. While S3AuthProtectedPolicy and S3AuthPrivatePolicy cover what a user can write to, the earlier policies allow anybody to read the protected/* prefix.

I have long been puzzled as to why Amplify’s documentation didn’t clearly define the access roles: public vs protected vs private. I couldn’t find any explanation in the past, but see that there are now some details here. However, I still find it a bit ambiguous. Now I see some justification for that: it’s up to you! Editing the parameters.json file lets you dictate the behavior. However, that too is not documented. 😔

Based on this quick study, I’ve put together a summary of what I think are the rules. I may be mistaken on some of it though; no guarantees. Assuming you select guest access and read-write for authenticated users, then this is the default behavior and the parameter to change if you wish:

• upload/Authenticated users may upload to this. (s3PermissionsAuthenticatedUploads)
  Guests may not. (s3PermissionsGuestUploads)
• public/*Authenticated users may read and write. (s3PermissionsAuthenticatedPublic)
  Guests may read. (s3PermissionsGuestPublic)
• protected/Anybody may read (not configurable)
• protected/{user-id}/*Anybody may read (not configurable)
  The matching authenticated user may write (s3PermissionsAuthenticatedProtected)
• protected/No access
• protected/{user-id}/*The matching authenticated user may read and write (s3PermissionsAuthenticatedPrivate)
• Authenticated users may list contents of any prefix they can read (AuthenticatedAllowList)
• Guests may list contents of any prefix they can read (GuestAllowList)
• If in the CLI you selected no guest access, then unauthenticated users have no access.

Possible values of these properties are a comma-separated list of any of s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject or simply DISALLOW.

The parameters file also includes selectedGuestPermissions and selectedAuthenticatedPermissions, yet I don’t see them used anywhere in the CloudFormation template. 🤔🤦‍♂️

For SqAC, I modified the parameters to disable the upload and public features entirely, while keeping the default behavior for protected and private.

CORS?

As part of the Storage / Using Amazon S3 documentation for Amplify, is this bit telling you to manually configure CORS policy on your S3 bucket. 😲 This would break Infrastructure as Code (IaC) and easy use of multiple environments! Fortunately, the documentation is a red herring — Amplify has already set the bucket policy as documented. (I have submitted a bug report to remove this.)

To be continued…

Find all of this in the part 4 pull request.

I had intended to include updating the client app to use storage as part of this post, but the security policy analysis turned this into a big post already, and the next one is shaping up to be a good bit of work as well.

Coming next time… using Amplify Storage!

This is part 5 in a series. If you haven’t been following it before now, here are the previous posts:

• Part 1: Background
• Part 2: Requirements & Architecture
• Part 3: Authentication
• Part 4: Add Cloud Storage

Now that I have added cloud storage via Amplify CLI, and configured it, it’s time to use the new storage in my app.

Preface

Generally this series is written as I use the tool and presented in the order that I created the content. There’s some editing from draft to publishing, but this series is about the journey and so I present it as such.

This time though, I’m coming back and adding this note before you get into the content of the post. This was absolutely the most challenging part of the migration so far, and I suspect it will remain so throughout the series. Roughly half of my time was spent reverse engineering Amplify and the Javascript SDK due to poor documentation and my frustration shows below. I want to state here that I really do like AWS, and I came into this really looking to succeed with Amplify. The engineers at AWS have created amazing tools and services that have transformed what it is to be a software developer over the past decade, making it possible to build products at scales and paces never before possible. So while I am frustrated with imperfection, I also acknowledge that without the amazing work at Amazon and AWS I’d probably still be back in part 1 building an authentication system. 😬

Be sure to read the conclusion too before getting scared off. Here we go…

Add Storage Support to Amplify App

The first step is to add the Amplify storage module to my Angular app as shown here. In anticipation of this need though, I already did that step when adding authentication. Moving on…

Basic Use — Not so Easy

The web app had been using Feathers’s client library to communicate with the Feathers backend. That must now be replaced with Amplify Storage. Fortunately, I designed the application pretty well by isolating all of the actual communication into a PersistenceService class. 🎉

The first thing to do is write something to storage, which leads to Storage.put(key, content, config). What are the config options? No single place tells you all of them. The Typescript declarations give nothing, only this section of the documentation tells you what may or may not be a complete list through a series of examples. There’s an API reference here, obviously generated from some Typescript and simply listing the config options as type any. 😕 The API for the get function is worse in a way, as it indicates that it returns “either a presigned url or the object”. Which one is a mystery perhaps controlled via the equally mysterious config?: any second parameter. Regardless, the Amplify Storage documentation states that get “retrieves a publicly accessible URL” so this would seem the most likely if less useful result. Perhaps so, but my earlier suspicion appeared confirmed when I tracked down the source code and found that the API documentation is not generated from the entirety of the code. The JsDoc proved more useful:

/**  
 * Get a presigned URL of the file or the object data when download:true  
 *  
 * @param {String} key - key of the object  
 * @param {Object} \[config\] - { level : private|protected|public, download: true|false }  
 * @return - A promise resolves to either a presigned url or the object  
 */  
public async get(key: string, config?): Promise<String | Object> {

Thank goodness for open source! The actual logic is in AWSS3Provider.ts the contents of which answered my next question regarding error handling. I see the Promise returns the raw response from the AWS-SDK’s s3.getObject. So tracking that API down reveals only that the reject data type is Error, as generic as it gets. 😕 Also, only here do I learn that upon choosing to download, the resulting Object will have a property named Body of “Typed Array”… whatever that is. Through experimentation (i.e. console.log), I found that the body is Uint8Array, and a toString(‘UTF-8’) on that gives back the string that I stored. (I'm generally under the opinion that any API with anything that amounts to a getter and setter aught to return a value in the same form it was set.)

The situation is even worse again for Storage.list(). The example here simply indicates that it returns a result. That’s it. It returns something. Once again I traced it down to the SDK s3.listObjects documentation, but this time the actual results did not even match what is there and thus showing that AWS’s documentation problem extends beyond Amplify. (FYI: The actual returned result is just the Contents array, and its fields start with lower-case, not the documented upper-case.)

I can probably dig further into the source, but this exercise has reached a level of absurdity. It is a failure of an API, and a sign of unmaintainable code, when anything past the documentation and function signature needs to be read by someone trying to use it. The point of Amplify is to make using AWS easy. Its storage module turned out to be just another layer of mystery to sleuth my way through. It’s particularly a miss here given that the Amplify source itself has a bit more documentation sitting there ready to provide help, but it is hidden away.

Here are highlight snippets of what actually worked with the Amplify Storage API. The full code can be found in PR #4.

import {AmplifyService} from 'aws-amplify-angular';  
import {StorageClass} from 'aws-amplify';

@Injectable()  
export class PersistenceService {  
  /** API to the cloud storage */
  private readonly cloud: StorageClass;

 constructor(private readonly amplifySvc: AmplifyService) {  
    this.cloud = this.amplifySvc.storage();  
  }

  async loadUser(): Promise<UserSettings> {  
    const downloadedObj = await this.cloud.get(  
      settingsKey,   
      {level: 'private', download: true}  
    );  
    const downloadedStr = (downloadedObj as any).Body.toString('utf-8');
    const downloadedJson = JSON.parse(downloadedStr) as UserSettingsJSON;  
    // etc  
  }

  private async saveModelToCloud<T extends AbstractStorableModel>(model: T, id: string, level: 'private'|'protected'): Promise<T> {  
    let json = model.toJSON() as AbstractStorableModelJSON;  
   await this.cloud.put(  
      id, JSON.stringify(json),  
      {level, contentType: "application/json"}  
    );
  }  
}

The good news is, I now have a file in my storage S3 bucket! 🎉

When sub is not sub

In part 4 of this series I explored the storage policies and found the policy secure the S3 storage path "private/${cognito-identity.amazonaws.com:sub}/" in the CloudFormation.

Once I stored a file though, I saw it actually create the S3 prefix “private/us-west-2:ecee2298-97c5-4331-867c-908eef1660c8/”, while by CognitoUser has sub "508903f1-9203-4cf6-b0d8-353fc54c2916". 🤔 Why aren’t these matching?

After a good bit of digging, I finally noticed that the first part of the CloudFormation says “cognito-identity”. This is the Cognito Identity Pool. Meanwhile, the CognitoUser sub is from the Cognito User Pool. Two different things. Whoops. This is a gotcha and it is talked about at length in Amplify CLI issue #1847 and Amplify JS issue #54, where some commenters have gone a bit bizerk with complicated workarounds. It’s a surprise, but a solvable one with just a couple lines of code. Simply, I call AuthService#currentUserInfo() and use the ID from this (Identity Pool ID) instead of the value from AuthService#authStateChange$. The code changes can be seen in this commit.

Storage NoSQL Option?

In digging through Amplify to discover how to use the Storage module, I learned that it is a thin wrapper over the AWS SDK and its behavior is tightly coupled to S3. What would happen if I had chosen the NoSQL (DynamoDB) option when doing the amplify add storage command? Would the same APIs provide entirely different results? I looked back in the amplify-js storage source code, and found only the S3 provider. 😲

Then I took a peek at the iOS and Android documentation. Like the Javascript documentation, they say to choose the “Content” option. (I missed that remark in my earlier readings.) I suppose the team writing the Angular CLI is simply ahead of the client library teams. I’m glad I choose the S3 option when contemplating this during the architecting phase of the project!

Conclusion

While the Amplify CLI makes setting up AWS infrastructure easier, and the Amplify Authentication module makes managing users really easy, the Storage module has been a bust. Given that I had to dig down to the underlying SDK to figure out how to use any of it, it would have been easier to just use that underlying SDK. It is far from a lost cause though. The capabilities are there; redemption only requires proper API documentation! Proper use of Typescript would also go a long way. (Avoid type any.)

This is unfortunately a too common problem with developer-driven products. We developers like to make features, not documentation. We make the feature to the “works for me” point, call it complete, and move on to the next shiny thing. Few of us like writing for humans. Even when there is documentation (and Amplify does have a lot of documentation written) it turns out to be more fluff. “Look at this shiny new feature! Use it to do great things!” When you actually try to use it though, we get things like this:

The list function takes two parameters of something, and returns a something. 🤷‍♂️ This is not an AWS problem, it’s an industry problem that I’ve seen throughout my career. Perhaps part of the reason open source has become so dominant isn’t economic, but rather because given a choice between an undocumented proprietary library and an undocumented open source library, we use the only one that can be used.

Coming next time…

I need to spend a bit more time testing my app and making sure this storage is working, so there may be a bit of delay before I can move on. The next part shall involve using Amplify CLI to setup the very small GraphQL API so that it will generate the DynamoDB table. To get data into that table though, I’ll be monitoring the S3 data bucket for changes and storing the meta data into DynamoDB. As seen in my architectural diagram (end of part 2), I may need to step outside of Amplify for this. More to come!

(Story originally published here in September 2019.)