HTTP 404 for Missing API Resources

Should an API return HTTP 404 status when the specified resource cannot be found? Of course, that’s exactly what (Not Found) means. As RFC2616 states pretty clearly, 404 Not Found means:

The server has not found anything matching the Request-URI.

However, if you think APIs are like web pages, you might be perplexed by such an interpretation. Maybe 404 feels like an error because it connotes that something went wrong with the web app. Oh no, the page wasn’t found! The world is coming to an end! Run for the hills! Save the women and children!

But APIs aren’t web apps, are they? They aren’t pages at all. APIs are cacheable, uniformly-addressable, resource-oriented interfaces. Given the address for a resource, if it’s not available, the API should return the HTTP 404 status in most cases.

Unfortunately, there’s so much momentum around the idea of 404 being an error condition that it’s really hard to get developers to think differently about how the web is supposed to work. Monitoring tools, log indexers and server health checks don’t help much for changing minds. So many of these tools dutifully treat 404 as an error condition that well-meaning API developers can’t see outside of the little boxes into which they’ve been coaxed.

One of the most troubling manifestations of this page-oriented bias is Microsoft Azure’s App Service server health check. Deploying a pure Web API to an Azure App Service, you’ll find that you’re subject to all sorts of page-specific constructs related to health. Most notably, if you return HTTP 404 from a server too often, the health checks will mark the instance as failing. Obviously, if pages are missing, something’s clearly wrong with that one naughty instance out of ten. The sheer narrow-mindedness of that idea just boggles my mind given that all the servers run the same code and connect to the same resources by definition.

If this incorrect behavior meant that Azure simply recycled instances more often than it needed to, that wouldn’t be too high a price to pay for their pedantic interpretation of the HTTP specification. Unfortunately, we don’t stop paying the price there. The standard load balancer for Azure App Service seems to be completely perplexed when specific server instances are marked as unhealthy. Performance degrades quickly as more and more instances are quarantined and taken out of rotation while new ones have to be created and spun up. What a mess!

If you’re using a managed API gateway like Azure API Management (APIM), there’s a fairly elegant way to deal with this particular problem. Have your API return 200-series statuses for these conditions, keeping the health checks ignorant and happy, then translate the outbound HTTP statuses to the ones you really want to convey at the edge. Here’s some outbound policy you can add to your API to pick up a special response header named RealHttpStatus and return that instead.

The policy begins by fetching the special response header called RealHttpStatus if it exists. The API should inject this header whenever it means to return something that might be misinterpreted or mishandled by the application server’s management tools. Next, the policy removes the special header so clients won’t see how our chicanery was perpetrated. Lastly, if the integer value of the special header is 404, the actual status returned by APIM will be 404, regardless of what the back end actually provided via the actual HTTP status code.

Of course, this policy’s <choose> element can be extended to include as many other interesting HTTP status types as you might require. Your API can go on respecting HTTP for all its beauty and prescience while keeping those fossilized ne’er-do-wells completely in the dark about your villainous plans.

Create SAS Tokens for Azure API Management with an Azure Function

Shared Access Signature (SAS) tokens are required to call Azure API Management’s original REST API. We can generate these manually on the Azure portal for testing. However, in production, if you want to invoke the APIM REST APIs programmatically, you’ll need to generate these tokens with a bit of code. There’s a snippet available in the APIM documentation that shows how to do this but it’s (possibly) got a flaw that I’ll address below. Moreover, with Azure Functions available these days, it makes sense to expose this token generator as a service. Here’s the code for an Azure Function to do just that:

This Azure Function requires two web application settings named APIM_SAS_ID and APIM_SAS_KEY to be used in the hashing process. You can fetch those from the APIM publisher portal (and wherever they may be on the main Azure portal once APIM is fully integrated there). The token that gets generated from this code will be good for ten minutes. You can add more time if you like by modifying the line of code that calls DateTime.AddMinutes(). Currently, APIM SAS tokens can be generated to last up to thirty days although it’s not good practice to make them last that long.

The problem that I found with the snippet of code that was shown in the APIM documentation is that the inclusion of the seconds in the expiration time caused it to fail validation no matter how the middle (EX) portion of the SAS token was formulated. Perhaps I was doing something else wrong but I found that by setting the seconds to zero in the expiration date, I was able to generate SAS tokens that are honored by the APIM REST API. Here’s a GET operation that fetches the value of a property in APIM named SOME_APIM_PROP using the SharedAccessSignature Authorization schema:

With this Azure Function in place (and the credentials to access it), I can generate SAS tokens for APIM any time I like using a simple, clean HTTP interface. Azure Functions are great architectural building blocks for any modern, API-centric design. If you agree or disagree with that assertion, let me know by reaching me on Twitter @KevinHazzard. Enjoy.

Extract JWT Claims in Azure API Management Policy

JSON Web Tokens (JWT) are easy to validate in Azure API Management (APIM) using policy statements. This makes integration with Azure Active Directory and other OpenID providers nearly foolproof. For example, one might add the following directive to the <inbound> policy for an API to ensure that the caller has attached a bearer token with acceptable audience, issuer and application ID values in the signed JWT:

That’s nice. A little bit of markup and all that nasty security plumbing is handled outside the API. But what if we want to pass some individual claims named inside the token on to the API backend? Unfortunately, Azure APIM doesn’t have that built into JWT token validation policy. Ideally, we’d be able to extract claims during validation into variables and pass them in HTTP headers before the request is forwarded to the backing API. Until that feature is added, here’s how you can do that:

In this code, I’ve added some script inside the <set-header> policy statement to fetch the Authorization header from the request, check that it’s a Bearer type token, attempt to parse it (which checks the token’s signature), then finally extracts the value of one specific claim. Most of that work already happens inside <validate-jwt> policy, as you can imagine. Until there’s an easier way to extract JWT claims individually, the solution shown here works nicely. Enjoy.

If you agree with me that this feature should be built right into the <validate-jwt> policy, please upvote the feature request I wrote on the APIM feedback site.

Kevin’s Career Upgrade v6.0 – Amazon AWS

I started my career in software development 34 years ago. I was 16 years old and a relative needed a program that helped her with a file classification project at a local hospital. I wrote a really bad piece of software in BASIC that took hours to solve the problem each time it was needed. I refactored the program over and over again until I got it to run in a few seconds instead. The process of refactoring the software to improve it was exciting to me. It wasn’t a battle with the machine. It was a battle in my mind and I believed, step by step, that I could win through intelligence and sheer will. My relative showed the program to the hospital staff and they bought it from me for $50. I was ecstatic. Moreover, I was hooked on software development for life.

Over the past 3.5 decades, I’ve made four major shifts in my career. When I started out, I wrote code in assembly language and C language. If you don’t know what those are, no worries. Just suffice it to say that they are super low-level abstractions and close to the actual hardware. I was really good at understanding the machine architecture and using it to my advantage. I even claimed that for a few minutes in 1984, I knew everything there was to know about the PC. Since then, I’ve been slipping. That’s true of everyone in this field though. There is no person who can know everything about all the complex systems that make up the modern PC and the Internet. I know a lot more than the average developer given my background that reaches all the way back to those low-level language days. But truth be known, I rarely use that old knowledge about electrical engineering and processor architecture to get work done today.

The first major shift in my career, which I’ll call v2.0, happened in the mid-1980s. The out-of-date mainframe pre-compiler I used during the metamorphosis was called C with Classes but the language had already been renamed C++. The idea of object-orientation was dazzlingly cool to me. Being able to hide data inside of objects that expose safe access methods was liberating and empowering. I became a real expert in object-oriented design and in the use of the C++ Standard Template Library, riding that wave for more than a decade to create some very cool software.

Career v3.0 for me came in the late 1990s when Java appeared on the scene. This highly expressive but simpler derivative of C++ promised to make our code safer and portable from one processor architecture to another. I was working with a group within the Intel Architecture Lab (IAL) that was implementing a high-performance Java Virtual Machine for the Intel processors. My team was creating all sorts of system-level software in Java that was going to change everything about the PC ecosystem, we believed. Then, one day, high-level folks from Microsoft visited our campus in Hillsboro, Oregon and spent all day in conferences with our IAL managers. Within days, all of the Java projects in IAL, including the screamingly fast new Java compiler and virtual machine were shut down. It was a real tragedy that changed the history of the PC forever. I left Intel and spent the next few years doing all sorts of interesting Java work that I enjoyed.

In 2001 while teaching C++ at a local college, a student asked what I thought of C# (pronounced C Sharp for those who don’t know). I had no idea what it was so I called a friend in IAL and he explained that the reason Microsoft had abandoned Java (and ostensibly got its partner Intel to do the same) was to make way for a new language that would compete with Java. I was intrigued so he put me in touch with someone at Microsoft who sent me a gold CD-R with “Cool” scrawled on it in red Sharpie ink. (Cool was the project code name for C#.) I popped the CD in a drive, ran the installer and started playing. In the matter of minutes, I could tell that C# was different from Java in some interesting ways. Over the next hour, I fell in love with it. Thus began v4.0 of my career. I dove head first into C# and into Microsoft’s .NET ecosystem. I was probably one of the first college professors to teach C# in the world, filling my first classroom with .NET fledglings in 2002. By 2008, I had become a Microsoft C# Most Valuable Professional (MVP) and stayed in the award program for the next seven years.

At the height of my experience as an MVP, I started work on a very large scale database project. Every software developer works with data but this project was really huge, at least one hundred times the size of any data project I’d worked on to date. I discovered that my skills as a programmer were massively inadequate to operate in that environment. The big problem was that my programming brain was wired to be iterative and imperative. I had spent 20 years telling computers what to do. The problem with really big data is that it’s often so big and so unwieldy that you just can’t tell it what to do. Instead, you must work with languages that allow you to describe what you want done instead of how it must be done. This is called the declarative programming model which is essentially the opposite of the imperative (command-based) programming I’d been doing for a couple of decades.

For the first time, I really needed to use declarative, set-based languages like Transact Structured Query Language (T-SQL) to get my work done. There was really no other way to pull it off in C# (or any other language I knew at the time). My mind was transformed through the process. I could never see the world in the same ways again. My job, which had always been to virtualize the world through silicon, became a search for patterns instead. There were scores of data patterns to be discovered. Code patterns emerged at every turn when I started looking for them. I became a pattern junkie in v5.0 of my career. I even wrote a book with my friend Jason Bock called Metaprogramming .NET which focuses on techniques for generating code based on patterns to make software fault tolerant and adaptive to change.

Version 6.0 of my career starts tomorrow. As I write this, I am sitting in a hotel room in Las Vegas, Nevada waiting for the start of’s 2015 AWS re:Invent conference. I am convinced that most companies will shift their infrastructures to the cloud in the next few years. And I’m sure that the way Amazon AWS products are structured, they will capture and keep the majority of cloud market for the next decade.

The reasons for my beliefs about the success of AWS are complex but it boils down to this. AWS is all about micro-architecture at any scale. That may sound like gibberish to those who haven’t built an enterprise system but it’s demonstrably true that all successful, complex systems are built from small parts working in concert to create value. AWS has generally designed their products in this way, deliberately or otherwise. Each product is a kind of gear that can connected to the others to create really interesting new things. There are serious integration challenges remaining but the AWS products fit together quite well, in general. Moreover, they scale dynamically so there’s appeal to small companies and large ones, alike.

We’re already seeing the leading edge of the conversion as small- and medium-sized companies are testing the waters of cloud computing in large numbers to gain some operational advantages over their larger competitors. In the next five years, the motivation for moving to the cloud will shift from operational, financial and tactical to strategic. Services will become available in the cloud that we could never even imagine in our racks of single-purpose servers sitting in private data centers. Machine learning will give way to what’s next: predictive analytics that permeates all of our data, all of our code naturally and automatically whether it’s shaped and purposed for that or not. Those companies trying to compete by running single-purpose, monolithic software will simply fall behind.

In reality, I started the shift to v6.0 of my career a while back. I’ve been using Microsoft Azure and Amazon AWS for a couple of years for my clients. But I’ve been using these services in the old-school way. When AWS Lambda was released a few months ago, that’s when the light bulb in my mind really lit up brightly. AWS Lambda is the simplest idea. What if you could write functions that could be instantiated in microseconds to get any sort of generic work done? And what if the system could scale out the number of available machines for running those functions heuristically and automatically?

Lastly, what if that system of scalable functions could be triggered from anywhere? Saving a file might run a function. That code might write to a database which invokes other functions. Et cetera. Suddenly, I saw a way to compose complex, scalable systems with simple, interconnected building blocks. Moreover, this is all done without owning a single server in the classical sense. In 34 years, I’ve come full circle from wanting to master everything about the machines I use to not wanting any machines at all. Version 6.0 of my career will be just as exciting as v1.0 was to me. For that, I’m truly grateful to be in this profession.

Let me finish by saying that although I think Amazon AWS will dominate in this space, Microsoft and Google will also do quite well. I’m not ignoring them. But the elegance, simplicity and highly composable nature of Amazon’s AWS products will make them a great choice for my clients. Viva Las Vegas!