Phil Writes Code

Friday, June 19, 2015

CSS: In the [head]? Or in the [body]? Also, Automagic Page Decoration!

Ran into an interesting problem recently.

My team created an HTML-based book preview application, called Slideshow, about two years ago. In one of its incantations, our app is embedded in a div on another page on our site. It looks like this:

This all worked well and good for many many months. About two weeks ago, however, a bug is assigned to us that says the navigation buttons are displaying as blank squares. Odd -- I jump onto our site to see if I can reproduce, and, sure enough, I see this:

The navigation buttons still behave properly, they just don't render on the page. I think maybe the assets are failing to load, but it appears that we're fetching, and successfully receiving, the proper graphic assets. So I inspect the elements, and there we find our problem:

The CSS for Slideshow's navigation buttons are being overridden by the host page's CSS, which has a global style giving all <a> tags white backgrounds and grey borders.

I check the revision history for the host page's CSS, thinking someone must've recently added this global anchor-tag style, but the css file hasn't been changed in months. What about Slideshow's CSS? Same story: it hasn't been changed in months. If none of the relevant CSS has changed, how could this have broken?

I search through some code, and I see that Slideshow's CSS is programmatically added to the end of the <head> element via JavaScript. Viewing the source of the host page, I see the offending CSS file is included within the <body> of the page. That sounds like it might be our problem: I'm betting that, even though Slideshow's CSS is added to the page last via a function call, it's being stomped on by the host page's CSS because it resides in the <body> of the page.

I investigate the host JSP: It includes a file called commonHeader.jspf, which basically looks like this:

<head>
<content tag="css">
<link rel="stylesheet" type="text/css" href="{our offending css}" >
<link rel="stylesheet" type="text/css" href="{some other css}" >
<link rel="stylesheet" type="text/css" href="{some other css}" >
</content>

<content tag="text/javascript">
{some javascript imports}
</content>
</head>

Strange -- it looks like the offending CSS *is* within the <head> element as defined by the JSP, but, when it is compiled into the served page, it ends up in the <body>. How does that happen? Well, it turns out we use sitemesh to decorate our page, and there is a totally separate header file that is automagically included in all of our site's rendered pages. That header file invokes a special tag to load content into itself (note the <content> tags in the above .jspf code), which means that the <head> and </head> tags in commonHeader.jspf are completely ignored by our site. Fun!

I investigate this newly found header file, and I see that someone made a change about a month ago that moved the tag that includes the "css" content from the <head> element to the <body> element. I swapped it back, and our Slideshow looked beautiful again. Hooray, our assumptions were correct! Loading the host page's CSS from within the <body> was causing it to overwrite Slideshow's CSS, even though Slideshow's CSS was loaded later.

So what did we learn from this adventure? First, we learned that our JSP rendering frameworks are not easy to trace, as it took an entire day's worth of searching -- and a little bit of luck -- to find the offending code. But, more importantly, we learned that it's a good idea to stick to the accepted standard of loading all external CSS files within the <head> of your page, even though most browsers, if not all browsers, don't strictly enforce it.

I did some Googling to see what other developers though about loading CSS in the <head> vs the <body> of a document, and most of the results spoke about best practices and page load-time (it turns out the browser is forced to re-render a page when it finds CSS in the middle of the document), which are great reasons in themselves. However, the issue we ran into shows that there can be actual render problems when CSS is included in the <body>, especially if other code is depending on that CSS to be located in the page's <head>, as Slideshow did when it added its own CSS to the dom.

What are your thoughts? Have you run into similar issues? Or, from the other side, have you ever found cause to include CSS, or JavaScript, within the <body> of your page? Tell me about it in a comment below!

Monday, June 15, 2015

Load and Performance Tests Should Emulate Real-World Situations! Also, Don't Be Dumb.

We've been working on a new application to allow customers to order prints of photos, and recently released it to production. After a week or so, we began getting notifications from Customer Service saying that customers were having trouble ordering our product. They were able to create the product in our application and add the product to the cart, but were seeing errors when they tried to check out.

This was surprising to us, because we had certainly tested the ability to order our product, and customers were able to order other products without fail. It was also frustrating, because it was happening sporadically, and we weren't able to reproduce the failures on our own machines.

We found a stack trace in the logs:

Unexpected error: java.lang.NumberFormatException: For input string: "undefined"
at java.lang.Integer.parseInt(Integer.java:492)
at [redacted].getRotation

Weird. Somehow the rotation of certain images is being set as "undefined". It's easy enough to solve this error -- just throw a try/catch around our Integer.parseInt(...) call, and set default values if an invalid number is passed. But why is this error happening in the first place?

We trace some code, and see that the value for rotation is set when images are first imported into our application. The initial import doesn't have all of the info we need (including rotation), so we make a service call to fetch rotation, along with other important image information. Is that service call failing somehow? We check the code -- everything looks good: no missing null checks, all of the lookups appear to be properly formatted, no apparent race conditions, and we're using methods that are used all over our website without fail.

Ok, so the code looks good. Could there be a server issue? Let's look at some error logs. Sure enough, our top error (by far) is a timeout issue on our fetchImageInfo call. In fact, 435 of the 652 errors we have seen over the past 24 hours -- that's 66 percent -- are related to that service call.

But how could our service be failing? We have passing Load and Performance tests that verify that the app can handle our expected load, and we're not exceeding our expectations (we're not even meeting them, as the app is in A/B testing and only being presented to 10% of our users), yet the errors we're seeing indicate that we're putting too much pressure on our service.

After digging into the Load and Performance tests, we found our problem. As you might have guessed from the title of this post, the problem ended up being that our L&P tests were not actually emulating our real world usage of the service.

Our service was designed to take an array of ImageIds in JSON format. It then fetched the appropriate image information, and returned an array of objects containing that image info. Our L&P tests grabbed 100 random test users, selected a random image album from each, and constructed an array of those images' IDs to be sent to the service, and then verified that proper data came back from the service. These test calls were run simultaneously, and the tests had thresholds for response time, throughput, and failure rate (<1).

The problem was that, instead of constructing an array of ImageIds, our application was sending each imageId to the service one at a time, as each image was imported. This meant that, instead of making one call to the service, the app was making n nearly simultaneous calls, which was quickly overloading the system if the user was importing a large collection of images. So a service that was expected to support <20 requests per second was sometimes being forced to handle hundreds of requests per second.

The solution, again, is somewhat easy: we construct a bundle of ImageIds as images are imported, and send that bundle to the service in a single call, then we inject the response data into the imported images.

The lessons learned are twofold: L&P tests are only useful if they emulate real-world situations, and developers need to ensure that they're optimizing their network traffic. In our case, we wrote our L&P tests to test the service as it was intended: processing bundles of imageIds. However, when the service was integrated into our application, laziness and our desire for simplicity caused us to request info for each image, one at a time, as it was loaded. After all, making individual calls meant we didn't need to uniquely track each image, as each response could be easily mapped to the requesting image. The proper solution meant we'd need to spend time looking up each item in the response, which meant we'd need some sort of map of IDs to images. The result of this misstep was that we had a service that, based on our tests, we thought could handle our expected load, but that buckled under the pressure of actual users.

Monday, July 1, 2013

AngularJS: Filtering Data

I'm currently learning how to use ng-repeat to display lists of items, and how to use filters to limit the data displayed in those lists.

Based on the AngularJS tutorials (docs.angularjs.org/tutorial/), I have a collection of phone JSON objects in the following format:

{
"age": 0,
"id": "motorola-xoom-with-wi-fi",
"imageUrl": "img/phones/motorola-xoom-with-wi-fi.0.jpg",
"name": "Motorola XOOM\u2122 with Wi-Fi",
"snippet": "The Next, Next Generation\r\n\r\nEx[eroemce the future with MOTOROLA XOOM, the world's first tablet powered by Android 3.0 (Honeycomb)."
}

There are a bunch of those; some are tablets, others are phones.

We then have some HTML with Angular directives to display a list of phones (showing only the pertinent parts -- a controller is referenced elsewhere in this HTML, and that controller knows where to find the 'phones' data):

Search: <input ng-model="query" />

<ul>
<li ng-repeat="phone in phones | filter: query">
{{phone.name}}
<p>{{phone.snippet}}</p>
</li>
</ul>

This is all grand. When I type search text in the "query" input, the data is restricted to only that data that contains that text. However, what happens when I want to limit my data to only phones, and not tablets? When I type "phone" into the search bar, it still shows me everything, because all of my images are under a directory called "phones". My options are to either a) rename my directory structure, or b) eliminate the imageUrl field from my filter. While my directory structure probably shouldn't reference "phones" if it contains more than just phones, renaming a structure can often affect your entire project, so let's modify the filter instead.

One option is to specify what portion of data you'd like to filter on.

I've learned two ways of doing this. The first is to specify the intended data on the input itself:

Search: <input ng-model="query.snippet" />
...
<li ng-repeat="phone in phones | filter: query">
...

This modifies the filter so that you'll only be searching in the 'snippet' field in the data. Now this technique is effective, but seems non-intuitive to me. Why should you be tweaking the input instead of tweaking the filter itself? To me, it looks like we've just changed the name of the input's model reference to "query.snippet". The fact that we're only querying the "snippet" field in the data is not obvious to me. I'm no expert, however, so if you like this method, then don't let me stop you -- in fact, I encourage you to add a comment to this post explaining why you like this method.

The second method I've learned, and the one I like more, allows you to tweak the actual filter:

Search: <input ng-model="query" />
...
<li ng-repeat="phone in phones | filter: {snippet:query}">
...

This code behaves in the exact same way as the code above, but, to me, is much more readable. It says "show me a list of all objects in 'phones', but filter the data so that the 'snippet' field contains whatever is in the 'query' model reference."

Note: If you run into an issue where the solution above results in ALL data being filtered out on app startup (and then working properly when you edit the input query), consider upgrading to a newer version of Angular. This was a problem for me with version 1.0.6, but is no longer a problem in version 1.1.5

But what if we want to filter on multiple fields? What if we want to filter on both 'snippet' and 'id'?

For that, we'd write a custom filter:

In the controller:

$scope.queryFilter = function(movie) {
if($scope.query != null) {
return movie.snippet.toLowerCase().indexOf($scope.query.toLowerCase()) != -1
|| movie.id.toLowerCase().indexOf($scope.query.toLowerCase()) != -1;
}
return true;
};

In the HTML:

<li ng-repeat="phone in phones | filter: queryFilter">

That will filter only on the fields we specified, namely 'id' and 'snippet'

Ideally, there'd be a way to filter against all fields *except* a particular field. For example, we'd like to filter against all fields except for 'imageUrl', without needing to specify 'snippet', 'id', 'name', etc. I imagine we could write a custom filter to iterate all fields of a particular object, ignoring 'imageUrl', but I'm not creative enough with JavaScript to do that. If you'd like to give it a shot, please add a comment.

I hope this helped someone. Please leave questions and comments below.

Monday, June 17, 2013

Problem 1: Can't install Git, try running Update; Can't run Update, please install Git

My CoffeeScript PeepCode screencast started off by telling me to install HomeBrew.

The HomeBrew installation is easy enough on a Mac, I literally just ran:

ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"

and HomeBrew was installed properly on my machine.

However, the instructions then said that I should run "brew doctor" before doing anything else, as a way of ensuring that everything was up and running properly. So, I run that command:

brew doctor

And it says:

You must: brew install git

So I do that:

brew install git

And it replies:

Failed to execute: make
Error: Homebrew doesn't know what compiler versions ship with your version
of Xcode (dunno). Please `brew update` and if that doesn't help, file
an issue with the output of `brew --config`:
https://github.com/mxcl/homebrew/issues

Ok, so I run "brew update," and I get:

You must: brew install git

Error: Failure while executing: git init

Hopefully, by now, you're picking up on my frustration. I can't install Git until I run "brew update", and I can't run Update until I install Git.

So I go to the Git HomePage and download the GUI installer. I run it, it says "Installation Successful!", but, when I try to run 'brew update' again, it still complains that I need to install Git. I try running "which git" to see where Git was installed, but my Mac can't seem to find it.

Asking around the office led me to the solution: the Git GUI installer installs Git to /usr/local/git/bin, instead of /usr/local/bin. So, I added /usr/local/git/bin to my PATH, and everything worked wonderfully.

-----------------------

Update

Turns out you can bypass all of this mess by simply installing Xcode, which is free from the Apple AppStore.

When my HomeBrew told me:

Error: Homebrew doesn't know what compiler versions ship with your version
of Xcode (dunno). Please `brew update` and if that doesn't help, file

I assumed it meant "you have Xcode installed, but there's something wonky with the compiler version." Turns out it could also mean "you don't have Xcode installed."

I installed XCode onto my machine, after which "brew install git" worked just fine.

I'll probably delete the "regular" version of Git so that it doesn't conflict with the Homebrew version.

The King is Dead; Long Live the King

So my foray into Android was pretty short-lived. Turns out that it's really difficult to find time to learn a new language when your place of employment has no use for that language. Sure, I could've spent more time learning at home, but, as I don't *need* to learn Android, it was never high enough on my priority list to actually sit down and do it.

So, it's on to a "new" language: JavaScript.

Now, I put "new" in quotes because I already "know" JavaScript -- at least I thought I did. After all, JavaScript was a huge part of my job description at Shutterfly when I was hired here 5+ years ago. However, there were so many advances in JavaScript and its use during the past 3 years that I've spent writing Flex code, that I feel like I don't know JavaScript anymore. To put it into perspective, I've never used JQuery, and that puts me into the "elementary" level of JavaScript coding by today's standards.

When I told some colleagues that I want to learn JavaScript, it was recommended that I learn CoffeeScript instead. While I initially rebuked, because why learn B when you're trying to learn A, some good points were made: most significantly the point that CoffeeScript translates *directly* to JavaScript, and can help you learn the "correct" ways to write JavaScript, without having to suffer through all of the "wrong" ways.

So I'm off to learn CoffeeScript using a screencast from PeepCode at https://peepcode.com/products/coffeescript/. PeepCode, unfortunately, is not free.

I'll also be referencing the O'Reilly book, JavaScript: The Good Parts, which can be downloaded for free at http://it-ebooks.info/book/274/.

Saturday, July 28, 2012

First Problem! AVD Creation with Android 4.0.3

Ran into my first error today, and it was before I wrote a single line of code.

When creating your Android Virtual Device (AVD), Professional Android 4 Application Development recommends using the Galaxy Nexus skin (WXGA720) to skin your emulator, like so:

So I went ahead and did that, and then tried to run my AVD...

The error reads:

Failed to allocate memory: 8

This application has requested the Runtime to terminate it in an unusual way.

Please contact the application's support team for more information.

Well that's not very helpful. Failed to allocate memory for what? And what does the 8 mean?

I did some Googling, and found my issue. It turns out the default resolution for the Galaxy Nexus skin is 720x1280, while my PC's resolution is a measly 1920x1080. When trying to launch the emulator, it seems Eclipse tries to fit the AVD's display inside your screen, and, if it doesn't fit, crashes with a not-so-helpful error message.

The Solution was simple enough: I switched my skin to WVGA854, with a native resolution of 480x854 -- small enough to fit on my screen -- and boom, Hello World!

"But you said you hadn't written a single line of code yet," you exclaim with bewilderment. "Hello World" is the default activity created by Eclipse when you start a fresh Android Application Project. Technically, I deleted a couple of lines that centered the text, but deleting is not writing...

Note: I also ran into the following error starting my AVD a few times:

ERROR: Unable to load VM from snapshot. The snapshot has been saved for a different hardware configuration.

This just means you've been messing with your AVD hardware configuration too often. Launch your AVD from the AVD manager with "Launch from snapshot" unchecked and "Save to snapshot" checked, and it should do the trick.

Thursday, July 26, 2012

The Android Adventure Begins

So I've decided to get serious about Android development. I still see it as a new technology, but it's really starting to spread its wings, and I'd love to get in on the ground floor, or, I suppose, as close to the ground floor as I can get with a technology that's been around for 5 years.

I've tinkered with some Android development over the past few months -- messing around with a prototype app at work -- but, up until now, my knowledge has come from cutting/pasting/tweaking existing code, and Googling "how do I [something] with Android?" I've realized that this is a frustrating approach to learning any language, as you end up reading questions posed by people at the same knowledge level as yourself (close to none), but answered by people who seem to have vast expert-level knowledge, whose tips and advice fly right over your head. I ended up with rough ideas of how to perform the specific tasks I was looking for, but I wasn't getting any closer to having a solid understanding of the Android language as a whole.

So today I went out in search of a book to help guide my way, and by "went out," I mean I went to Amazon.com (c'mon, who goes to book stores anymore?). After scanning reviews for a while, I settled on "Professional Android 4 Application Development" by Wrox (as of this writing, the Kindle edition is $24.74).

I'm just getting started with this book, but I'm already pretty fired up about tackling an interesting new (ish) language. Not only do I hope to enhance my marketability, but I also hope to be able to create some cool editions to my own phone. It should be fun.

Addendum:

Since this is my first blog post, I guess I should lay out a mission statement of sorts...

This blog is intended to be an amalgam of my experiences with software application development: musings on particular coding issues, code samples, and anything else related to development that might pop into my head. I hope that it not only helps me to keep track of my growth as a developer, but also acts as a helpful resource to other developers who might be tinkering in similar areas. For the record, I don't expect this first post to help anyone, except for maybe Amazon if anyone reads this and decides to buy the same book I bought. So, you're welcome, Amazon.