Here’s a talk about federated search and Amalgamatic that I gave at HTML5DevConf in October. Hopefully, the conference will post a better quality video where you can actually see the demos. But for now, this is what I recorded.
Academic libraries offer many resources, but users cannot be expected to search in, say, a half dozen different interfaces to find what they’re looking for. So academic libraries typically offer federated search.
Sometimes, a solution is purchased. Many libraries, for example, use 360 Search.
Here at UCSF, we are among the libraries that have built our own federated search. Twice.
Collecting metrics is important. But we all know that many metrics are chosen for collection because they are inexpensive and obvious, not because they are actually useful.
(Quick pre-emptive strike #1: I’m using metrics very broadly here. Yes, sometimes I really mean measurements, etc. For better or for worse, this is the way metrics is used in the real world. Oh well.)
(Quick pre-emptive strike #2: Sure, if you’re Google or Amazon, you probably collect crazy amounts of data that allow highly informative and statistically valid metrics through sophisticated tools. I’m not talking about you.)
I try to avoid going the route of just supplying whatever numbers I can dig up and hope that it meets the person’s need. Instead, I ask the requester to tell me what it is they are trying to figure out and how they think they will interpret the data they receive. If pageviews have gone up 10% from last year, what does that tell us? How will we act differently if pageviews have only gone up 3%?
This has helped me avoid iterative metric fishing expeditions. People often ask for statistics hoping that, when the data comes back, it will tell an obvious story that they like. Usually it doesn’t tell any obvious story or tells a story they don’t like, so they start fishing. “Now can you also give me the same numbers for our competitors?” “Now can you divide these visitors into various demographics?”
When I first started doing this, I was afraid that people would get frustrated with my push-back on their requests. For the most part, that didn’t happen.
Instead, people started asking better questions as they thought through and explained how the data would be interpreted. And I felt better about spending resources getting people the information they need because I understood its value.
Just like IT leaders need to “consistently articulate the business value of IT”, it is healthy for data requesters to articulate the value of their data requests.
After integrating Lorenzo’s very helpful sample into our code base, we ran into a couple of issues. First, the script was exiting before the specs were finished running. The sample code had a small number of specs to run so it never ran into that problem. Ilios has hundreds of specs and seemed to exit after running around 13 or so.
I patched the code to have it wait until it saw an indication in the DOM that the specs had finished running. Now we ran into the second issue: The return code from the script indicated success even when one of the specs failed. For Travis, it needed to supply a return code indicating failure. That was an easy enough patch, although I received props for cleverness from a teammate.
I sent a pull request to the original project so others could benefit from the changes. Lorenzo not only merged the pull request but put a nice, prominent note on the article letting people know, even linking to my GitHub page (which I then hurriedly updated).
So, if you’re using Jasmine and Travis but don’t have the two yet integrated, check out Lorenzo’s repo on GitHub and stop procrastinating!
We had a day-long session of hacking trying to improve the code quality and test coverage of Ilios.
This post is clearly not a step-by-step instruction manual on transforming an intimidatingly large pile of spaghetti code into a software engineering masterpiece. I hope video is posted soon of Dan Gribbin’s jQuery Conference presentation from last month to see what wisdom and techniques I can steal acquire.
In the meantime, here are a few small things we learned doing the first session.
- Give everyone concrete instructions ahead of time regarding how to get the app set up and how to get all the existing tests running. Have new folks arrive early for a get-setup session. This allows new folks to hit the ground running and let’s experienced folks begin coding or help people with interesting problems, rather than help people just get started.
- Decide on a focus ahead of time. Try to fix one class of bugs, write one type of test, complete coverage for one large component, or whatever. This allows for more collaboration as people are working on similar things.
- Do it often or you lose momentum! I suspect that weekly is too often. We’re trying once every two-to-three weeks.
P. S. If you recognized the reference in this post’s title, then clearly you have all the skills required to work here. We’re hiring a Front-End Engineer and the job is so good that HR pasted the description twice into the ad. Submit your résumé and knock our socks off.
We use Behat for testing the Ilios code base. (We also use PHPUnit and Jasmine.) We started out using Cucumber but switched to Behat on the grounds that we should use PHP tools with our PHP project. Our thinking was that someone needing to dig in deep to write step code shouldn’t have to learn Ruby when the rest of the code was PHP.
We use Travis for continuous integration. Naturally, we needed to get our Behat tests running on Travis. Fortunately, there is already a great tutorial from about a year ago explaining how to do this.
Now let’s say you want to take things a step further. Let’s say you want your Behat tests to run on a variety of browsers and operating systems, not just whatever you can pull together on the Linux host running your Travis tests. One possibility is Sauce Labs, which is free for open source projects like Ilios.
Secure Environment Variables
travis Ruby gem to generate secure environment variable values for your
.travis.yml file containing your
SAUCE_USERNAME and your
SAUCE_ACCESS_KEY. See the heplful Travis documentation for more information.
You may be tempted to use the Travis addon for Sauce Connect. I don’t because, using the addon, Travis builds hang (and thus fail) when running the CI tests in a fork. This is because forks cannot read the secure environment variables generated in the previous step.
Instead, I check to see if
SAUCE_USERNAME is available and, if so, then I run Sauce Connect using the same online bash script (located in a GitHub gist) used by the addon provided by Travis. (By the way, you can check for
TRAVIS_SECURE_ENV_VARS if that feels better than checking for
The specific line in
.travis.yml that does this is:
- if [ "$SAUCE_USERNAME" ] ; then (curl -L https://gist.github.com/santiycr/5139565/raw/sauce_connect_setup.sh | bash); fi
Use the Source, Luke
Now it’s time to get Behat/Mink to play nicely with Sauce Labs.
The good news is that there is a
saucelabs configuration option. The bad news is that, as far as I can tell, it is not documented at the current time. So you may need to read the source code if you want to find out about configuration options or troubleshoot. Perhaps it’s intended to be released and documented in the next major release. Regardless, we’re using it and it’s working for us. Enable it in your
default: extensions: Behat\MinkExtension\Extension: saucelabs: ~
We keep our Behat profile for Sauce in it’s own file, because it’s special. Here’s our
Note that we configured our app within Travis-CI to run over HTTPS. In a typical setting, you will want the protocol of your
base_url to specify HTTP instead.
Here’s the line in our
.travis.yml to run our Behat tests using the Sauce profile:
- if [ "$SAUCE_USERNAME" ] ; then (cd tests/behat && bin/behat -c sauce.yml); fi
Of course, if you’re using a different directory structure, you will need to adjust the command to reflect it.
That’s All, Folks!
I hope this has been helpful. It will no doubt be out of date within a few months, as things move quickly with Behat/Mink, Sauce Labs, and Travis-CI. I will try to keep it up to date and put a change log here at the bottom. Or if a better resource for this information pops up, I’ll just put a link at the top. Thank you for reading!
Responsive images and offline application caching do not play well together. (For an explanation, see “Gotcha #6” of Jake Archibald’s application cache article with a moderately NSFW title.)
Pickyfill (partially) solves the problem using Scott Jehl’s picturefill as a starting point. Pickyfill stores responsive images as data URLs in LocalStorage. If your page is using the HTML5 offline Appcache, pickyfill will detect this and store picturefill images as they are loaded. It will only store the images that your device displays, so (for example) an iPhone will only cache iPhone-sized images; it will not download and store crazy large images designed for large screens.
Pickyfill makes the cached images available if the user is offline. It can also improves page load time if the user is on a slow network.
Support, or Where Won’t This Work?
Pickyfill requires ApplicationCache, LocalStorage, and Canvas. If a browser that does not support these features visits a site that uses pickyfill, then pickyfill will do nothing and the experience will gracefully degrade to straight-up picturefill.
|Internet Explorer||10.0+||full support|
|Internet Explorer||< 10.0||degrades to regular picturefill|
|Safari (OS X, Windows)||4.0+||full support|
|Safari (iOS)||3.2+||full support|
|Android||All versions||it’s complicated**|
*In Firefox, pickyfill will cache images on load, but not on resize. This is to avoid caching truncated/corrupted images. On resize, behavior gracefully degrades to regular picturefill behavior.
**Android browser in Android 2.3 (and probably others) does not implement
toDataURL() completely/correctly. Pickyfill will detect the problem and degrade gracefully to regular picturefill behavior. Otherwise, pickyfill is fully supported in Android.
How do I use it?
Use picturefill the same way you would without pickyfill. The only changes will be to load
picturefill.js and to give your site/page an HTML5 Appcache manifest.
<html manifest="/manifest.appcache"> <head> ... <script src="/assets/js/matchmedia.js"></script> <script src="/assets/js/picturefill.js"></script> <script src="/assets/js/pickyfill.js"></script> ...
Although it would be better to minify and concatenate the JS files, the above code is for clarity.
Because pickyfill will only cache images that are actually displayed, it is possible for a user to visit the site, then visit the site again while offline, resize their browser, and end up with a broken image (because the image that is required at the new browser size was never downloaded before and therefore has not been cached). For this reason, it is important to have an appropriate small
FALLBACK image in your offline Appcache.
FALLBACK: imgs/ imgs/fallback.png
Shibboleth authentication via simpleSAMLphp broke on one of our apps when the system clock on the Shibboleth server drifted more than a minute into the future.
I am not the administrator of the Shibboleth server. But it occured to me that even with no shell access to the server, I could detect clock drift by examining the time stamp in the HTTP headers from the server.