You are here

Feed aggregator

LITA: Jobs in Information Technology: December 17

planet code4lib - Wed, 2014-12-17 18:54

New vacancy listings are posted weekly on Wednesday at approximately 12 noon Central Time. They appear under New This Week and under the appropriate regional listing. Postings remain on the LITA Job Site for a minimum of four weeks.

New This Week

Coordinator for Digital Collection Services, Colorado State University, Fort Collins, CO

Visit the LITA Job Site for more available jobs and for information on submitting a  job posting.

DPLA: DPLA Contributes to National History Day in Missouri

planet code4lib - Wed, 2014-12-17 18:24

Serendipity can still top search. I learned about the Digital Public Library of America (DPLA) not online, but in a print article about efforts in Minnesota to share its history digitally. It was intriguing. Thus began my exploration of DPLA: search, sign up, receive some emails, open and read a few, get intrigued and before you know it you are applying to be a DPLA Community Rep.

Brent Schondelmeyer, DPLA Community Rep in Missouri

I live in Missouri – in the vast Middle Border country – which is an odd amalgamation of several histories; border state, Midwest, gateway to the west and approaching its bicentennial of its entrance into the Union. Missouri: the land of Mark Twain, Jesse James, Harry Truman, Scott Joplin, Charlie Parker and Jon Hamm. I was intrigued by the way in which DPLA provides a new way to discover, organize and share a history already known, and to deliver it digitally. The approach is disruptive – often a feature of technology innovation – and challenging to old-school approaches to doing history. Being involved with local and state historical societies you can get a skewed view of the general public interest given that these membership organizations are challenged to renew their base and reassess their role.

But your harsh assessment changes after attending a National History Day in Columbia (MO). It had been some time since I had seen that many folks at a history-related event outside of a local appearance by David McCullough. I left that experience renewed in the thought that local and regional history has a promising, but perhaps, different future. Young students want to discover, share and interpret stories of places, events, and ideas. But they likely will do it new ways. Search and discover begins not at the library, but on an electronic device.

This year DPLA is an official sponsor of National History Day in Missouri. With the excellent assistance of the DPLA staff, teaching guides and materials were prepared sharing DPLA resources related to this year’s theme: Leadership and Legacy in History. Also a prize was created for the student whose work made the best use of DPLA-related resources at the state finals next spring. My simple hope is these young folks will move from searching online, to helping get the history of our state online. Otherwise we may lose much near-history of the Show-Me State or have it hidden to others. We have extensive records of centuries past – paper records stored, few finding aids and limited public hours for research in libraries and archives. Getting online – the indexes, if not actual documents, images and audio files – is essential. And before more recent history is lost–not to the dustbins of history, but landfills–as this “greatest generation” leaves us, we have a chance learn what they did, know and experience.

HangingTogether: Working Together to Clarify Approaches to Library Linked Data

planet code4lib - Wed, 2014-12-17 17:40

As we announced earlier in the month, we have been communicating and collaborating with the Library of Congress over our different approaches to library linked data.

The Library of Congress is developing BIBFRAME, which is slated to eventually replace MARC by providing the added benefits that will accrue to using a linked data solution for our library metadata. Meanwhile, we here at OCLC have a different set of use cases, largely around syndicating library data to the wider web, and we have chosen to base our efforts on the metadata standard most widely adopted by web search engines,

We feel that these approaches are not in competition and by better understanding our different approaches we can all learn about how best to make our data assets available on the web as linked data. The first major step in this process is a whitepaper, currently in collaborative development by OCLC and LC staff, that will compare and contrast our different approaches. The goal is to publish this in time for ALA Midwinter at the end of January 2015, so watch for it!

About Roy Tennant

Roy Tennant works on projects related to improving the technological infrastructure of libraries, museums, and archives.

Mail | Web | Twitter | Facebook | LinkedIn | Flickr | YouTube | More Posts (84)

LITA: Register Now for LITA Midwinter Institutes

planet code4lib - Wed, 2014-12-17 17:14

Whether you’ll be attending Midwinter or are just looking for a great one day continuing education event in the Chicago/Midwest area, we hope you’ll join us.

When? All workshops will be held on Friday, January 30, 2015, from 8:30-4:00 at McCormick Place in Chicago IL.

Cost for LITA Members: $235 (ALA $350 / Non-ALA $380, see below for details)

Here’s this year’s terrific line up:

Developing mobile apps to support field research
Instructor: Wayne Johnston, University of Guelph Library

Researchers in most disciplines do some form of field research. Too often they collect data on paper which is not only inefficient but vulnerable to date loss. Surveys and other data collection instruments can easily be created as mobile apps with the resulting data stored on the campus server and immediately available for analysis. The apps also enable added functionality like improved data validity through use of authority files and capturing GPS coordinates. This support to field research represents a new way for academic libraries to connect with researchers within the context of a broader research date management strategy.

Introduction to Practical Programming
Instructor: Elizabeth Wickes, University of Illinois at Urbana-Champaign

This workshop will introduce foundational programming skills using the Python programming language. There will be three sections to this workshop: a brief historical review of computing and programming languages (with a focus on where Python fits in), hands on practice with installation and the basics of the language, followed by a review of information resources essential for computing education and reference. This workshop will prepare participants to write their own programs, jump into programming education materials, and provide essential experience and background for the evaluation of computing reference materials and library program development. Participants from all backgrounds with no programming experience are encouraged to attend.

From Lost to Found: How user Testing Can Improve the User Experience of Your Library Website
Instructors: Kate Lawrence, EBSCO Information Services; Deirdre Costello, EBSCO Information Services; Robert Newell, University of Houston

When two user researchers from EBSCO set out to study the digital lives of college students, they had no idea the surprises in store for them. The online behaviors of “digital natives” were fascinating: from students using Google to find their library’s website, to what research terms and phrases students consider another language altogether: “library-ese.” Attendees of this workshop will learn how to conduct usability testing, and participate in a live testing exercise via Participants will leave the session with the knowledge and confidence to conduct user testing that will yield actionable and meaningful insights about their audience.

More information about Midwinter Workshops.

Registration Information:
LITA members get one third off the cost of Mid-Winter workshops. Use the discount promotional code: LITA2015 during online registration to automatically receive your member discount. Start the process at the ALA web sites:

Conference web site:
Registration start page: 
LITA Workshops registration descriptions:

When you start the registration process and BEFORE you choose the workshop, you will encounter the Personal Information page. On that page there is a field to enter the discount promotional code: LITA2015
As in the example below. If you do so, then when you get to the workshops choosing page the discount prices, of $235, are automatically displayed and entered. The discounted total will be reflected in the Balance Due line on the payment page.

Please contact the LITA Office if you have any registration questions.

FOSS4Lib Updated Packages: Goobi

planet code4lib - Wed, 2014-12-17 16:10

Last updated December 17, 2014. Created by Peter Murray on December 17, 2014.
Log in to edit this page.

Goobi is an open source software application for digitisation projects and workflow management in libraries, museums and archives.

Goobi allows to model, manage and supervise freely definable production processes. Goobi includes importing data from library catalogues, scanning and content-based indexing and the digital presentation of results in standardised formats.

Package Type: Digital RepositoryLicense: OtherDevelopment Status: Production/Stable Package Links Operating System: Browser/Cross-PlatformTechnologies Used: TomcatProgramming Language: JavaDatabase: MySQLOpen Hub Link: Hub Stats Widget: 

ACRL TechConnect: This is How I Work (Bryan J. Brown)

planet code4lib - Wed, 2014-12-17 14:00

Editor’s Note: This post is part of ACRL TechConnect’s series by our regular and guest authors about The Setup of our work.


After being tagged by Eric Phetteplace, I was pleased to discover that I had been invited to take part in the “This is How I Work” series. I love seeing how other people view work and office life, so I’m happy to see this trend make it to the library world.

Name: Bryan J. Brown (@bryjbrown)

Location: Tallahassee, Florida, United States

Current Gig: Web Developer, Technology and Digital Scholarship, Florida State University Libraries

Current Mobile Device: Samsung Galaxy Note 3 w/ OtterBox Defender cover (just like Becky Yoose!). It’s too big to fit into my pants pocket comfortably, but I love it so much. I don’t really like tablets, so having a gigantic phone is a nice middle ground.

Current Computer: 15 inch MacBook Pro w/ 8GB of RAM. I’m a Linux person at heart, but when work offers you a free MBP you don’t turn it down. I also use a thunderbolt monitor in my office for dual-screen action.

Current Tablet: 3rd gen. iPad, but I don’t use it much these days. I bought it for reading books, but I strongly prefer to read them on my phone or laptop instead. The iPad just feels huge and awkward to hold.

One word that best describes how you work: Structured. I do my best when I stay within the confines of a strict system and/or routine that I’ve created for myself, it helps me keep the chaos of the universe at bay.

What apps/software/tools can’t you live without?

Unixy stuff:

  • Bash: I’ve tried a few other shells (tcsh, zsh, fish), but none have inspired me to switch.
  • Vim: I use this for everything, even journal entries and grocery lists. I have *some* customizations, but it’s pretty much stock (except I love my snippets plugin).
  • tmux: Like GNU Screen, but better.
  • Vagrant: The idea of throwaway virtual machines has changed the way I approach development. I do all my work inside Vagrant machines now. When I eventually fudge things, I can just run ‘vagrant destroy’ and pretend it never happened!
  • Git: Another game changer. I shouldn’t have waited so long to learn about version control. Git has saved my bacon countless times.
  • Anaconda: I’m a Python fan, but I like Python 3 and the scientific packages. Most systems only have Python 2, and a lot of the scientific packages fail to build for obscure reasons. Anaconda takes care of all that nonsense and allows you to have the best, most current Python goodness on any platform. I find it very comforting to know that I can use my favorite language and packages everywhere no matter what.
  • Todo.txt-CLI: A command line interface to the Todo.txt system, which I am madly in love with. If you set it to save your list to Dropbox, you can manage it from other devices, too. My work life revolves around my to-do list which I mostly manage at my laptop with Todo.txt-CLI.


  • Dropbox: Keeping my stuff in order across machines is a godsend. All my most important files are kept in Dropbox so I can always get to them, and being able to put things in a public folder and share the URL is just awesome.
  • Google Drive: I prefer Dropbox better for plain storage, but the ability to write documents/spreadsheets/drawings/surveys at will, store them in the cloud, share them with coworkers and have them write along with you is too cool. I can’t imagine working in a pre-Drive world.
  • Trello: I only recently discovered Trello, but now I use it for everything at work. It’s the best thing for keeping a group of people on track with a large project, and moving cards around is strangely satisfying. Also you can put rocket stickers on cards.
  • Quicksilver for Mac: I love keyboard shortcuts. A lot. Quicksilver is a Mac app for setting up keyboard shortcuts for everything. All my favorite apps have hotkeys now.
  • Todo.txt for Android: A nice mobile interface for the Todo.txt system. One of the few apps I’ve paid money for, but I don’t regret it.
  • Plain.txt for Android: This one is kind of hard to explain until you use it. It’s a mobile text editor for taking notes that get saved in Dropbox, which is useful in more ways than you can imagine. Plain.txt is my mobile interface to the treasure trove of notes I usually write in Vim on my laptop. I keep everything from meeting notes to recipes (as well as the previously mentioned grocery lists and journal entries) in it. Second only to Todo.txt in helping me stay sane.

What’s your workspace like?

My office is one of my favorite places. A door I can shut, a big whiteboard and lots of books and snacks. Who could ask for more? I’m trying out the whole “standing desk” thing, and slowly getting used to it (but it *does* take some getting used to). My desk is multi-level (it came from a media lab that no longer exists where it held all kinds of video editing equipment), so I have my laptop on a stand and my second monitor on the level above it so that I can comfortably look slightly down to see the laptop or slightly up to see the big display.

What’s your best time-saving trick?

Break big, scary, complicated tasks into smaller ones that are easy to do. It makes it easier to get started and stay on track, which almost always results in getting the big scary thing done way faster than you thought you would.

What’s your favorite to-do list manager?

I am religious about my use of Todo.txt, whether from the command line or with my phone. It’s my mental anchor, and I am obsessive about keeping it clean and not letting things linger for too long. I prioritize things as A (get done today), B (get done this week), C (get done soon), and D (no deadline).

I’m getting into Scrum lately, so my current workflow is to make a list of everything I want to finish this week (my sprint) and mark them as B priority (my sprint backlog, either moving C tasks to B or adding new ones in manually). Then, each morning I pick out the things from the B list that I want to get done today and I move them to A. If some of the A things are complicated I break them into smaller chunks. I then race myself to see if I can get them all done before the end of the day. It turns boring day-to-day stuff into a game, and if I win I let myself have a big bowl of ice cream.

Besides your phone and computer, what gadget can’t you live without?

Probably a nice, comfy pair of over-the-ear headphones. I hate earbuds, they sound thin and let in all the noise around you. I need something that totally covers my ears to block the outside world and put me in a sonic vacuum.

What everyday thing are you better at than everyone else?

I guess I’m pretty good at the whole “Inbox Zero” thing. I check my email once in the morning and delete/reply/move everything accordingly until there’s nothing left, which usually takes around 15 minutes. Once you get into the habit it’s easy to stay on top.

What are you currently reading?

  • The Information by James Gleick. I’m reading if for Club Bibli/o, a library technology bookclub. We just started, so you can still join if you like!
  • Pro Drupal 7 Development by Todd Tomlinson and John K. VanDyk. FSU Libraries is a Drupal shop, so this is my bread and butter. Or at least it will be once I get over the insane learning curve.
  • Buddhism Plain and Simple by Steve Hagen. The name says it all, Steve Hagen is great at presenting the core parts of Buddhism that actually help you deal with things without all the one hand clapping nonsense.

What do you listen to while you work?

Classic ambient artists like Brian Eno and Harold Budd are great when I’m in a peaceful, relaxed place, and I’ll listen to classical/jazz if I’m feeling creative. Most of the time though it’s metal, which is great for decimating to-do lists. If I really need to focus on something, any kind of music can be distracting so I just play static from This blocks all the sound outside my office and puts me in the zone.

Are you more of an introvert or an extrovert?

Introvert for sure. I can be sociable when I need to, but my office is my sanctuary. I really value having a place where I can shut the door and recharge my social batteries.

What’s your sleep routine like?

I’ve been an early bird by necessity since grad school, the morning is the best time to get things done. I usually wake up around 4:30am so I can hit the gym when it opens at 5am (I love having the whole place to myself). I start getting tired around 8pm, so I’m usually fast asleep by 10pm.

Fill in the blank: I’d love to see _________ answer these same questions.

Richard Stallman. I bet he’d have some fun answers.

What’s the best advice you’ve ever received?

Do your best. As simple as it sounds, it’s a surprisingly powerful statement. Obviously you can’t do *better* than your best, and if you try your best and fail then there’s nothing to regret. If you just do the best job you can at any given moment you’ll have the best life you can. There’s lots of philosophical loopholes buried that perspective, but it’s worked for me so far.

Hydra Project: Sufia 4.3.1 released

planet code4lib - Wed, 2014-12-17 13:46

We are pleased to announce the release of version 4.3.1 of the Sufia gem.  (The 4.3.0 version was a misfire, so the upgrade path goes from 4.2.0 directly to 4.3.1.)

The 4.3.1 release of Sufia includes the ability to store users’ ORCID identifiers and display them in the user profile, and includes enhancements to usage statistics. It also includes support for Blacklight 5.8 and a good number of bugfixes.

Thanks to Carolyn Cole, Michael Tribone, Valerie Maher, Adam Wead, Misty DeMeo, and Mike Giarlo for their work on this release.

View the upgrade notes and a complete changelog on the release page:

Hydra Project: Submit Your OR2015 Proposal: Conference System Now Open

planet code4lib - Wed, 2014-12-17 13:42

Of interest to the Hydra Community:

The Tenth International Conference on Open Repositories, OR2015, will be held June 8-11, 2015 in Indianapolis (Indiana, USA). The organizers are pleased to invite you to contribute to the program.

The conference theme is LOOKING BACK, MOVING FORWARD: OPEN REPOSITORIES AT THE CROSSROADS. It is an opportunity to reflect on and to celebrate the transformative changes in repositories, scholarly communication, and research data over the last decade. More critically, however, it will also help to ensure that open repositories continue to play a key role in supporting, shaping, and sharing those changes and an open agenda for research and scholarship.

The organizers invite you to review the full call for proposals here:, and to submit your proposal here: by January 30, 2015. There are several different formats provided to encourage your participation in this year’s conference, all described on the OR2015 website.


The Open Repositories Steering Committee is pleased to announce the release of the new OR Code of Conduct The OR Code of Conduct underscores the OR Conference core value of openness by providing a welcoming and positive experience for everyone, whether they are in a formal session or a social setting, or are taking part in activities online.


  • 30 January 2015: Deadline for submissions and Scholarship Programme applications
  • 27 March 2015: Submitters notified of acceptance to general conference
  • 10 April 2015: Submitters notified of acceptance to Interest Groups
  • 8-11 June 2015: OR2015 conference

The conference system is now open and  is linked from the conference web site: We look forward to welcoming you to Indianapolis!



Program Co-Chairs

  • Holly Mercer, University of Tennessee
  • William J Nixon, University of Glasgow
  • Imma Subirats, FAO of the United Nations


Terry Reese: MarcEdit 6 Update Posted

planet code4lib - Wed, 2014-12-17 06:25

A new update has been posted.  The changes are noted below:

  • Enhancement: Installation changes – for administrators, the program will now allow for quite installations and an option to prevent users from enabling automated updates.  Some IT admins have been asking for this for a while.  The installation program will take an command-line option: autoupdate=no, to turn this off.  The way this is disabled (since MarcEdit manages individual profiles) is a file will be created into the program directory that if present, will present automatic updates.  This file will be removed (or not recreated) if this command-line isn’t set – so users doing automated installations will need to remember to always set this value if they wish to prevent this option from being enabled.  I’ve also added a not in the Preferences window noting if the administrator has disabled the option.
  • Bug Fix: Swap Field Task List – one of the limiters wasn’t being passed (the process one field per swap limiter)
  • Bug Fix: Edit Field Task List – when editing a control field, the positions text box wasn’t being shown. 
  • Bug Fix: Edit Field Regular Expression options – when editing control fields, the edit field function evaluated the entire field data – not just the items to be edited.  So, if I wanted to use a regular expression to evaluate two specific values, I couldn’t.  This has been corrected.
  • Enhancement: Linked Data Linker – added support for FAST headings. 
  • Bug Fix: Linked Data Linker – when processing data against LC’s, some of the fuzzy matching was causing values to be missed.  I’ve updated the normalization to correct this.
  • Enhancement: Edit Subfield Data – Moving Field data – an error can occur if the field having data moved to is a control field, and the control field is smaller than the position where the data should be moved to.  An error check has been added to ensure this error doesn’t pop up.
  • Bug Fix: Auto Translation Plug-in – updated code because some data was being dropped on translation, meaning that it wouldn’t show up in the records.

Update can be found at: or via the automated updating tool.  The plug-in updates can be downloaded via the Plug-in Manager within the MarcEdit Application.


Information Technology and Libraries: Usability Testing for Greater Impact: A Primo Case Study

planet code4lib - Wed, 2014-12-17 05:00

This case study focuses on a usability test conducted by four librarians at Texas Tech University. Eight students were asked to complete a series of tasks using OneSearch, the TTU Libraries’ implementation of the Primo discovery tool. Based on the test, the team identified three major usability problems, as well as potential solutions. These problems typify the difficulties patrons face while using library search tools, but have a variety of simple solutions.


Library Tech Talk (U of Michigan): Practical Relevance Ranking for 11 Million Books, Part 3: Document Length Normalization

planet code4lib - Wed, 2014-12-17 00:00
Relevance is a complex concept which reflects aspects of a query, a document, and the user as well as contextual factors. Relevance involves many factors such as the user's preferences, task, stage in their information-seeking, domain knowledge, intent, and the context of a particular search. This post is the third in a series by Tom Burton-West, one of the HathiTrust developers, who has been working on practical relevance ranking for all the volumes in HathiTrust for a number of years.

District Dispatch: Families and Work Institute seeks library and museum input

planet code4lib - Tue, 2014-12-16 19:13

If your library or museum has organized programs to help children develop their skills and their mind, the Families and Work Institute (FWI) asks that you fill out their brief survey. Data pulled from this survey will aid FWI in developing a nation report on the roles that libraries and museums play in supporting children, families, and the professionals who work with them.

For the past 14 years, Mind in the Making, a program of FWI, has been sharing research on what we can do to help children thrive now and in the future. They have been calling attention to the importance of early brain development and promoting Executive Function skills, which study after study reveals have been a critical missing piece in efforts to promote school readiness and school, work and life success.

Surveys should be submitted by December 22, 2014 to be included. More information is available at the IMLS blog, Museums and Libraries: Be a Part of our Brain Building Journey.

The post Families and Work Institute seeks library and museum input appeared first on District Dispatch.

Jenny Rose Halperin: Leaving Mozilla as staff

planet code4lib - Tue, 2014-12-16 16:58

December 31 will be my last day as paid staff on the Community Building Team at Mozilla.

One year ago, I settled into a non-stop flight from Raleigh, NC to San Francisco and immediately fell asleep. I was exhausted; it was the end of my semester and I had spent the week finishing a difficult databases final, which I emailed to my professor as soon as I reached the hotel, marking the completion of my coursework in Library Science and the beginning of my commitment to Mozilla.

The next week was one of the best of my life. While working, hacking, and having fun, I started on the journey that has carried me through the past exhilarating months. I met more friendly faces than I could count and felt myself becoming part of the Mozilla community, which has embraced me. I’ve been proud to call myself a Mozillian this year, and I will continue to work for the free and open Web, though currently in a different capacity as a Rep and contributor.

I’ve met many people through my work and have been universally impressed with your intelligence, drive, and talent. To David, Pierros, William, and particularly Larissa, Christie, Michelle, and Emma, you have been my champions and mentors. Getting to know you all has been a blessing.

I’m not sure what’s next, but I am happy to start on the next step of my career as a Mozillian, a community mentor, and an open Web advocate. Thank you again for this magical time, and I hope to see you all again soon. Let me know if you find yourself in Boston! I will be happy to hear from you and pleased to show you around my hometown.

If you want to reach out, find me on IRC: jennierose. All the best wishes for a happy, restful, and healthy holiday season.

District Dispatch: Down with the divide – support the E-rate program

planet code4lib - Tue, 2014-12-16 16:35

Librarians know how essential the E-rate has been and will be to meeting their communities’ needs for high-speed, broadband internet service and public access to the internet in the 21st century. That’s why ALA fought hard to create the program and, for the past 18 months, to encourage the Federal Communications Commission to dramatically increase the program’s funding and streamline its application procedures.

Libraries did it! Starting in 2015, an additional $1.5 billion will be available to libraries across the country that they can use to further narrow and ultimately close the digital divide . . . and funds will be easier to apply for.

According to the FCC, the E-rate modernization will make the program more efficient, maximize the use of ratepayer funds, and will provide support for libraries and schools across the country.  An FCC fact sheet notes that “the demand for broadband is growing at least 50% per year, which means that total bandwidth costs will continue to grow even with significant broadband price reductions…Chairman Wheeler’s…$1.5 billion cap increase is consistent with all schools and libraries achieving the long-term goals…because Wi-Fi within every classroom and library space is an essential element of 21st century learning.”

Some Members of Congress have expressed concerns with the FCC action and may not fully appreciate the urgent need in our library community for E-rate modernization. Some have gone as far as questioning the justification of the E-rate program’s existence at all.

Your help is needed to ensure Congress does not overturn the additional E-rate funding for our patrons. We urge you to contact your members of Congress during the December recess and inform them of what services E-rate enables you to supply to your community.

Our message to Congress:

  • Libraries across the country are far behind the broadband capacity they need. A 21st century E-rate program with additional funding will allow libraries to offer state-of-the-art connectivity and critical services to patrons. Many patrons can only access the internet through libraries.
  • The current 20th Century E-rate program has failed to keep pace with inflationary cost increases and has resulted in cost-prohibitive commercially available connectivity. Bringing the program into the 21st Century ensures libraries can secure affordable high-speed connectivity for their patrons.
  • The increasing demands on patrons to connect to the internet – for employment and entrepreneurship, education, community engagement, and individual empowerment – has placed tremendous need for greater bandwidth and faster access.
  • E-rate modernization benefits patrons at libraries of all sizes and in communities across the country, whether urban, suburban, or rural.
  • Please provide Congress with examples of the range of programs and services you offer to patrons benefiting the local community.

We have previously reported on E-rate here and here. Click here to take action now.

The post Down with the divide – support the E-rate program appeared first on District Dispatch.

David Rosenthal: Hardware I/O Virtualization

planet code4lib - Tue, 2014-12-16 16:00
At, Timothy Prickett Morgan has an interesting post entitled A Rare Peek Into The Massive Scale Of AWS. It is based on a talk by Amazon's James Hamilton at the re:Invent conference. Morgan's post provides a hierarchical, network-centric view of the AWS infrastructure:
  • Regions, 11 of them around the world, contain Availability Zones (AZ).
  • The 28 AZs are arranged so that each Region contains at least 2 and up to 6 datacenters.
  • Morgan estimates that there are close to 90 datacenters in total, each with 2000 racks, burning 25-30MW.
  • Each rack holds 25 to 40 servers.
AZs are no more than 2ms apart measured in network latency, allowing for synchronous replication. This means the AZs in a region are only a couple of kilometres apart, which is less geographic diversity than one might want, but a disaster still has to have a pretty big radius to take out more than one AZ. The datacenters in an AZ are not more than 250us apart in latency terms, close enough that a disaster might take all the datacenters in one AZ out.

Below the fold, some details and the connection between what Amazon is doing now, and what we did in the early days of NVIDIA.

Amazon uses custom-built hardware, including network hardware, and their own network software. Doing so is simpler and more efficient than generic hardware and software because they only need to support a very restricted set of configurations and services. In particular they build their own network interface cards (NICs). The reason is particularly interesting to me, as it is to solve exactly the same problem that we faced as we started NVIDIA more than two decades ago.

The state-of-the-art of PC games, and thus PC graphics, were based on Windows, at that stage little more than a library on top of MS-DOS. The game was the only application running on the hardware. It didn't have to share the hardware with, and thus need the operating system (OS) to protect it from, any other application. Coming from the Unix world we knew how the OS shared access to physical hardware devices, such as the graphics chip, among multiple processes while protecting them (and the operating system) from each other. Processes didn't access the devices directly, they made system calls which invoked device driver code in the OS kernel that accessed the physical hardware on their behalf.

We understood that Windows would have to evolve into a multi-process OS with real inter-process protection. Our problem, like Amazon's, was two-fold; latency and the variance of latency. If the games were to provide arcade performance on mid-90s PCs, there was no way the game software could take the overhead of calling into the OS to perform graphics operations on its behalf. It had to talk directly to the graphics chip, not via a driver in the OS kernel.

If there would have been only a single process, such as the X server, doing graphics this would not have been a problem. Using the Memory Management Unit (MMU), the hardware provided to mediate access of multiple processes to memory, the OS could have mapped the graphic chip's IO registers into that process' address space. That process could access the graphics chip with no OS overhead. Other processes would have to use inter-process communications to request graphics operations, as X clients do.

SEGA's Virtua Fighter on NV1Because we expected there to be many applications simultaneously doing graphics, and they all needed low, stable latency, we needed to make it possible for the OS safely to map the chip's registers into multiple processes at one time. We devoted a lot of the first NVIDIA chip to implementing what looked to the application like 128 independent sets of I/O registers. The OS could map one of the sets into a process' address space, allowing it to do graphics by writing directly to these hardware registers. The technical name for this is hardware I/O virtualization; we pioneered this technology in the PC space. It provided the very low latency that permitted arcade performance on the PC, despite other processes doing graphics at the same time. And because the competition between the multiple process' accesses to their virtual I/O resources was mediated on-chip as it mapped the accesses to the real underlying resources, it provided very stable latency without the disruptive long tail that degrades the user experience.

Amazon's problem was that, like PCs running multiple graphics applications on one real graphics card, they run many virtual machines (VMs) on each real server. These VMs have to share access to the physical network interface card (NIC). Mediating this in software in the hypervisor imposes both overhead and variance. Their answer was enhanced NICs:
The network interface cards support Single Root I/O Virtualization (SR-IOV), which is an extension to the PCI-Express protocol that allows the resources on a physical network device to be virtualized. SR-IOV gets around the normal software stack running in the operating system and its network drivers and the hypervisor layer that they sit on. It takes milliseconds to wade down through this software from the application to the network card. It only takes microseconds to get through the network card itself, and it takes nanoseconds to traverse the light pipes out to another network interface in another server. “This is another way of saying that the only thing that matters is the software latency at either end,” explained Hamilton. SR-IOV is much lighter weight and gives each guest partition on a virtual machine its own virtual network interface card, which rides on the physical card.This, as shown on Hamilton's graph, provides much less variance in latency:
The new network, after it was virtualized and pumped up, showed about a 2X drop in latency compared to the old network at the 50th percentile for latency on data transmissions, and at the 99.9th percentile the latency dropped by about a factor of 10X. The importance of reducing the variance of latency for Web services at Amazon scale is detailed in a fascinating, must-read paper, The Tail At Scale by Dean and Barroso.

Amazon had essentially the same problem we had, and came up with the same basic hardware solution - hardware I/O virtualization.

DuraSpace News: DSpace-CRIS News from Cineca

planet code4lib - Tue, 2014-12-16 00:00

FromMichele Mennielli, Cineca, International Relations

DuraSpace News: NOW OPEN: OR2015 Conference System–Submit Your Open Repositories Conference Proposal

planet code4lib - Tue, 2014-12-16 00:00

From the organizers of the Open Repositories 2015 (OR2015) Conference.

Indianapolis, IN  The Tenth International Conference on Open Repositories, OR2015, will be held June 8-11, 2015 in Indianapolis (Indiana, USA). The organizers are pleased to invite you to contribute to the program.

HangingTogether: Gifts for archivists and librarians: from the practical to the luxurious

planet code4lib - Mon, 2014-12-15 23:55

We asked for suggestions for gifts that would be suitable for librarians or archivists and the community responded! Thank you so much for all the wonderful and thoughtful gift ideas!

Here are the nominations: if you have other ideas please leave them in the comments below. To ensure that you get what you want, think about leaving this page on computers in your reading room or information commons — I’m sure that certain someone will get the hint.

Lumio Book Lamp

Practical gifts: some information professionals are very focused on getting the job done. For these folks, a gift that helps them do the work at hand is just the thing. Gifts in this category include:

  • A mobile scanner: Laura suggests that perhaps the Flip-Pal might be useful for those who are zipping around “scanning madly.”
  • Of course it’s not all about shelving books or arranging collections. We also attend lots of meetings and conferences. How about a fountain pen? Nadia Nasr suggests the Cross Stratford as a nice looking model that’s affordable.
  • For all that professional reading, what about a book shaped lamp? Lumio’s book lamp (although pricey) was suggested by Stephanie as being “pretty rad.” Comes in dark walnut and blonde maple to compliment any decor.
  • What is more painful that losing your place in a book? Hunting around for a bookmark. Lynn Jones suggests the Albatros Bookmark — you never need to look for your bookmark because it’s in the book — it also places itself.  Comes in packs of 6.
  • What about a card catalog shaped flash drive? These will be available soon from Unshelved. Thanks to Carol Street for the suggestion.

The Archivist Wine

Food and drink: everyone likes to eat and drink. Here are some suggestions vetted by librarians and archivists

  • The chefs among us might appreciate cookbooks from historical societies. Melissa M. loves her cookbooks from the King’s Landing Historical Settlement in New Brunswick. I couldn’t find those online but you can find plenty of good ideas in Cookbook Finder. I noted that King’s Landing does have an historic inn that serves period food, so check with your local historical society!
  • Beer for archivists: Although I normally hate to reinforce stereotypes about archivists that involve either attics or cellars, I was pleased to hear Jill Tatem’s nomination for Cellar Dweller, which is only available at the Great Lakes Brewing Company in Cleveland. Since this is the site of the 2015 Society of American Archivists annual meeting, I know many archivists will take a rain check on this brew.
  • A toast to archivists! From Sonoma Estate Vintners, the Archivist. Pick your poison: cab, chardonnay, or pinot noir. The description includes the word “appraise” so you know you are in the right place.


Pride and Prejudice Tote

Clothing and accessories: suggestions range from items that are practical to those that show your style.

  • Melissa M. says, “every processing archivist could use steel-toed boots (required for the first archival job I ever had, and I actually managed to find quite a stylish pair).” Melissa was not able to find her boots, which fetched compliments outside the workplace, but perhaps something like these engineer boots would work.
  • To go with your boots, perhaps some library card socks from NYPL? (Hat tip to Bruce Washburn.)
  • You can wear your heart on your sleeve, and now you can wear your favorite book, as a t-shirt, or water-resistant tote. From Lithographs. Also available, posters and (temporary) tatoos. From Lorcan Dempsey and Pam Kruger.
  • A favorite from last year was the microfiche jewelery from Oinx. Styles have been updated and now you and spread the “I’d rather be fiching” message via t-shirt and bumper sticker.

Mini Hollinger Document Cartons

Little luxuries: sometimes it’s the little things

  • Candles are a great seasonal gift. You can choose between The Archivist candles from Greenmarket (lots of choose from, particularly if you like the idea of “fragrance records accumulated to preserve moments, stories, and people they represent”) and Library candles from Paddywax (which feature scents that will conjure your favorite author). Thanks to Casey Davis and Carol Street for calling these to our attention!
  • Hollinger boxes are a staple for archivists, and mini document boxes have long been a popular giveaway at conferences — so popular that Hollinger now sells them as a separate item. Jennifer suggests that in addition to being just plain adorable, they would be the perfect way to pop the question.
  • Cream for hands, dried out from processing documents and handling other materials, was a popular item on last year’s list. This year, Melissa M. recommends Lush’s Charity Pot lotion.

Can’t buy happiness: of course, the things that everyone really wants can’t be purchased. At the top of almost every information professional’s wish list is space (to put anything, as our anonymous contributor put it). Another thing that we’d all like to see is reflected in this lovely blog post by Maarja Krusten:

…the greatest gift you can give archivists and librarians is the opportunity to share physically and virtually the knowledge found in their collections and holdings.

Now, that sentiment is something I think we can all get behind! Happy holidays to all of you!

About Merrilee Proffitt

Mail | Web | Twitter | Facebook | LinkedIn | More Posts (276)

Information Technology and Libraries: President's Message: Twitter Nodes to Networks: Thoughts on the #litaforum

planet code4lib - Mon, 2014-12-15 23:28
President's Message: Twitter Nodes to Networks: Thoughts on the #litaforum

Harvard Library Innovation Lab: Link roundup December 15, 2014

planet code4lib - Mon, 2014-12-15 19:44

Beach balls, stickers, books, moveable cities, and figuring out what you want. This is a good batch of links.

Eyeo 2014 – Santiago Ortiz

“A question that should be answered with action, not thought.”

The City That Is Moving Down the Road

What if the city were on legs? Fascinating piece on potential mobility of libraries, and other community commons.

Best Books of 2014 : NPR

What if the library had an interface like this, but weekly? Acquisitions of the week. Good covers. Fun to browse.

Who’s been naughty and nice this year? Our 2014 draft survey results | SchoolStickers

Clever use of data for holiday fun.

Spinning Beach Ball of Death

The spinning beach ball can be beautiful.


Subscribe to code4lib aggregator