You are here

Feed aggregator

DuraSpace News: Quarterly Report from Fedora, October - December 2014

planet code4lib - Tue, 2015-03-03 00:00

From The Fedora Steering Group

Fedora Development - In the past quarter, the development team released the production release of Fedora 4.0; detailed release notes are here:

SearchHub: Thoughts on “Search vs. Discovery”

planet code4lib - Mon, 2015-03-02 22:21
“Search vs discovery” is a common dichotomy that is used in discussions about search technology, where the former is about finding specific things that are either known or assumed to exist, and the latter is about using the search/browse interface to discover what content is available. A single user session may include both of these “agendas”, especially if a users’ assumption that a certain piece of content exists, is not quickly verified by finding it. Findability is impaired when there are too many irrelevant or noise hits (false positives), which obscures or camouflages the intended results. This happens when metadata is poorly managed, search relevance is poorly tuned or when the users’ query is ambiguous and no feedback is provided by the application (such as autocomplete, recommendation or did you mean) to help improve it. Content Visibility Content visibility is important because a document must first be included in the result set to be found (obviously), but it is also critical for discovery especially with very large content sets. User experience has shown that faceted navigation is one of the best ways to provide this visualization especially if it includes dimensions that focus on “aboutness” and “relatedness”. However if a document is not appropriately tagged, it may become invisible to the user once the facet that it should be included in (but is not) is selected. Data quality really matters here! (My colleague Mark Bennett has authored a Data Quality Toolkit to help with this.  The venerable Lucene Index Toolbox or “Luke” which can be used to inspect the back end Lucene index is also very useful. The LukeRequestHandler is bundled with Solr. ) Without appropriate metadata, the search engine has no way of knowing what is related to what. Search engines are not smart in this way – the intelligence of a search application is built into its index. Search and Content Curation Findability and visibility are also very important when the search application is used as a tool for content curation within an organization. Sometimes, the search agenda is to see if something has been created before, as a “do diligence” activity before creating it. Thus, the phrase “out of sight, out of mind” becomes important when content that can’t be found tends to be re-created. This leads to unnecessary duplication, which is wasteful but also counter-productive to search both by adding to the repository size and by increasing the possibility of obfuscation by similarity. Applying “deduplication” processes after the fact is a band-aid – we should make it easier to find things in the first place so we don’t have to do more work later to clean up the mess. We also need to be confident in our search results, so that if we don’t find it, it is likely that it doesn’t exist – see my comments on this point in Introducing Query Autofiltering. Note that this is always a slippery slope. In science, absence of evidence does not equate to evidence of absence – hence “Finding Bigfoot”!  (If they ever do find “Squatch” then no more show – or they have to change the title to “Bigfoot Found!” – which would be very popular but also couldn’t be a series!  That’s OK, I only watched it once to discover that they don’t actually “find” Bigfoot – hence the ‘ing’ suffix. I suppose that “Searching for” sounds too futile to tune it in even once.) Auto-classification Tuning Auto-classification technology is a potential cure in all of the above cases, but can also exacerbate the problem if not properly managed. Machine Learning approaches or using an ontologies and associated rules, provide ways to enhance the relevance of important documents and to organize them in ways that improve both search and discovery. However, in the early phases of development it is likely that an auto-classification system will make two types of errors, that if not fixed can lead to problems of both findability and visibility.  First, it will tag documents erroneously leading to the camouflage or noise problem and second, it will not tag documents that it should – leading to a problem with content visibility. We call these “precision” and “recall” errors respectively. The recall error is especially insidious because if not detected will cause documents to be dropped from consideration when a navigation facet is clicked. Also, errors of omission are more difficult to detect, and require the input of persons who understand the content set well enough to know what the autoclassifier “should” do. Manual tagging, while potentially more accurate, is simply not feasible in many cases because Subject Matter Experts are difficult to outsource. Data quality analysis/curation is the key here. Many times the problem is not the search engines fault. Garbage-In-Garbage-Out as the saying goes. Data Visualization – Search-Driven Analytics I think that one of the most exciting usages of search as a discovery tool is the combination of the search paradigm with analytics. This used to be the purview of the relational database model which is at the core of what we call “Business Intelligence” or BI. Reports generated by analysts from relational data go under the rubric of OLAP (online analytical processing) which typically involves a Data Analyst who designs a set of relational queries, the output of which are then input to a graphing engine to generate a set of charts. When the data changes, the OLAP “cube” is re-executed and a new report emerges. Generating new ways to look at the data require the development, testing, etc of new cubes. This process by its very nature leads to stagnation – cubes are expensive to create and this may stifle new ideas since there is some expert labor required to bring these ideas to fruition. Search engines and relational databases are very different animals. Search engines are not as good as RDBMS at several things – ACID transactions, relational joins, etc — but they are much better at dealing with complex queries that include both structured and unstructured (textual) components. Search indexes like Lucene can include numerical, spatial and temporal data alongside textual information.  Using facets, they can also count things that are the output of these complex queries. This enables us to ask more interesting questions about data – questions that get to “why” something happened rather than just “what”.  Furthermore, recent enhancements to Solr have added statistical analyses to the mix – we can now develop highly interactive data discovery/visualization applications which remove the data analyst from the loop. While there is still a case for traditional BI, search-driven discovery will fill the gap by allowing any user – technical or not – to do the “what if” questions. Once an important analysis has been discovered, it can be encapsulated as an OLAP cube so that the intelligence of its questions can be productized/disseminated. Since this section is about visualization and there are no pictures in this post, you may want to “see” examples of what I am talking about. First, check out Chris Hostetter (aka “Hoss”)’s blog post “Hey, You Got Your Facets in My Stats! You Got Your Stats In My Facets!!” , and his earlier post on pivot facets. Another way cool demonstration of this capability comes from Sam Mefford when he worked at Avalon Consulting – this is a very compelling demonstration of how faceted search can be used as a discovery/visualization tool. Bravo Sam! This is where the rubber meets the road folks!

The post Thoughts on “Search vs. Discovery” appeared first on Lucidworks.

District Dispatch: Free webinar: Expanding immigrant access through libraries

planet code4lib - Mon, 2015-03-02 22:17

Hartford Public Library

Library services to immigrants are extensive and include world language collections, multicultural programming, ESL, citizenship, computer classes, and information brokering. Learn how your library can better support immigrants in “We Belong Here: Expanding Immigrant Access to Government and Community,” a free webinar hosted by e-government service Lib2gov from the American Library Association’s Washington Office and University of Maryland’s iPAC.

This webinar will focus on e-government services that open access for immigrants, using the Hartford Public Library’s American Place Initiative as a national model for immigrant services, resources, and engagement through public libraries.

Homa Naficy, chief adult learning officer for the Hartford Public Library, will lead the interactive webinar. Homa Naficy joined Hartford Public Library in 2000 to design and direct The American Place program for Hartford’s immigrants and refugees. Born in Paris, a native of Iran and now an American citizen, Multicultural Services Director Homa Naficy began her library career as a reference librarian at Newark Public Library. Before joining the staff of Hartford Public Library, she served as a reference librarian at Yonkers Public Library and later as librarian for Adult Services and Outreach for the Westchester Library System.

The American Place has become a magnet for new arrivals seeking immigration information, resources for learning English and preparing for United States citizenship. In 2010, the program was awarded two major grants, a citizenship education grant from the United States Citizenship and Immigration Services (the only library in the nation to receive such funding), and a National Leadership grant from the Institute of Museum and Library Services designed to promote immigrant civic engagement. On completion, this project will serve as a model for other libraries nationally. The American Place program is also the only library in the state to receive funding for adult basic education from the Connecticut Department of Education. In 2001, Ms. Naficy received the Connecticut Immigrant of the Year Award, and in 2013 she was chosen a “Champion of Change” by The White House.

Webinar title: We Belong Here: Expanding Immigrant Access to Government and Community
Date: March 11, 2015
Time: 2:00-3:00 p.m. EST
Register now

The webinar will be archived.

The post Free webinar: Expanding immigrant access through libraries appeared first on District Dispatch.

Islandora: Islandora/Fedora 4 Project Update II

planet code4lib - Mon, 2015-03-02 20:21

On Friday, February 27th, the Fedora 4 Interest Group met for the second time to discuss the progress of our big upgration (the first meeting was back at the end of January). The full notes from the meeting are here, but I'll summarize some of the highlights:

Project Updates

The project has entered its second month with plenty accomplished. Nick was sent to Code4Lib 2015 in Portland, Oregon to work with our Technical Lead, Danny Lamb. The two worked on the proof-of-concept, and it was presented as a lightning talk (video demo). Additionally, Nick and Danny worked with the Hydra and Fedora communities on a shared data model, Hydra Works, which evolved into the Fedora Community Data Model.

After Code4Lib 2015, Nick and Danny focused on updating the Technical Design document, that provides:

  1. an understanding of the Islandora 7.x-2.x design rationale
  2. the importance of using an integration framework
  3. the use of camel
  4. inversion of control and camel
  5. camel and scripting languages
  6. Islandora Sync
  7. Solr and Triple store indexing
  8. Islandora (Drupal).

Or, to sum up the new ways of Islandora in one imge:

Nick and Danny also focused on the development virtual environment (DevOps) for the project. Nick decided to move away from using Chef and Berkshelf due to dependency support. The DevOps setup was moved to basic bash scripts and Vagrant. Contributors to the project can now spin a virtual development environment (which includes the proof-of-concept) in about 5 minutes with a single command: vagrant up. Instructions here.

Nick also focused on project documentation and documentation deployment. All document for the project resides in the git repository for the project, in Markdown format. The documentation can be generated into a static site with MkDocs and thendeployed to GitHub Pages. The documentation for the project can be viewed here, and information about how the documentation is built and deployed can be found here. There is also an outline of how you can contribute to the project here (regardless of your background. We need far more than programmers).

A new use case template makes bringing your ideas to the table much easier. Check out some of the existing use cases for examples - and add yours!

Nick, Danny, and Melissa also did an interview for Duraspace.


The upgration portion of the project is dependant on a couple of sub-items of the project to play out, but continues in tandem.

The first sub-item is the Fedora Audit Service. The Islandora community make use of the audit service in Fedora 3.x for PREMIS and other provance services. It currently does not exist in Fedora 4.x, so the community has come together to plan our the service over two conference calls that will outline use cases and functional requirements, which will then translate to JIRA tickets for a Fedora code sprint in late March. Notes from the first meeting are here. Nick has been tasked with identifying if the community should use the PROV-O ontology, the PREMIS ontology, or a combination of both. The second item is bridging the work of Mike Durbin’s migration-utils and Danny’s Apache Camel work in the Islandora & Fedora 4 project. While Nick was working to create test fixtures for Mike and Danny, he discovered a bug in Fedora 3.8.0, which will need to be resolved before any test fixtures can come out of York University's upgration pilot.

Nick and Danny will most likely focus on migration work and community contributed developer tasks in March.


The Islandora Foundation is pleased to welcome Simon Fraser University as a Partner for their support of the Fedora 4 project. Longtime member PALS has also earmarked some of their membership dues to help out the upgration. If you or your instition are interested in being financial supporters, please drop me a line.

Other News

Contributor Kevin Bowrin wrote up an account of exprience installing and trying out the work our team has done so far. Check it out.



District Dispatch: Join March 6 free webinar on mapping inclusion: Public library technology and community needs

planet code4lib - Mon, 2015-03-02 20:15

As economic, education, health and other disparities grow, equitable access to and participation in the online environment is essential for success. And yet, communities and individuals find themselves at differing levels of readiness in their ability to access and use the Internet, engage a range of digital technologies and get and create digital content.

The Digital Inclusion Survey examines the efforts of public libraries to address these readiness gaps by providing free access to broadband, public access technologies, digital content, digital literacy training and a range of programming that helps build digitally inclusive communities. A new interactive mapping tool places these library resources in a community context, including unemployment and education rates.

Join researchers and data visualization experts at a free webinar on March 6, 1-2 p.m. EST, to explore the intersections of public access technologies and education, employment, health & wellness, digital literacy, e-government and inclusion. Speakers will share new tools and demonstrate how to locate and interpret national and state-level results from the survey for planning and advocacy purposes, as well as present cases for the interactive mapping tool, with suggestions for creating a digital inclusion snapshot of your public library.

The survey, which is funded by the Institute of Museum and Library Services (IMLS) and conducted by the ALA Office for Research & Statistics and the Information Policy & Access Center (iPAC) at the University of Maryland. The International City/County Management Association and the ALA Office for Information Technology Policy are grant partners.

Learn more about the webinar and speakers from iPAC, Community Attributes, IMLS and OITP here.

The post Join March 6 free webinar on mapping inclusion: Public library technology and community needs appeared first on District Dispatch.

District Dispatch: Rising to the newest (Knight) challenge

planet code4lib - Mon, 2015-03-02 20:02

DC Public Library in Washington, D.C. Photo by Maxine Schnitzer Photography.

It has been said that “libraries are the cornerstone of our democracy” so the newest Knight News Challenge on Elections should be right up our alley. From candidate forums to community conversations, about half of all public libraries report to the Digital Inclusion Survey that they host community engagement events. What is your library doing that you might want to expand or what new innovative idea would you like to seed? Knight is inviting all kinds of ideas: “We see democratic engagement as more than just the act of voting. It should be embedded in every part of civic life…”

So—what’s your best idea for: How might we better inform voters and increase civic participation before, during and after elections?

There are several ways you can participate and learn more:

  1. Check out and comment on the growing number of applications. Which of these could best help address issues you see and hear in your community and your library? On a quick scan, I could definitely see a library or libraries as partners for the Knowledge Swap Market, or a similar project, for instance. Also—how might an application be made stronger and more useful? You don’t have to be an applicant to contribute to the conversation, and comments are accepted through April 13.
  2. BUT—you should definitely consider applying! With more than $3 million available, a wide-open invitation to interpret the question as you see fit, encouragement to partner with others, and the opportunity to get feedback from others to improve your application, there’s a lot to be gained in participating.
  3. Learn more about the whole process at “virtual office hours” open Tuesday, March 3, from 1-2 p.m. Eastern Time and on Tuesday, March 17, from 1-2 p.m. ET. Information about these virtual office hours and in-person events in cities across the county can be accessed here. I attended the event in D.C., and it was a great opportunity to meet people and make connections for possible collaboration.

The challenge is a collaboration between the John S. and James L. Knight Foundation, a leading funder of news and media innovation, and three other foundations: the Democracy Fund, the Rita Allen Foundation and the William & Flora Hewlett Foundation. Winners will receive a share of more than $3 million, which includes up to $250,000 from the Democracy Fund.

This news challenge and the recent NetGain challenge are great opportunities to gain visibility and support for library projects working to address community needs and challenges in innovative ways. These invitations to engage with other community and national stakeholders also resonate with the emerging national policy agenda for libraries and the Aspen Institute report (pdf) on re-envisioning public libraries.

I hope you’ll consider joining the conversation. If so, please leave a note here in comments, so others can look for your proposal.

The post Rising to the newest (Knight) challenge appeared first on District Dispatch.

Islandora: Community Contributor Kudos: Diego Pino

planet code4lib - Mon, 2015-03-02 16:12

It has been a while since we have done a Community Contributor Kudos post, but if anyone is worthy of reviving the feature, it is this week's subject: Diego Pino.

Diego is a freelance developer who specializes in addressing the needs of the scientific community with open source solutions. Right now he is also working as an IT Project Manager for a project that aims to build a national biodiversity network, funded by the Chilean government. If you have gone to the listserv with a question in the past several months, you will also recognize him as one of the most helpful troubleshooters in the Islandora community - pretty remarkable given that he only started using the software about a year ago:

Islandora is still new for me and still amazes me. All started about a year ago. I was given the task to find a way of storing and sharing Biodiversity occurrence records, and thus build a federated network that could help scientists to collaborate and share research data. The primary need was to move data to GBIF for storage, described with Darwin Core metadata, so I started researching what was going on in terms of preserving digital content for science. Until then I thought everything could be solved using a relational database and some custom coding (how wrong I was!)   He started by exploring eSciDoc (created by Matthias Razum), a project based on Fedora 3.x. It was designed to address a need that Diego had been working on for some time: how to involve researchers and scientists directly in the process of sharing and curating their own data. This, and the project's own documentation, sold Diego on Fedora 3.x, but he wanted more - not only the ability to ingest and preserve digital content, but a fully working framework/API that would allow him to focus on the user experience.   And then I found the Islandora's google forum and it was exactly what I needed: A big and nice community of human beings, with problems similar to mine, and with an incredible piece of software, a.k.a. Islandora. I must admit the learning curve was hard; some needed things were not developed and I had to add to my new knowledge Drupal, Solr and Web Semantics (my favourite subject right now), but the community was great and helpful, and meeting Giancarlo Birello was an inspiration to keep working and also to help other users on the forum. I have received so much; giving a little back is a must.

Currently Diego is developing and managing a four repo configuration, with each running a stock Islandora 7.x-1.4/Fedora 3.7.1, using an external Tomcat and other goodies, but sharing a common Solr Cloud index. As Diego describes it, "one collection, many shards, many replicas." He had to fine tune the way objects were indexed to avoid duplicated PIDs and to be able to distinguish during search which repo the object lives in. The repos are also running his Redbiodiversidad Solution Pack , which handles Darwin Core based objects, maps, EML, and GBIF DC archives; and the Ontologies Solution Pack, which allows objects to be related by multiple overlapping ontologies- and which Diego is particularly proud of.

My favourite thing about this configuration is that I can search across all existing repos and their collections, use existing solution packs like PDF or Scholar to describe publications and people, relate local objects to remote ones, and build nice linked data graphs. These expand the notion of plain, independent metadata records encapsulated in objects, to a fully new dimension for us (maybe exaggerating here!) that is helping local scientists to understand their data in a more ample context: in my opinion the needed transition from information to knowledge.

A very simple and trivial example. A Chilean scientist can now discover what other biological occurrences (associated species) are found near a place where they made a discovery; who found them, when, under which method, and filter by many parameters in a few steps or clicks,  thanks to Solr search module + linked data. They can expand their knowledge, collaborate, and  manage their own research data in ways their previous workflows (excel?) did not allow. And my favourite part: if something is not working as expected I can fix it using Islandora's API. There are some many nice hooks available and more to come.

As for projects coming down the pipeline, Diego is working on a new visual workflow to ingest and manage relationships between objects, reusing the way the Ontologies SP currently displays a linked object graph. The end goal is to allow people to interactively add new objects, connect them using rules present in multiple OWLs, and finally save this new "knowledge" representation as a whole. Essentially, every ontology becomes a programable workflow. Using this system will maintain a consistent network of repositories with well-related objects, while still giving users control of their data. He has promised the community an OCR editor, which remains high on his TODO list. As an active member of the Fedora 4 Interest Group, Diego is also involved in planning and developing the next generation of Islandora (and taking a stand for those who don't want to see XML Forms vanish into the night).

Diego does all of this amazing work from his home office in a little village named Pucón in southern Chile, nestled next to an active volcano and a lake. He credits this environment with giving him the peace to code - that, and his small herd of dogs:

Lastly, none of this work using Islandora could have be done without the great support of the community and the also very important support  and patience of my wife and my 4 Dogs, who by this time already hate ontologies.

His Red Biodiversidad repo is still in development, but a beta site is online, showing Solr results from their cloud, fetched from the real repos' collections. And here is one of those collections, full of biological data and growing all the time. You can find more of Diego's work on his GitHub page, and you can usually find him making the Islandora community better one solution at a time on our listerv (it's quite remarkable how many search results for 'diego' in our Google Group turns up some variation of the phrase "thanks, Diego").

Someone in Diego's family is a remarkable photographer, so when I asked him to send along a photo I could use with this blog so the community could put a face to all of those awesome listserv posts, it was difficult to choose. I leave it the community to decide which image best suits Diego Pino: Programmer on a Mountain or Man Hugs Dog:


LibUX: WordPress for Libraries

planet code4lib - Mon, 2015-03-02 14:44

Amanda and Michael are teaching simultaneous online classes on WordPress for Libraries – at least sixty hours worth of tutorial for beginners and developers. Back to back, these classes take you from using WordPress out-of-the-box to create and manage a library website through the custom development of an event management plugin.

Using WordPress to Build Library Websites

WordPress is an open-source content management system that helps you create, design, and maintain a website. Its intuitive interface means that there’s no need to learn complex programming languages — and it’s free, you can do away with purchasing expensive web development software. This course will guide you in applying WordPress tools and functionality to library content. You will learn the nuts and bolts of building a library website that is both user friendly and easy to maintain. Info

Advanced WordPress

WordPress is an incredible out-of-the-box tool, but libraries with ambitious web services will find it needs to be customized to meet their unique needs. This course is built around a single project: the ground-up development of an event management plugin, which will provide a thorough understanding of WordPress as a framework–hooks, actions, methods–that can be used to address pressing and ancillary issues like content silos and the need to C.O.P.E. – create once, publish everywhere. Info


American Library Association eCourses are asynchronous with mixed-media materials available online and at no additional cost. So, you don’t have to get a text book. You can usually proceed at your own pace and submit material through the forums, unless the facilitator changes it up — and we probably won’t, unless it makes sense to keep the class proceeding together. Both of our courses are six weeks, beginning March 16, 2015 – but we want you to squeeze as much as you can out of these classes, so we are available to explain, walkthrough, and answer questions for as long as you need. We really want you to walk away with real-world applicable skills.

The post WordPress for Libraries appeared first on LibUX.

Ed Summers: Repetition

planet code4lib - Mon, 2015-03-02 14:00

To be satisfied with repeating, with traversing the ruts which in other conditions led to good, is the surest way of creating carelessness about present and actual good.

John Dewey in Human Nature and Conduct (p. 67).

Mark E. Phillips: DPLA Metadata Analysis: Part 4 – Normalized Subjects

planet code4lib - Mon, 2015-03-02 13:43

This is yet another post in the series DPLA Metadata Analysis that already has three parts, here are links to part one, two and three.

This post looks at what is the effect of basic normalization of subjects on various metrics mentioned in the previous posts.


One of the things that happens in library land is that subject headings are often constructed by connecting various broader pieces into a single subject string that becomes more specific.  For example the heading “Children–Texas.” is constructed from two different pieces,  “Children”, and “Texas”.  If we had a record that was about children in Oklahoma it could be represented as “Children–Oklahoma.”.

The analysis I did earlier took the subject exactly as it occurred in the dataset and used that for the analysis.  I had a question asked about what would happen if we normalized the subjects before we did the analysis on them,  effectively turning the unique string of “Children–Texas.” into two subject pieces of “Children” and “Texas” and then applied the previous analysis to the new data. The specific normalization includes stripping trailing periods, and then splitting on double hyphens.

Note:  Because this conversion has the ability to introduce quite a bit of duplication into the number of subjects within a record I am making the normalized subjects unique before adding them to the index.  I also apply this same method to the un-normalized subjects.  In doing so I noticed that the item that had the  most subjects previously at 1,476 was reduced to 1,084 because there were a 347 values that were in the subject list more than once.  Because of this the numbers in the resulting tables will be slightly different than those in the first three posts when it comes to average subjects and total subjects,  each of these values should go down.


My predictions before the analysis are that we will see an increase in the number of unique subjects,  a drop in the number of unique subjects per Hub for some Hubs, and an increase in the number of shared subjects across Hubs.


With the normalization of subjects,  there was a change in the number of unique subject headings from 1,871,884 unique headings to 1,162,491 unique headings after normalization,  a reduction in the number of unique subject headings by 38%.

In addition to the reduction of the total number of unique subject headings by 38% as stated above,  the distribution of subjects across the Hubs changed significantly, in one case an increase of 443%.  The table below displays these numbers before and after normalization as well as the percentage change.

# of Hubs with Subject # of Subjects # of Normalized Subjects % Change 1 1,717,512 1,055,561 -39% 2 114,047 60,981 -47% 3 21,126 20,172 -5% 4 8,013 9,483 18% 5 3,905 5,130 31% 6 2,187 3,094 41% 7 1,330 2,024 52% 8 970 1,481 53% 9 689 1,080 57% 10 494 765 55% 11 405 571 41% 12 302 453 50% 13 245 413 69% 14 199 340 71% 15 152 261 72% 16 117 205 75% 17 63 152 141% 18 62 130 110% 19 32 77 141% 20 20 55 175% 21 7 38 443% 22 7 23 229% 23 0 2 N/A

The two subjects that are shared across 23 of the Hubs once normalized are “Education” and “United States”

The high level stats for all 8,012,390 records are available in the following table.

 Records Total Subject Strings Count Total Normalized Subject String Count Average Subjects Per Record Average Normalized Subjects Per Record Percent Change 8,012,390 23,860,080 28,644,188 2.98 3.57 20.05%

You can see the total number of subjects went up 20% after they were normalized, and the number of subjects per record increased from just under three per record to a little over three and a half normalized subjects per record.

Results by Hub

The table below presents data for each hub in the DPLA.  The columns are the number of records, total subjects, total normalized subjects, the average number of subjects per record, the average number of normalized subjects per record, and finally the percent of change that is represented.

Hub Records Total Subject String Count Total Normalized Subject String Count Average Subjects Per Record Average Normalized Subjects Per Record Percent Change ARTstor 56,342 194,883 202,220 3.46 3.59 3.76 Biodiversity Heritage Library 138,288 453,843 452,007 3.28 3.27 -0.40 David Rumsey 48,132 22,976 22,976 0.48 0.48 0 Digital Commonwealth 124,804 295,778 336,935 2.37 2.7 13.91 Digital Library of Georgia 259,640 1,151,351 1,783,884 4.43 6.87 54.94 Harvard Library 10,568 26,641 36,511 2.52 3.45 37.05 HathiTrust 1,915,159 2,608,567 4,154,244 1.36 2.17 59.25 Internet Archive 208,953 363,634 412,640 1.74 1.97 13.48 J. Paul Getty Trust 92,681 32,949 43,590 0.36 0.47 32.30 Kentucky Digital Library 127,755 26,008 27,561 0.2 0.22 5.97 Minnesota Digital Library 40,533 202,456 211,539 4.99 5.22 4.49 Missouri Hub 41,557 97,111 117,933 2.34 2.84 21.44 Mountain West Digital Library 867,538 2,636,219 3,552,268 3.04 4.09 34.75 National Archives and Records Administration 700,952 231,513 231,513 0.33 0.33 0 North Carolina Digital Heritage Center 260,709 866,697 1,207,488 3.32 4.63 39.32 Smithsonian Institution 897,196 5,689,135 5,686,107 6.34 6.34 -0.05 South Carolina Digital Library 76,001 231,267 355,504 3.04 4.68 53.72 The New York Public Library 1,169,576 1,995,817 2,515,252 1.71 2.15 26.03 The Portal to Texas History 477,639 5,255,588 5,410,963 11 11.33 2.96 United States Government Printing Office (GPO) 148,715 456,363 768,830 3.07 5.17 68.47 University of Illinois at Urbana-Champaign 18,103 67,954 85,263 3.75 4.71 25.47 University of Southern California. Libraries 301,325 859,868 905,465 2.85 3 5.30 University of Virginia Library 30,188 93,378 123,405 3.09 4.09 32.16

The number of unique subjects before and after subject normalization is presented in the table below.  The percent of change is also included in the final column.

Hub Unique Subjects Unique Normalized Subjects % Change Unique ARTstor 9,560 9,546 -0.15 Biodiversity Heritage Library 22,004 22,005 0 David Rumsey 123 123 0 Digital Commonwealth 41,704 39,557 -5.15 Digital Library of Georgia 132,160 88,200 -33.26 Harvard Library 9,257 6,210 -32.92 HathiTrust 685,733 272,340 -60.28 Internet Archive 56,911 49,117 -13.70 J. Paul Getty Trust 2,777 2,560 -7.81 Kentucky Digital Library 1,972 1,831 -7.15 Minnesota Digital Library 24,472 24,325 -0.60 Missouri Hub 6,893 6,757 -1.97 Mountain West Digital Library 227,755 172,663 -24.19 National Archives and Records Administration 7,086 7,086 0 North Carolina Digital Heritage Center 99,258 79,353 -20.05 Smithsonian Institution 348,302 346,096 -0.63 South Carolina Digital Library 23,842 17,516 -26.53 The New York Public Library 69,210 36,709 -46.96 The Portal to Texas History 104,566 97,441 -6.81 United States Government Printing Office (GPO) 174,067 48,537 -72.12 University of Illinois at Urbana-Champaign 6,183 5,724 -7.42 University of Southern California. Libraries 65,958 64,021 -2.94 University of Virginia Library 3,736 3,664 -1.93

The number and percentage of subjects and normalized subjects that are unique and also unique to a given hub is presented in the table below.

Hub Subjects Unique to Hub Normalized Subject Unique to Hub % Subjects Unique to Hub % Normalized Subjects Unique to Hub % Change ARTstor 4,941 4,806 52 50 -4 Biodiversity Heritage Library 9,136 6,929 42 31 -26 David Rumsey 30 28 24 23 -4 Digital Commonwealth 31,094 27,712 75 70 -7 Digital Library of Georgia 114,689 67,768 87 77 -11 Harvard Library 7,204 3,238 78 52 -33 HathiTrust 570,292 200,652 83 74 -11 Internet Archive 28,978 23,387 51 48 -6 J. Paul Getty Trust 1,852 1,337 67 52 -22 Kentucky Digital Library 1,337 1,111 68 61 -10 Minnesota Digital Library 17,545 17,145 72 70 -3 Missouri Hub 4,338 3,783 63 56 -11 Mountain West Digital Library 192,501 134,870 85 78 -8 National Archives and Records Administration 3,589 3,399 51 48 -6 North Carolina Digital Heritage Center 84,203 62,406 85 79 -7 Smithsonian Institution 325,878 322,945 94 93 -1 South Carolina Digital Library 18,110 9,767 76 56 -26 The New York Public Library 52,002 18,075 75 49 -35 The Portal to Texas History 87,076 78,153 83 80 -4 United States Government Printing Office (GPO) 105,389 15,702 61 32 -48 University of Illinois at Urbana-Champaign 3,076 2,322 50 41 -18 University of Southern California. Libraries 51,822 48,889 79 76 -4 University of Virginia Library 2,425 1,134 65 31 -52 Conclusion

Overall there was an increase (20%) in the total occurrences of subject strings in the dataset when subject normalization was applied. The total number of unique subjects decreased significantly (38%) after subject normalization.  It is easy to identify Hubs which are heavy users of the LCSH subject headings for their subjects because the percent change in the number of unique subjects before and after normalization is quite high, examples of this include the HathiTrust and the Government Printing Office. For many of the Hubs,  normalization of subjects significantly reduced the number and percentage of subjects that were unique to that hub.

I hope you found this post interesting,  if you want to chat about the topic hit me up on Twitter.

Alf Eaton, Alf: Organising, building and deploying static web sites/applications

planet code4lib - Mon, 2015-03-02 09:34
Build remotely

At the simplest end of the scale is GitHub Pages, which uses Jekyll to build the app on GitHub’s servers:

  • The config files and source code are in the root directory of a gh-pages branch.

  • Jekyll builds the source HTML/MD, CSS/SASS and JS/CS files to a _site directory - this is where the app is served from.

  • For third-party libraries, you can either download production-ready code manually to a lib folder and include them, or install with Bower to a bower_components folder and include them directly from there.

The benefit of this approach is that you can edit the source files through GitHub’s web interface, and the site will update without needing to do any local building or deployment.

Jekyll will build all CSS/SASS files (including those pulled in from bower_components) into a single CSS file. However, it doesn’t yet have something similar for JS/CoffeeScript. If this was available it would be ideal, as then the bower_components folder could be left out of the built app.

Directory structure of a Jekyll GitHub Pages app Build locally, deploy the built app as a separate branch

If the app is being built locally, there are several steps that can be taken to improve the process:

  • Keep the config files in the root folder, but move the app’s source files into an app folder.

  • Use Gulp to build the Bower-managed third-party libraries alongside the app’s own styles and scripts.

  • While keeping the source files in the master branch, use Gulp to deploy the built app in a separate gh-pages branch.

A good example of this is the way that the Yeoman generator for scaffolding a Polymer app structures a project (other Yeoman generators are similar):

  • In the master branch, install/build-related files are in the root folder (run npm install and bower install to fetch third-party components, use bower link for any independent local components).

  • The actual app source files (index.html, app styles, app-specific elements) are in the app folder.

  • gulp builds all the HTML, CSS/SASS and JS source files to the dist folder; gulp serve makes the built files available over HTTP and reloads on changes; gulp deploy pushes the dist folder to a remote gh-pages branch.

Directory structure of a Polymer app Tools

District Dispatch: Reminder: Last chance to apply for Google summer fellowship

planet code4lib - Mon, 2015-03-02 06:43

Google Policy Fellows

The American Library Association’s Washington Office is calling for graduate students, especially those in library and information science-related academic programs, to apply for the 2015 Google Policy Fellows program. Applications are due by March 12, 2015.

For the summer of 2015, the selected fellow will spend 10 weeks in residence at the ALA policy office in Washington, D.C., to learn about national policy and complete a major project. Google provides the $7,500 stipend for the summer, but the work agenda is determined by the ALA and the selected fellow. Throughout the summer, Google’s Washington office will provide an educational program for all of the fellows, such as lunchtime talks and interactions with Google Washington staff.

The fellows work in diverse areas of information policy that may include digital copyright, e-book licenses and access, future of reading, international copyright policy, broadband deployment, telecommunications policy (including e-rate and network neutrality), digital divide, access to information, free expression, digital literacy, online privacy, the future of libraries generally, and many other topics.

Margaret Kavaras, a recent graduate from the George Washington University, served as the 2014 ALA Google Policy Fellow. Kavaras was later appointed as an OITP Research Associate shortly after participating in the Google Fellowship program.

Further information about the program and host organizations is available at the Google Public Policy Fellowship website.

The post Reminder: Last chance to apply for Google summer fellowship appeared first on District Dispatch.

James Cook University, Library Tech: Readings & Past Exams/Reserver Online/Masterfile access issues

planet code4lib - Sun, 2015-03-01 23:33
Not of interest to anyone outside of JCU, just using my blog to list the workarounds for a local issue: <!--[if gte mso 9]> <![endif]--> <!--[if gte mso 9]> Normal 0 false false false EN-AU X-NONE X-NONE <![endif]--><!--[if gte mso 9]>

Open Library Data Additions: An error occurred

planet code4lib - Sun, 2015-03-01 05:43
The RSS feed is currently experiencing technical difficulties. The error is: Search engine returned invalid information or was unresponsive

David Rosenthal: Don't Panic

planet code4lib - Sat, 2015-02-28 17:51
I was one of the crowd of people who reacted to Wednesday's news that Argonne National Labs would shut down the NEWTON Ask A Scientist service, on-line since 1991, this Sunday by alerting Jason Scott's ArchiveTeam. Jason did what I should have done before flashing the bat-signal. He fed the URL into the Internet Archive's Save Page Now, to be told "relax, we're all over it". The site has been captured since 1996 and the most recent capture before the announcement was Feb 7th. Jason arranged for captures Thursday and today.

As you can see by these examples, the Wayback Machine has a pretty good copy of the final state of the service and, as the use of Memento spreads, it will even remain accessible via its original URL.

Hydra Project: Hydra Europe events Spring 2015

planet code4lib - Sat, 2015-02-28 10:02

Registration is now open for two Hydra events in Europe this spring:

Hydra Europe Symposium – a free event for digital collection managers, collection owners and their software developers that will provide insights into how Hydra can serve your needs

  • Thursday 23rd April – Friday 24th April 2015 | LSE, London

Hydra Camp London – a training event enabling technical staff to learn about the Hydra technology stack so they can establish their own implementation

  • Monday 20th April – lunchtime Thursday 23rd April 2015 | LSE, London

Full details and booking arrangements for both events can be found here.

Mark E. Phillips: DPLA Metadata Analysis: Part 3 – Where to go from here.

planet code4lib - Fri, 2015-02-27 23:31

This is the last of three posts about working with the Digital Public Library of America’s (DPLA) metadata to demonstrate some of the analysis that can be done using Solr and a little bit of time and patience. Here are links to the first and second post in the series.

What I wanted to talk about in this post is how can we use this data to help improve access to our digital resources in the DPLA, and also be able to measure that we’ve in fact improved when we go out to spend resources, both time and money on metadata work.

The first thing I think we need to do is to make an assumption to frame this conversation. For now let’s say that the presence of subjects in a metadata record is a positive indicator of quality. And that for the most part a record that has three or more subjects (controlled, keywords, whatever) improves the access to resources in metadata aggregation systems like the DPLA which doesn’t have the benefit of full-text for searching.

So out of the numbers we’ve looked at so far, which ones are the ones to pay the most attention to.

Zero Subjects

For me it is focusing on the number of records that have zero subject headings that are already online. Going from 0-1 subject headings is much more of an improvement for access than going from 1-2, 2-3,3-4,4-8,8-15 subjects per record. So once we have all records with at least one subject we can move on. We can measure this directly with the metric for how many records have zero subjects that I introduced last post.

There are currently 1,827,276 records in the DPLA that have no subjects or keywords. This accounts for 23% of the DPLA dataset analyzed for these blog posts. I think this is a pretty straightforward area to work on related to metadata improvement.

Dead end subjects

One are we could work to improve is when we have subjects that are either only used once in the DPLA as a whole, or only once within a single Hub. Reducing this number would allow for more avenues for navigation between records by connecting them via subject when available. There isn’t anything bad about unique subject headings within a community, but if a record doesn’t have a way to get you to like records (assuming there are like records within a collection) then it isn’t as useful as one that connects you to more, similar items.  There of course are many legitimate reasons that there is only one instance of a subject in a dataset and I don’t think that we should strive to remove them completely,  but reducing the number overall would be an indicator of improvement in my book.

In the last post I had a table that had the number of unique subjects and the number of subjects that were unique to a single Hub.  I was curious about the percentage of subjects from a Hub that were unique to just that Hub based on the number of unique subjects.  Here is that table.

Hub Name Records Unique Subjects # of subjects unique to hub % of subjects that are unique to hub ARTstor 56,342 9,560 4,941 52% Biodiversity Heritage Library 138,288 22,004 9,136 42% David Rumsey 48,132 123 30 24% Digital Commonwealth 124,804 41,704 31,094 75% Digital Library of Georgia 259,640 132,160 114,689 87% Harvard Library 10,568 9,257 7,204 78% HathiTrust 1,915,159 685,733 570,292 83% Internet Archive 208,953 56,911 28,978 51% J. Paul Getty Trust 92,681 2,777 1,852 67% Kentucky Digital Library 127,755 1,972 1,337 68% Minnesota Digital Library 40,533 24,472 17,545 72% Missouri Hub 41,557 6,893 4,338 63% Mountain West Digital Library 867,538 227,755 192,501 85% National Archives and Records Administration 700,952 7,086 3,589 51% North Carolina Digital Heritage Center 260,709 99,258 84,203 85% Smithsonian Institution 897,196 348,302 325,878 94% South Carolina Digital Library 76,001 23,842 18,110 76% The New York Public Library 1,169,576 69,210 52,002 75% The Portal to Texas History 477,639 104,566 87,076 83% United States Government Printing Office (GPO) 148,715 174,067 105,389 61% University of Illinois at Urbana-Champaign 18,103 6,183 3,076 50% University of Southern California. Libraries 301,325 65,958 51,822 79% University of Virginia Library 30,188 3,736 2,425 65%

Here is the breakdown when grouped by type of Hub,  either Service-Hub or Content-Hub

Hub Type Records Unique Subjects Subjects unique to Hub Type % of Subjects unique to Hub Type Content Hubs 5,736,178 1,311,830 1,253,769 96% Service Hubs 2,276,176 618,081 560,049 91%

Or another way to look at how the subjects are shared between the different types of Hubs is the following graph.

Subjects unique to and shared between Hub Types.

It appears that there is a small number (3%) of subjects that are shared between Hub types.  Would increasing this number improve the ability for users to discover resources better from multiple Hubs?

More, More, More

I think once we’ve looked at the ways mentioned above I think that we should work to up the number of subjects per record within a given Hub. I don’t think there is a magic number for everyone, but at UNT we try and have three subjects for each record whenever possible. So that’s what we are shooting for. We can easily see improvement by looking at the mean and see if it goes up (even ever so slightly up)

Next Steps

I think that there is some work that we could do to identify which records need specific kinds work for subjects based on more involved processing of the input records, but I’m going to leave that for another post and probably another flight somewhere to work on.

Hope you enjoyed these three posts and hope they resonate at least a bit with you.

Feel free to send me a not on twitter if you have questions, comments, or idea for me about this.

District Dispatch: Copyright Office should modernize its operation

planet code4lib - Fri, 2015-02-27 21:06

Photo by Steve Akers

The U.S. House Judiciary Committee has mulled its way through 16 well-attended and sometimes contentious hearings on comprehensive copyright reform since 2013. Thursday’s hearing—“the U.S. Copyright Office: Its Function and Resources—sounds like one that keen copyright followers might think is mundane enough to skip, but they would be wrong. The Office’s functions tremendously affect how libraries, businesses, authors, and other creators also operate, because the Office holds the official record of what works are protected and who holds the copyright. Plus there was a little excitement.

The witnesses were Keith Kupferschmid, Software & Information Industry Association general counsel; Lisa Dunner of Dunner Law on behalf of the American Bar Association; Nancy Mertzel of Schoeman Updike on behalf of the American Intellectual Property Law Association; and Bob Brauneis, George Washington University Law School professor.

In an unusual accord, the witnesses agreed, the Representatives agreed, probably everybody in the room agreed that the time is now for the modernization of the Copyright Office. Stuck in the 1970s, the Copyright Office is ill-equipped to manage its basic function—recording what works are protected by copyright and who holds the rights to those works. Based on their testimony, the panel agreed that the Copyright Office requires more technical expertise and additional resources to build a 21st century digital infrastructure for registration, recordation and search. The Judiciary committee member statements demonstrated that in light of the Office’s role in enabling an effective and efficient structure to enable transactions in the copyright industry— an industry that Representative Deutch said was worth over a trillion dollars—something must be done. A functioning, modern registration system would help alleviate the orphan works problem by making it possible to locate rights holders and track the provenance of copyrighted works. (Perhaps now all stakeholders can agree with the Library Copyright Alliance’s position that a full scale searchable and interoperable system that meets the needs of commerce, creators, and the public is preferable to an orphan works solution legislated by Congress).

Only Representative Zoe Lofgren went off script, asking cynically if modernization could also help the Copyright Office represent “a greater diversity of viewpoints?” Rep. Lofgren continued by saying the Office has made some “bone-headed mistakes.” Its strong endorsement of Stop Online Piracy Act (SOPA) did not take into account the public interest, and resulted in an unpreceded backlash. A recorded 14 million people contacted Congress to protest SOPA and the world witnessed the first internet blackout campaign. Lofgren continued that it was the Copyright Office that recommended not extending the 1201 exemption for cell phone unlocking, leading to another public outcry, a “We The People” petition to the President, and the need for Congress to pass the “Unlocking Consumer Choice and Wireless Competition Act.” Lofgren was not done. At an earlier 1201 rulemaking the Office advised against an exemption for circumvention of e-readers to enable to enable text to speech functionality for people with print disabilities. Really? Thankfully, this decision was overturned by the Librarian of Congress in his final decision.

Lofgren, who represents constituents in Silicon Valley, has butted heads with the Copyright Office in the past. At an earlier hearing, Register Maria Pallante was quizzed by Lofgren over the purpose of the copyright. In the American Bar Association Landslide Magazine, Pallante was quoted as saying that “copyright is for the author first and the nation second.” Lofgren countered “it seems to me when you look at the Constitution, which empowers Congress to grant exclusive rights in creative works in order, and I quote, “to promote the progress of science and the useful arts.” It seems to me that the Constitution is very clear that copyright does not exist inherently for the author but for the benefit for society at large.”

It is in the public interest to provide the necessary resources and expertise to upgrade the Copyright Office’s infrastructure. Whether the Copyright Office’s will be able to balance the interests of all parties for the benefit for society at large in other areas of copyright review is another more fundamental matter. Protection of legacy business models and copyright enforcement continue to dominate policy discussions in both the legislative and executive branches of government, so the Copyright Office is not alone in its perspective that the public interest is a secondary matter. Let’s hope that the public continues to pay close attention to the House Judiciary copyright review. They have been more than willing to speak up on their behalf.

The post Copyright Office should modernize its operation appeared first on District Dispatch.

Chris Prom: Rights in the Digital Era

planet code4lib - Fri, 2015-02-27 20:31

When working with electronic records, we often feel like we stand on shaky ground.

For example, when I teach my DAS course, some of the most difficult questions that course participants raise are related to rights: “How do we know what is copyrighted?” “How do we identify private materials?” “How can we provide access to ________”  “How can we track rights information?” “What do we do if we get sued?”

As archivists, we like to provide as much access as possible to the great things our repositories hold.  How can we do that and still sleep soundly at night?

These are the kinds of questions that led the SAA Publications Board to commission the newest entry in our Trends in Archives Practice Series, Rights in the Digital Era.    As with the courses in the SAA Digital Archives Specialist curriculum, I see the current and forthcoming works in this series as falling very much within the original spirit of this blog, making digital archives work practical and accessible.

Rights in the Digital Era, for instance, lays out risk management strategies and techniques you can use to provide responsible access to analog and digital collections, while meeting legal and ethical obligations, whatever your professional status or repository profile  As Peter B. Hirtle notes in his introduction to the volume, “A close reading of the modules will provide you with the rights information that all archivists should know, whether you are a repository’s director, reference archivist, or processing assistant.”

  • Module 4: Understanding Copyright Law by Heather Briston – provides a short-but-sweet introduction to copyright law and unpublished materials, emphasizing practical steps that can be taken to make archives more accessible and useful.
  • Module 5: Balancing Privacy and Access in Manuscript Collections by Menzi Behrnd-Klodt – navigates the difficult terrain of personal privacy, developing a roadmap to the law and describing risk management strategies applicable for a range or repositories and collections.
  • Module 6: Balancing Privacy and Access in the Records of Organizations by Menzi Behrnd-Klodt – unpacks legal and ethical requirements for records that are restricted under law, highlighting hands-on techniques you can use to develop thoughtful access policies and pathways in a variety of repository settings.
  • Module 7: Managing Rights and Permissions by Aprille McKay – examines provides methods and steps you can take to control rights information, supplying many useful approaches that you can easily adapt to your own circumstances.

It gives me and the entire SAA Publications Board great pride to see the fruits of our authors’ thinking and of our society’s work emerge in high quality, peer reviewed, and attractively designed books.

But don’t just take my word for it—read Rights in the Digital Era or any of the dozens of other publications that comprise the SAA catalog of titles!




Subscribe to code4lib aggregator