CIDOC vs Solr — Steamrolling your model

Leonardo da Vinci’s Small nude figures swinging axes; notes and diagrams on the theory of light and shade. More.
  • 300K triples
  • Data model is CIDOC
  • Four separate graphs representing the three catalog editions and a “project data” graph
  • Data in the triple store is expected to change or enriched over time
  • External URIs are used (ULAN, VIAF, AAT) and remote data would like to be leveraged in the portal
  • Query the remote SPARQL endpoint for each resource to acquire all the data needed for each record.
  • Download each graph, create a local triple store in memory and query that with SPARQL.
  • Use native ruby datatypes like hashes and arrays to build lookup tables need for building each record.
Partial data model diagram by Alexandra Provo
  1. Bulk download each of the three edition graphs from the SPARQL endpoint.
  2. Build lookup hashes with native ruby for all the multiple entities so you can assign properties to the resources, not entities attached to the resource.
  3. Resolve any external URIs like VIAF and cache that data locally in the file system, expire the cache if it is over 30 days old.
  4. Download the project data graph (contains image links, museum links, etc).
  5. Build a Solr document using all the in memory data cherry picking the best data (1968 title vs 1903 missing title for example) and also build the document so it contains repeated data elements for each edition in the Solr document.
{
"id": "0001192-Berenson",
"language_facet": [
"By",
"By",
"By"
],
"edition_facet": [
"Berenson 1903",
"Berenson 1938",
"Berenson 1961"
],
"owner_facet": [
"Royal Library (Windsor)"
],
"technique_facet": [
"ink"
],
"technique_recto_t": [
"ink"
],
"technique_recto_uri": [
"http://vocab.getty.edu/aat/300015012"
],
"technique_verso_t": [
"ink"
],
"technique_verso_uri": [
"http://vocab.getty.edu/aat/300015012"
],
"has_image_i": 1,
"museum_url_s": "https://www.royalcollection.org.uk/collection/search#/1/collection/919149/recto-notes-on-optics-etc-with-diagrams-verso-notes-on-optics-etc-with-diagrams",
"inventory_number_s": "19149",
"inventory_number": "19149",
"owners_label_t": [
"Royal Library"
],
"owners_uri_t": [
"http://viaf.org/viaf/153159158/"
],
"owners_geo_label_t": [
"Windsor"
],
"owners_geo_uri_t": [
"http://sws.geonames.org/2633842"
],
"owners_label_display": "",
"contributors_t": [
"Leonardo da Vinci"
],
"contributors_alt_t": [
"Leonardo",
"Vinci, Leonardo da",
"Leonardo, da Vinci",
"da Vinci, Leonardo",
"Léonard de Vinci",
"Léonardo de Vinci",
"Leonardo di Ser Piero da Vinci",
"Leonardo Da Vinci",
"Da Vinci, Leonardo",
"da Vinci Leonardo",
"Lionardo",
"Liyūnārdū Dāvīnshī",
"Vinchi, Leonardo da",
"Leonardo da Vinchi",
"Léonard",
"Lieh-ao-na-to",
"Леонардо да Винчи",
"Леонардо",
"לאונרדו",
"ליאונארדו",
"ליאונרדו דא וינצ׳י",
"ליאורנרדו",
"李奥纳多·达·文西",
"レオナルド・ダ・ヴィンチ",
"Leonardo de Vinza",
"Leonardo D'Vinci",
"Leonardo da Vince",
"Leonardo d'Avinci",
"Leonard Davincy",
"Leonardo De Vinci"
],
"contributors_ulan_t": [
"http://vocab.getty.edu/ulan/500010879"
],
"contrubtor_preflabel": "",
"title_sort": "Appunti di ottica / Piccole figure nude che manovrano scuri; appunti e diagrammi sulla teoria della luce e delle ombre.",
"bcn_t": [
"1192",
"1192",
"1192"
],
"bnc_1903_s": "1192",
"verso_title_1903_t": "--",
"recto_title_1903_t": "Small figures swinging axes. Some manuscript notes.",
"verso_note_1903_t": "--",
"recto_note_1903_t": "--",
"verso_figures_1903_t": [
],
"recto_figures_1903_t": [
],
"page_number_1903_s": "",
"image_plate_display": null,
"image_bm_display": null,
"image_plate_number_s": null,
"image_plate_number_roman_s": null,
"image_thumb_display": null,
"image_page_display": null,
"museum_image_url_display": [
"https://d9y2r2msyxru0.cloudfront.net/sites/default/files/collection-online/0/2/282067-1464772643.jpg"
],
"museum_image_text_display": [
"Recto"
],
"author_1903_display_s": "Leonardo da Vinci (By)",
"bnc_1938_s": "1192",
"verso_title_1938_t": "Small nude figures swinging axes; notes and diagrams on the theory of light and shade.",
"recto_title_1938_t": "--",
"verso_note_1938_t": "--",
"recto_note_1938_t": "Vertically to them in red chalk a nude seated figure bending down, not by Leonardo as Clark correctly affirms.",
"verso_figures_1938_t": [
],
"recto_figures_1938_t": [
],
"page_number_1938_s": "",
"author_1938_display_s": "Leonardo da Vinci (By)",
"bnc_1961_s": "1192",
"verso_title_1961_t": "Piccole figure nude che manovrano scuri; appunti e diagrammi sulla teoria della luce e delle ombre.",
"recto_title_1961_t": "Appunti di ottica",
"verso_note_1961_t": "--",
"recto_note_1961_t": "--",
"verso_figures_1961_t": [
],
"recto_figures_1961_t": [
],
"page_number_1961_s": "",
"author_1961_display_s": "Leonardo da Vinci (By)",
"title_t": "Appunti di ottica / Piccole figure nude che manovrano scuri; appunti e diagrammi sulla teoria della luce e delle ombre.",
"title_display": "Appunti di ottica / Piccole figure nude che manovrano scuri; appunti e diagrammi sulla teoria della luce e delle ombre.",
"subtitle_t": "-- / Small nude figures swinging axes; notes and diagrams on the theory of light and shade.",
"subtitle_display": "-- / Small nude figures swinging axes; notes and diagrams on the theory of light and shade.",
"subject_topic_facet": [
"Leonardo da Vinci"
],
"author_display": "Leonardo da Vinci (By)",
"author_sort": "Leonardo da Vinci (By)",
"author_t": [
"Leonardo da Vinci"
],
"authorsuggest": "Leonardo da Vinci (By)",
"thumbnail_url_s": "https://s3-eu-west-1.amazonaws.com/florentinedrawings/thumbs/0001192-Berenson.jpg"
}

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

What it’s like to work as a software engineer at ClassPass

Dockerize Flask Application-AWS ECS+Flask

Better Apps with Domain-Driven Design

Snowflake: Zero Copy Clone & Demo

Unlock The Factors Affecting React Native App Development Cost in 2021

Deploying Multi-Tenant Presto Server to Amazon ECS

Overseas Software Development — The Right Step to Maximize Productivity

Richardson Maturity Model for REST APIs

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Matt Miller

Matt Miller

More from Medium

Enhancing Google Maps: Zomato

The Maps API Handcuffs — and everyone is a victim

The Impact of Artificial Intelligence in Medicine & Healthcare Systems

The Best-First search method and the A* search method