The BLE IMS uses modular components and relies on external APIs and open source resources to minimize development burden and promote reusability. We welcome discussion and feedback on our system, and we love collaborating with other information managers on improving the state of IM in LTER. See what we're up to on GitHub, and take a peek into our information management system below.
We keep track of datasets submitted by the BLE team using a Postgres database based on the LTER Core Metabase schema. This database is designed to be simple and extensible while supporting the creation of EML metadata for datasets described therein. To interact with the database, we like the free DBeaver software.
When a scientist on our team submits data, they include a spreadsheet modeled after tables in our Core Metabase to provide metadata. The spreadsheet includes validation and other forms of guidance to help the scientist supply rich metadata.
Our IM team evaluates the data and metadata and iterates with the scientist as needed. When the metadata are satisfactory, the IM appends records from the spreadsheet to the Core Metabase. Because the spreadsheet follows the Core Metabase design, no custom tooling is needed for the import.
We use an open source set of R scripts created by Li Kui (IM for SBC LTER) to generate EML from our Core Metabase.
Rather than maintain our own online database with search functionality and metadata display, we use an API to tap into the Arctic Data Center catalog where we archive our data. Naturally, this keeps our website in sync with our official archive. However, this may not work as well for you if metadata for your datasets is maintained at disparate archives.
Check out these links for example clients including live demos:
While there are some interesting service-based search services out there, we went with Lunr for static search on our website. Our Lunr GitHub Project demonstrates how to build a search index for your website and provide search functionality. It seems to work well for small sites, but performance may suffer if you index more than 1000 pages. We like that the search interface is encapsulated within our site without any external branding.
By relying on external APIs and sticking with static HTML, we can implement our website without having to maintain any servers. This means we don't incur the operating cost of a dynamic server (unlike WordPress or Drupal sites for example), and we aren't plagued by dynamic CMS security flaws.
There are lots of free hosts for static websites. We like Netlify the best. They minify our assets to improve page load times, provide HTTPS, and host our site on a content delivery network with servers distributed around the world to maximize uptime and speed, among other benefits. As soon as we make a new commit to our site on GitHub, Netlify picks it up and distributes it across their network. They also support forms, functions, and secure logins, though we haven't played with those features yet.
Eventually we may adopt a static site generator to assist with page creation and website redesign. For now we use a simple script to update headers, footers, and navigation across the entire site when needed.
Our IMS would be nothing without the skills, dedication, and enthusiasm of our team, both on the IM side and the domain science side. This system works because our scientists are willing and able to incorporate information management into their workflows up front (e.g., by using simple data tables), and because there is open, friendly, and regular dialog between scientists and information managers within BLE.
Our IM team enjoys extending that dialog into the broader LTER community. If you are facing an IM challenge involving GIS, Python, or any part of the stack described above, feel free to contact us.