Skip to content
This repository has been archived by the owner on Mar 18, 2020. It is now read-only.

Ingest Using Excel Input Stream

Jump to bottom
hweng edited this page Aug 12, 2014 · 7 revisions

  1. Excel Input Stream Source Validation

  2. Potential Pre-Processing

    • For files already ingested previously, either re-ingest or ingested in DAMS3, the arks need to be provided in a column with column title "ARK", otherwise, new arks will be assigned for each object.
    • For the 'Collection' column, it is preferable to use ARK in the collection column. DOMM will enter (or replace the title name of the collection with) the collection's ARK. This ARK should have been created/minted with the collection level record. But it need to enter the collection's Title for now until the above feature is implemented.

  3. Ingest using DAMS Manager

    • Create a new folder (or use any existed empty folder) in the DAMS Staging area, then copy the Excel Input Stream source file to there. For example, \darry.ucsd.edu\dams_staging\dmca\source.
    • Go to the External Import Tool in DAMS Manager's Control Panel: /damsmanager/controlPanel.do
    • Click on the "Import" tab, check the checkbox next to label "External Import":
    • Select "Metadata only" for "Import Options" if you don't want to ingest the files at this time.
    • Choose the folder that contains the Excel Input Stream source file (s) by clicking at the file selector button with label "..." that next to the input with label "Metadata location". This will pop up the file chooser. Form the file chooser, track down to the folder that contains the source file(s) by clicking on the directory tree with labels, click the "OK" button at the bottom of the form to finish the folder selection.
    • Validate the path in the input next to label "Metadata location".
    • Click button "Perform Operation" to submit the request to ingest it. [Optionally] To ingest the master files at the same time, select option "Metadata and files" instead of"Metadata only" for "Import Options", then choose the "Files location" from DAMS Staging , and make sure that the fileStore "openStack" is selected for RCI collections by clicking on the "Preference" tab (or got to /damsmanager/controlPanel.do?activeButton=preferenceButton), otherwise the "localStore" should be choosen.
    • The progress report page will be shown up and a mail with title "DAMS Manager Invocation Result" will be sent out once the ingest completed.

  4. Potential Post Processing

    • Identify issues that may need Post Processing:
      • For re-ingesting or any DAMS3 files: add derivative File element(s) along with each master file.
      • Each master file need to have a file use property, otherwise the default files use property will be provided as follows:
        .avi => video-source
        .mov => video-source
        .mp4 => video-service
        .tif => image-source
        .jpg => image-service
        .png => image-service
        .pdf => document-service
        .wav => audio-source
        .mp3 => audio-service
        .tar => data-service
        .tgz => data-service
        .tar.gz => data-service
        .zip => data-service
    • Export RDF/XML from DAMS Managers /damsmanager/controlPanel.do?activeButton=cdlButton, check the checkbos next to label "Metadata Export:", then hit button "Perform Operation" on the bottom to submit the request.
    • Edit the RDF/XML to fix any issues identified above.
    • Upload the RDF/XML metadata: From the Import tab in DAMS Manager's control panel (or /damsmanager/controlPanel.do?activeButton=sipButton), check check box "Metadata Import" and select the option "Replace the whole record with metadata submitted", then hit button "Perform Operation" on the bottom to submit the request.

  5. Objects validation and master files ingesting

    • [Optional] Choose the collection that was just ingested, click on the "Object" tab and check the check box by label "File Count Validation", then hit the "Perform Operation" to check for files that are existed in DAMS4 unexpectedly. You should see the message showing that all the files are missing if the files haven't being ingested yet. Please click on the Preference tab to check and make sure that the right fileStore is selected.
    • For DAMS3 files, choose the collection that was ingested and click on the "Object" tab or go to /damsmanager/controlPanel.do?activeButton=validateButton if you haven't done it. The number of objects showing up by the label "File Count Validation" should be matched the number in the collection. Checked the checkbox with label "File Count Validation", select option "Rename dams3 files to DAMS4 naming conventions.", and then hit button "Perform Operation" on the bottom to rename the dams3 files to DAMS4.
    • For Re-ingest or for ingest with files in DAMS3, choose the collection that was ingested and click on the "Object" tab (or go to /damsmanager/controlPanel.do?activeButton=validateButton) if you haven't done it yet. Then checked the check box with label "Jhove Report", and select option with "Update technical metadata:" and hit the "Perform Operation" button to extract the technical metadata for the files. Please click on the Preference tab to check and make sure that the right fileStore is selected.
    • To ingest the master files, click on the "Object" tab or go to /damsmanager/controlPanel.do?activeButton=validateButton if not in the Object tab, then check the check box with label "File Count Validation", and check option by label "Ingest missing files from staging", then choose the master files location by clicking at the button with text "..." next to the input box, and hit the "Perform Operation" to submit the request. Please click on the Preference tab to check and make sure that the right fileStore is selected.

  6. Re-ingest

    Repeat steps 1, 2, 3, 4 and 5 above:

    • In step 2, make sure that the ARKs are provided in a column with title "ARK", otherwise objects will be new ingest with new ARKs assigned.
    • In step 3, check the checkbox with label "Replace all records submitted that have ark url assigned" under "External Import".

  7. Examples

    Department of Music Concert Archive (DMCA)] is the most recent collection that was ingested with the Standard Excel Input Stream, in which the master .wav files and the .mp3 derivatives are ingested into DAMS3 in the past that need to go through all the processes above. Please visit Jira https://lib-jira.ucsd.edu:8443/browse/DI-15 for raw Excel Input Stream and the example of valid Excel source that were pre-processed for ingest.

Clone this wiki locally