[MRG]: Add rdf serialization #49

satra · 2014-06-23T12:03:07Z

This is ready for merge!

… bundle support

* upstream/master: Fixed Build status link Added build status from Travis-CI doc: fixed docstring for assertLess fix: remove redundant ls fix: remove checking for py3k fix: added assertLess fix: added support for asserts to unittest fix: more set fixes fix: support for 2.6 set fix: added python versions to tests fix: updated dateutil name enh: add travis testing file

trungdong · 2014-07-14T10:40:56Z

Hi @satra,

FYI, prov now has many more tests (https://github.com/trungdong/prov/tree/master/prov/tests), which you can use to test the RDF export. Since we don't have RDF import, you won't be able to do the round-trip tests as in the test cases, but even a one-way export test could be useful.

* upstream/master: Fixed: Cloning the records when creating a new document from them Bugfix regarding a software agent record.

satra · 2014-10-11T16:13:58Z

@trungdong - i'm slowly making my way through rdf deserialization. in terms of comparing documents, how do you ensure that order of attributes don't matter?

ACTUAL: u'document
  prefix ex <http://example.org/>

  activity(ex:a2, -, -, [prov:label="bonjour"@fr, prov:label="hello", prov:label="activity2", prov:label="bye"@en])
endDocument'
 DESIRED: u'document
  prefix ex <http://example.org/>

  activity(ex:a2, -, -, [prov:label="activity2", prov:label="hello", prov:label="bye"@en, prov:label="bonjour"@fr])
endDocument'

to me these are the same graphs, but the roundtrip fails because the orders of attributes are different.

trungdong · 2014-10-11T16:34:13Z

Hi @satra,
Glad you have some time to get on with this. Thanks.

I suggest you do prov.model --> RDF --> prov.model. Comparing two ProvDocument instances is not sensitive to ordering (of attributes or records) as it uses set instead of list.

satra · 2014-10-11T17:01:10Z

thanks @trungdong - i was doing assert_equal(g.get_provn(), g1.get_provn()), but i changed to assert_equal(g, g1)

satra · 2014-10-11T19:37:14Z

even sets are not quite doing their job - will have to look into this further.

document
  prefix ex <http://example.org/>

  activity(ex:a2, -, -, [prov:type="a", prov:type=1, prov:type=2014-06-23T12:28:53.843000+01:00,
 prov:type="ex:abc" %% xsd:QName, prov:type="http://example.org/hello" %% xsd:anyURI,
 prov:type="1.0" %% xsd:float, prov:type="true", prov:label="activity2"])

endDocument

vs

document
  prefix ex <http://example.org/>

  activity(ex:a2, -, -, [prov:type="a", prov:type=1, prov:type=2014-06-23T12:28:53.843000+01:00, 
prov:type="http://example.org/hello" %% xsd:anyURI, prov:type="1.0" %% xsd:float, prov:type="ex:abc"
 %% xsd:QName, prov:type="true", prov:label="activity2"])

endDocument

sets

[(<QualifiedName: prov:type>, u'a'), (<QualifiedName: prov:type>, 1), (<QualifiedName: prov:type>, 
datetime.datetime(2014, 6, 23, 12, 28, 53, 843000, tzinfo=tzoffset(None, 3600))), (<QualifiedName:
 prov:type>, <Identifier: http://example.org/hello>), (<QualifiedName: prov:type>, <Literal: "1.0" %%
 xsd:float>), (<QualifiedName: prov:type>, <Literal: "ex:abc" %% xsd:QName>), (<QualifiedName:
 prov:type>, u'true'), (<QualifiedName: prov:label>, u'activity2')]

vs

[(<QualifiedName: prov:type>, u'a'), (<QualifiedName: prov:type>, 1), (<QualifiedName: prov:type>,
 datetime.datetime(2014, 6, 23, 12, 28, 53, 843000, tzinfo=tzoffset(None, 3600))), (<QualifiedName:
 prov:type>, <XSDQName: ex:abc>), (<QualifiedName: prov:type>, <Identifier: http://example.org/hello>),
 (<QualifiedName: prov:type>, <Literal: "1.0" %% xsd:float>), (<QualifiedName: prov:type>, u'true'),
 (<QualifiedName: prov:label>, u'activity2')]

satra · 2014-10-11T19:37:56Z

forgot to say that the graphs in the previous comment are failing the assert.

satra · 2014-10-11T19:59:39Z

nevermind - found it - it's the QName

* upstream/master: Fixed: PROV-N representation for xsd:dateTime (closed trungdong#58) Fixed: Unintended merging of Identifier and QualifiedName values

* upstream/master: fix: formal attributes were not being included in all attributes

* upstream/master: Fixed trungdong#60 but no need to touch ProvRecord.formal_attributes (as per trungdong#61)

cmaumet · 2014-12-04T12:22:46Z

Hi @satra. You will find below an example in which the serialization to rdf adds extra (unwanted) qualified relations:

from prov.model import ProvDocument
from exporter.objects.constants import *

if __name__ == '__main__':
        doc = ProvDocument()

        activity_id = NIIRI["activity"]
        doc.activity(activity_id)

        entity_1 = NIIRI["entity_1"]
        doc.entity(entity_1)

        entity_2 = NIIRI["entity_2"]
        doc.entity(entity_2)

        doc.used(activity_id, entity_1)
        doc.wasGeneratedBy(entity_1, activity_id)
        doc.wasDerivedFrom(entity_1, entity_1)

        ttl_file = "example.ttl"
        ttl_fid = open(ttl_file, 'w');
        ttl_fid.write(doc.serialize(format='rdf'))

Obtained turtle export:

@prefix niiri:  .
@prefix prov:  .
@prefix rdf:  .
@prefix rdfs:  .
@prefix xml:  .
@prefix xsd:  .
niiri:entity_2 a prov:Entity .
niiri:activity a prov:Activity ;
    prov:qualifiedUsage [ a prov:Usage ;
            prov:entity niiri:entity_1 ] ;
    prov:used niiri:entity_1 .
niiri:entity_1 a prov:Entity ;
    prov:qualifiedDerivation [ a prov:Derivation ;
            prov:usedEntity niiri:entity_1 ] ;
    prov:qualifiedGeneration [ a prov:Generation ;
            prov:activity niiri:activity ] ;
    prov:wasDerivedFrom niiri:entity_1 ;
    prov:wasGeneratedBy niiri:activity .

Unfortunately, I did not find the fix...

I hope this example is useful. Let me know if I can help you to track this down!

satra · 2014-12-04T13:33:19Z

@cmaumet - should entity_1 be derived from entity_1?

satra · 2014-12-04T13:35:44Z

also the qualified relations aren't unwanted - that's how the representation for derivation is intended to be.

a wasDerivedFrom is a relationship, i.e. an edge between two nodes. the qualified derivation allows describing properties of that edge.

this is partly what makes the deserialization difficult.

cmaumet · 2014-12-04T14:46:45Z

thank you

… them properly)

satra · 2016-03-19T17:34:07Z

a few more things to finalize:

support bundles (via trig)
make the tests run on travis
check py3 support
settle an issue with Decimal representation (see Literal comparison for Decimal values #77)
skip scruffy round trip tests for now - just ensure they can be read without error for the moment.

…n scruffy tests to still resolve

satra · 2016-03-20T22:29:13Z

these tests fail round trip - need to figure out a way to skip.

FAIL: test_scruffy_end_2 (prov.tests.test_rdf.RoundTripRDFTests)
FAIL: test_scruffy_end_3 (prov.tests.test_rdf.RoundTripRDFTests)
FAIL: test_scruffy_end_4 (prov.tests.test_rdf.RoundTripRDFTests)
FAIL: test_scruffy_generation_2 (prov.tests.test_rdf.RoundTripRDFTests)
FAIL: test_scruffy_invalidation_2 (prov.tests.test_rdf.RoundTripRDFTests)
FAIL: test_scruffy_start_2 (prov.tests.test_rdf.RoundTripRDFTests)
FAIL: test_scruffy_start_3 (prov.tests.test_rdf.RoundTripRDFTests)
FAIL: test_scruffy_start_4 (prov.tests.test_rdf.RoundTripRDFTests)
FAIL: test_scruffy_usage_2 (prov.tests.test_rdf.RoundTripRDFTests)

satra · 2016-03-20T22:59:38Z

@trungdong - some of the issues here are unfortunately due to rdflib interactions. but this is good for review. i still need to figure out the python 3 errors, again an interaction with rdflib! and how to suppress the failing scruff tests.

* fix/literal: fix: extra curly bracket fix: test setup fix: only escape triple quotes in the triple quote case fix: string representation containing double quotes or triple quotes - closes trungdong#79

* upstream/master:

coveralls · 2016-06-27T07:45:44Z

Coverage decreased (-1.3%) to 89.442% when pulling 1debd4b on satra:enh/rdf-1.x into a556fa7 on trungdong:master.

* upstream/master: Remove networkx versioning also in setup.py Relaxed networkx requirement. Closed trungdong#84. Fix deprecated usage of cgi.escape since Python 3.3

coveralls · 2016-10-08T17:04:07Z

Coverage decreased (-1.5%) to 89.207% when pulling 1f8955e on satra:enh/rdf-1.x into 5ddeade on trungdong:master.

coveralls · 2016-10-08T17:07:49Z

Coverage decreased (-0.4%) to 90.311% when pulling 5a29a1b on satra:enh/rdf-1.x into 5ddeade on trungdong:master.

coveralls · 2016-10-08T17:07:49Z

Coverage decreased (-0.4%) to 90.311% when pulling 5a29a1b on satra:enh/rdf-1.x into 5ddeade on trungdong:master.

satra · 2016-10-08T17:10:03Z

@trungdong - finally got some time to fix and this is ready for merge :)

satra · 2016-10-08T23:06:58Z

closes #1

trungdong · 2016-10-09T21:52:53Z

Excellent! Thank you very much @satra!!!
I'll try to get some time this week to clean up the current master branch and will make a new release with RDF support soon.

new feature from: trungdong/prov#49

satra added 12 commits May 17, 2014 21:00

fix: rename contants -> constants

e5ffce8

enh: first pass at serialization

d1d1407

remove debug print

5799160

starting deserialization

ac37255

enh: updated to reflect latest upstream changes for attributes, added…

f2c263d

… bundle support

compat: remove bundle statement to be compatible with Luc's output

95bde98

fix: ensure that URIRefs remain as URIRefs

c3198a3

fix: resolved conflict

73b9e57

fix: updated QName -> QualifiedName

1e2a8c6

fix: remove diff statement

08d25d3

fix: add rdflib to travis

481fe13

satra added 3 commits September 4, 2014 12:47

resolve conflicts

0ac1617

updated requirements

32fac75

Merge remote-tracking branch 'upstream/master' into enh/rdf-1.x

7f2cd65

* upstream/master: Fixed: Cloning the records when creating a new document from them Bugfix regarding a software agent record.

satra added 5 commits October 14, 2014 08:51

updated rdf serialization

b201b3d

Merge remote-tracking branch 'upstream/master' into enh/rdf-1.x

54cfc77

* upstream/master: Fixed: PROV-N representation for xsd:dateTime (closed trungdong#58) Fixed: Unintended merging of Identifier and QualifiedName values

Merge remote-tracking branch 'upstream/master' into enh/rdf-1.x

089ae65

* upstream/master: fix: formal attributes were not being included in all attributes

Merge remote-tracking branch 'upstream/master' into enh/rdf-1.x

68c129e

* upstream/master: Fixed trungdong#60 but no need to touch ProvRecord.formal_attributes (as per trungdong#61)

current state

4909e4f

satra added 5 commits April 6, 2015 22:19

broken attempt

88cb118

fix: resolved conflicts with upstream

2202ad8

temporary state

9b73c8d

next temporary state

1471517

passing all tests except bundle and invalid scruffy variations (reads…

3c7a8c9

… them properly)

satra added 3 commits March 19, 2016 22:37

fixed bundle support and removed loggers

d4f1eaf

fixing built-in roundrip tests - first pass

01fc278

fixed most remaining cases - float, literal/decimal, and round trip o…

89aa39d

…n scruffy tests to still resolve

fix: switch to text_type from six

ecdce75

satra added 3 commits April 19, 2016 20:41

Merge branch 'fix/literal' into enh/rdf-1.x

bc822d4

* fix/literal: fix: extra curly bracket fix: test setup fix: only escape triple quotes in the triple quote case fix: string representation containing double quotes or triple quotes - closes trungdong#79

Merge remote-tracking branch 'upstream/master' into enh/rdf-1.x

56e7815

* upstream/master:

fix: skipping known roundtrip and literal representation failures

1debd4b

satra added 2 commits October 8, 2016 11:59

Merge remote-tracking branch 'upstream/master' into enh/rdf-1.x

949ed48

* upstream/master: Remove networkx versioning also in setup.py Relaxed networkx requirement. Closed trungdong#84. Fix deprecated usage of cgi.escape since Python 3.3

fixed tests and skipping round trip tests that don't pass

1f8955e

fix: upgrade setuptools before installing htmllib

5a29a1b

satra changed the title ~~WIP: Add rdf serialization~~ [MRG]: Add rdf serialization Oct 8, 2016

trungdong merged commit 4d2c236 into trungdong:master Oct 9, 2016

cmaumet mentioned this pull request Oct 12, 2016

fix: RDF serialisation, prov:atTime attribute #88

Closed

cmaumet pushed a commit to cmaumet/nidmresults that referenced this pull request Oct 13, 2016

Export as RDF using prov library

0b78abf

new feature from: trungdong/prov#49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MRG]: Add rdf serialization #49

[MRG]: Add rdf serialization #49

satra commented Jun 23, 2014 •

edited

Loading

trungdong commented Jul 14, 2014

satra commented Oct 11, 2014

trungdong commented Oct 11, 2014

satra commented Oct 11, 2014

satra commented Oct 11, 2014

satra commented Oct 11, 2014

satra commented Oct 11, 2014

cmaumet commented Dec 4, 2014

satra commented Dec 4, 2014

satra commented Dec 4, 2014

cmaumet commented Dec 4, 2014

satra commented Mar 19, 2016 •

edited

Loading

satra commented Mar 20, 2016

satra commented Mar 20, 2016

coveralls commented Jun 27, 2016 •

edited

Loading

coveralls commented Oct 8, 2016 •

edited

Loading

coveralls commented Oct 8, 2016

coveralls commented Oct 8, 2016 •

edited

Loading

satra commented Oct 8, 2016

satra commented Oct 8, 2016

trungdong commented Oct 9, 2016

[MRG]: Add rdf serialization #49

[MRG]: Add rdf serialization #49

Conversation

satra commented Jun 23, 2014 • edited Loading

trungdong commented Jul 14, 2014

satra commented Oct 11, 2014

trungdong commented Oct 11, 2014

satra commented Oct 11, 2014

satra commented Oct 11, 2014

sets

satra commented Oct 11, 2014

satra commented Oct 11, 2014

cmaumet commented Dec 4, 2014

satra commented Dec 4, 2014

satra commented Dec 4, 2014

cmaumet commented Dec 4, 2014

satra commented Mar 19, 2016 • edited Loading

satra commented Mar 20, 2016

satra commented Mar 20, 2016

coveralls commented Jun 27, 2016 • edited Loading

coveralls commented Oct 8, 2016 • edited Loading

coveralls commented Oct 8, 2016

coveralls commented Oct 8, 2016 • edited Loading

satra commented Oct 8, 2016

satra commented Oct 8, 2016

trungdong commented Oct 9, 2016

satra commented Jun 23, 2014 •

edited

Loading

satra commented Mar 19, 2016 •

edited

Loading

coveralls commented Jun 27, 2016 •

edited

Loading

coveralls commented Oct 8, 2016 •

edited

Loading

coveralls commented Oct 8, 2016 •

edited

Loading