Headers are added to source chain and published for subsequent commits of the same entry

[willemolding]

When committing an entry there is an assumption that it idempotent. Commuting the same content multiple times should 'dedup' from the perspective of a single agent.

We have confirmed that this is NOT the case. Committing the same content multiple times will add to the source chain each time and also cause a new header entry to be published. Because of the design patterns used in many DNAs (e.g. committing a base/anchor every time assuming it will dedup) this is causing loads of useless entries to be published.

A few points:

• What is the actual expected behavior in this case? We assumed it is that identical content should dedup but is this actually what we want?
• Is a fix as simple as checking the chain CAS before going ahead with the commit_entry workflow?

How to replicate:

• Fire up holoscape
• Use passthrough DNA to commit an entry
• Commit it again
• Observe multiple headers in the chain and DHT stores

[pjkundert]

For Holofuel, I explicitly check the local source-chain for matching transaction-initiating commits, to ensure idempotency -- I use the Promise/Request commit Address as the unique "TxOrigin" for a transaction, so cannot allow duplicates.

I don't think it is practical or reasonable to dedup local source chain commits; The identical data can and will be committed, and each local source-chain commit is expected to persist as a separate entry with a different ChainHeader. However, I had assumed that they would have the same Address, and therefore would only exist once in the DHT, with one Provenance (AgentId, Signature) entry for each unique Agent that committed them.

So, if one Agent ID committed the same identical commit Address multiple times, we would only see one instance in the DHT, but multiple instances when we scanned their source chain.

[lucksus]

Hm, I'm also wondering where this assumption (that it is idempotent) is stemming from?

I agree with @pjkundert that a new commit should always result in a new chain entry plus header.

[freesig]

If we can't do this is it a reasonable idea to add something to the hdk like get_entry_local which looks up an address locally.

The problem that I've seen popping up is because of these headers developers are checking if an entry exists before committing. But because get_entry also checks the network it's possible for an agent to always be requesting the same entry from the network and never holding it on disk (although this is probably only an issue once we aren't all holding all the data).

Usecase

I need to get links from a base and I need to know the base has been committed for that to work.

Adds a new header every time I want to get links:

    let entry = Entry::App("my_entry".into(), "some_base".into())
    let address = hdk::commit_entry(&entry)?;
    hdk::get_links(
        &address,
        LinkMatch::Exactly("my_link"),
        LinkMatch::Any,
    )?

May never store the entry locally if the agent is connected to a network where another agent has already committed this entry.

    let entry = Entry::App("my_entry".into(), "some_base".into())
    let address = entry.address();
    if let Ok(None) = hdk::get_entry(&address) {
        hdk::commit_entry(&entry)?;
    }
    hdk::get_links(
        &address,
        LinkMatch::Exactly("my_link"),
        LinkMatch::Any,
    )?

I know we can use query but it seems wasteful to iterate through a vec when we have the hash address.