[BUG] No error for conflict extensions in manifest.json. #217

tomasliu-agora · 2024-10-29T14:44:44Z

Description

{
"type": "extension",
"name": "agora_rtc",
"version": "=0.8.0-rc2"
},
{
"type": "extension",
"name": "agora_rtm",
"version": "0.1.2"
}

two extensions require different runtime (0.2 and 0.3)

when run tman install or make build, no error is displayed and no change on the workspace.

tomas@WAGIT03652:~/CodeBase/github/czhen_TEN/agents$ tman install
🔍 Resolving packages...
💡 The following local packages do not appear in the dependency tree:
extension:[email protected]
extension:[email protected]
extension:[email protected]
system:[email protected]
extension:[email protected]
extension:[email protected]
extension:[email protected]
extension:[email protected]
extension:[email protected]
extension:[email protected]
extension:[email protected]
extension:[email protected]
extension:[email protected]
extension_group:[email protected]
extension:[email protected]
extension:[email protected]
system:[email protected]
extension:[email protected]
extension:[email protected]
extension:[email protected]
extension:[email protected]
extension:[email protected]
system:[email protected]
system:[email protected]+build331418
extension:[email protected]
extension:[email protected]
extension:[email protected]
system:[email protected]
extension:[email protected]
extension:[email protected]
extension:[email protected]
extension:[email protected]
system:[email protected]
extension:[email protected]
extension:[email protected]
📦 Installing packages...
[00:00:00] [########################################] 0/0 Done
🏆 Install successfully in 18 seconds

Environment

windows wsl ubuntu 22.04

Steps to reproduce

{
"type": "extension",
"name": "agora_rtc",
"version": "=0.8.0-rc2"
},
{
"type": "extension",
"name": "agora_rtm",
"version": "0.1.2"
}

two extensions require different runtime (0.2 and 0.3)

Expected behavior

report error and fail on the execution

Actual behavior

no error reported

Version

tman 0.3.0

Severity

Major

Additional Information

No response

The text was updated successfully, but these errors were encountered:

tomasliu-agora · 2024-10-30T01:33:03Z

attach whole manifest.json if needed.
{
"type": "app",
"name": "ten_agent",
"version": "0.4.0",
"dependencies": [
{
"type": "system",
"name": "ten_runtime_go",
"version": "0.3"
},
{
"type": "extension",
"name": "py_init_extension_cpp",
"version": "0.3"
},
{
"type": "extension",
"name": "agora_rtc",
"version": "=0.8.0-rc2"
},
{
"type": "extension",
"name": "agora_rtm",
"version": "0.1.2"
},
{
"type": "system",
"name": "azure_speech_sdk",
"version": "1.38.0"
},
{
"type": "extension",
"name": "azure_tts",
"version": "=0.5.1"
}
]
}

leoadonia · 2024-10-30T02:56:13Z

tman resolves the package candidates based on the semantic version syntax.

The agora_rtm extension declared in manifest.json is:

{
  "type": "extension",
  "name": "agora_rtm",
  "version": "0.1.2"
}

The version 0.1.2 means the candidates in the range [0.1.2, 0.2.0), and the highest version will be resolved if there is no conflict dependencies.

Extension agora_rtm has a version 0.1.3, which depends on ten_runtime with version 0.3:

{
  "type": "system",
  "name": "ten_runtime",
  "version": "0.3"
}

So, after tman install, the following packages will be resolved:

ten_runtime: 0.3.1
agora_rtc: 0.8.0-rc2
agora_rtm: 0.1.3

The behavior is as expected.

leoadonia · 2024-10-30T02:58:01Z

However, if we change the extension agora_rtm in the manifest.json to =0.1.2, which means only the version 0.1.2 is wanted.

Then, after tman install, the output will be as follows:

🔍  Resolving packages...
🔒  Creating manifest-lock.json...
📦  Installing packages...
  [00:00:00] [########################################]       0/0       Done                                                                                                                                                                            🏆  Install successfully in 27 seconds

No package is resolved.

To be fixed.

leoadonia · 2024-10-31T10:20:14Z

现状

目前 tman 中是使用 clingo 来解析依赖树的最佳版本, 会根据每个依赖的 version 获取符合条件的 candidates 作为问题描述, 由 clingo 给出可选的答案. 但如果 clingo 给不出答案, clingo 只会给出每个 model 不满足 main.lp 中的约束条件时定义的错误信息. 即错误信息的内容是面向的一个 model 维度的, 并无法给出一个 model 不满足条件的具体的影响因子是什么.

尤其是当前 main.lp 中会约束只能输出一个答案, 如下:

% Select only 1 version for the dependency package.
1 { depends_on(PkgType, PkgName, PkgVersion, DepType, DepName, DepVersion) : depends_on_declared(PkgType, PkgName, PkgVersion, DepType, DepName, DepVersion) } 1
  :- selected_pkg_version(PkgType, PkgName, PkgVersion),
     depends_on_declared(PkgType, PkgName, PkgVersion, DepType, DepName, _).

就导致就算在 main.lp 中增加如下的错误条件时, 在无法匹配到答案时, 也不会输出可选的 model 列表或者错误信息.

error(0, "No suitable version found for dependency '{0}::{1}'", PkgName, PkgVersion)
  :- not selected_pkg_version(DepType, DepName, DepVersion),
     depends_on_declared(PkgType, PkgName, PkgVersion, DepType, DepName, DepVersion).

目标

在无法匹配整个依赖树的最佳版本时, 能够给出两棵子树的具体哪个子依赖冲突导致的.

其他包管理工具的实现

cargo

比如, 在 ten_manager 和 ten_rust 中分别声明依赖 serde_json 的两个不同的版本, cargo tree 会给出冲突的具体路径:

error: failed to select a version for `serde_json`.
    ... required by package `ten_rust v0.1.0 (/home/adonia/00_code/ten_framework_internal_base/ten_framework/core/src/ten_rust)`
    ... which satisfies path dependency `ten_rust` (locked to 0.1.0) of package `ten_manager v0.1.0 (/home/adonia/00_code/ten_framework_internal_base/ten_framework/core/src/ten_manager)`
versions that meet the requirements `=1.0.108` are: 1.0.108

all possible versions conflict with previously selected packages.

  previously selected package `serde_json v1.0.132`
    ... which satisfies dependency `serde_json = "=1.0.132"` of package `ten_manager v0.1.0 (/home/adonia/00_code/ten_framework_internal_base/ten_framework/core/src/ten_manager)`

failed to select a version for `serde_json` which could resolve this conflict

判断依赖冲突的主要逻辑:

首先, 创建一个全局的 activations, (activations 是已解析判断没有版本冲突的 candidates)
根据直接依赖构建依赖的子树列表, 作为 remaining_deps.
依次从 remaining_deps 取出一颗子树, 获取 root 节点的可用的 candidates ( 按照 rust feature 和 version 匹配), 每个子树维度维护一个 ConflictMap. 按深度优先的顺序遍历每个 candidate 的依赖列表, 检查是否与全局的 activations 中相同的 package 存在版本冲突, 如果有, 跳过当前 candidate.
如果一棵子树中无法获取到合适的 candidate, 依赖树解析失败.

实现

不采用与 cargo 相同的方式, 因为目前 clingo 的职责就是解析依赖树的最佳匹配的 candidate, cargo 的实现与clingo的方式有重复的部分.

可以先确认一个正常的子树, 以及与子树产生冲突的package.

以 root package (app 或者 extension) 的直接依赖 (即 manifest.json dependecies) 作为输入, direct_deps

  for i := 2 -> len(direct_deps):
     sub_graph = direct_deps[0..i]
     generate_input_lp(sub_graph)
     sucess := solve(generate_input_lp)
     if success:
         continue
     else:
          break

即可确认 direct_deps[i] 与 direct_deps[0..i-1] 子树有冲突. 同时, 可以把 direct_deps[0..i-1] 解析出的结果保存.

确认具体哪个子依赖产生冲突

上一步确认了产生冲突前的最大子树的 lock 内容, 以及产生冲突的直接依赖 (conflicted_pkg). lock 内容已经是确认了的具体版本的 candidate.

而且 conflicted_pkg 的所有可选的 candidate 对应的子树中一定有 package 与 lock 内容中的 package 产生了版本冲突. 同时, tman 目前默认是选择最高版本的 candidate. 所以, conflicted_pkg 的最高版本的 candidate 的子树中一定有 package 与 lock 内容的 package 产生冲突.

可以以 conflicted_pkg 为 root, 广度遍历依赖, 通过 semantic version 规则判断, 直到遇到冲突为止.

只能输出 conflicted_pkg 的哪个依赖与现有的有冲突, 如果要输出类似 cargo 的报错, 需要根据查找到的依赖 package 从 lock 内容形成的依赖树中查找.
更进一步, 可以将 conflicted_pkg 的 version_req 改成 "*", 重新通过 clingo 匹配, 给出相匹配的版本号, 作为修复建议.

整体上, 增加 determine_conflict(), 在原有的 solve_all() 返回为空时调用.

halajohn · 2024-11-03T14:19:31Z

不需要这么麻烦.

我改了一版纯 answer set programming 的解法, 已经合入 main branch, 你用原本的例子测试看看, 如果可以达成原本想要达成的样子, 可以关掉这笔 issue.

Answer Set Programming 本身就可以做到冲突侦测, 追踪依赖关系, 并输出用户易懂的错误讯息.
逻辑全部集中在 answer set programming 中, 会比分散一部分在 ASP, 一部分在 rust 逻辑代码中要好很多.
或者逻辑全部集中在 rust 中, 也是另一种统一. 但由于依赖冲突的解决是 NP-complete 问题, 所以我们引入 ASP 来做我们的 solver, 因此直接在 ASP program 中解决这个问题应该是最适当的.

底下是纯粹修改 ASP program 完后的结果. 先是正常情况下的 terminal 输出.

$ tman install
🔍  Resolving packages...
📥  Installing packages...
  [00:00:00] [########################################]       3/3       Done                                                                                                       
🏆  Install successfully in 0 seconds

底下是错误情况下的 terminal 输出. 可以看到只凭 answer set programming 就可以作到侦测冲突, 侦测冲突的依赖原因, 并最终输出易懂的 terminal output.

$ tman install
🔍  Resolving packages...
❌  Error: Select more than 1 version of '[system]ten_runtime': '@0.3.0' introduced by '[app][email protected]', and '@0.4.0' introduced by '[extension][email protected]'

Dependency chain leading to [system][email protected]:
└─ [app][email protected]
   └─ [system][email protected]

Dependency chain leading to [system][email protected]:
└─ [app][email protected]
   └─ [extension][email protected]
      └─ [system][email protected]

❌  Error: Dependency resolution failed.

leoadonia · 2024-11-04T03:16:57Z

验证可以检测冲突:

🔍  Resolving packages...
❌  Error: Select more than 1 version of '[system]ten_runtime': '@0.2.0' introduced by '[extension][email protected]', and '@0.3.1' introduced by '[system][email protected]'

Dependency chain leading to [system][email protected]:
└─ [app][email protected]
   └─ [extension][email protected]
      └─ [extension][email protected]
         └─ [system][email protected]

Dependency chain leading to [system][email protected]:
└─ [app][email protected]
   └─ [extension][email protected]
      └─ [system][email protected]
         └─ [system][email protected]

❌  Error: Dependency resolution failed.

leoadonia · 2024-11-04T03:32:49Z

还有错误提示的小问题:

错误信息中提示的是尝试的版本, 不是开发者具体定义在 manifest.json 的 version .

按错误信息来看, version weight 的作用不太一致.

case 1:

app :

{
   "type": "extension",
   "name": "agora_rtm",
   "version": "<=0.1.2"
 }

执行多次, 输出的错误信息中的版本号是不同的:

Dependency chain leading to [system][email protected]:
└─ [app][email protected]
  └─ [extension][email protected]
     └─ [system][email protected]

Dependency chain leading to [system][email protected]:
└─ [app][email protected]
  └─ [extension][email protected]
     └─ [system][email protected]
        └─ [system][email protected]

Dependency chain leading to [system][email protected]:
└─ [app][email protected]
  └─ [extension][email protected]
     └─ [system][email protected]

Dependency chain leading to [system][email protected]:
└─ [app][email protected]
  └─ [extension][email protected]
     └─ [system][email protected]
        └─ [system][email protected]

分别是 0.1.0 和 0.1.2.

case 2:

app :

  {
     "type": "extension",
     "name": "agora_rtm",
     "version": "<0.1.5"
   }

edge_metrics_extension_go :

{
   "type": "extension",
   "name": "agora_rtm",
   "version": ">0.1.3"
 }

edge_ncs_extension_go :

{
   "type": "extension",
   "name": "agora_rtm",
   "version": "=0.1.3"
 }

错误信息如下:

```text
❌  Error: Select more than 1 version of '[extension]agora_rtm': '@0.1.3' introduced by '[app][email protected]', and '@0.1.5' introduced by '[extension][email protected]'

Dependency chain leading to [extension][email protected]:
└─ [app][email protected]
   └─ [extension][email protected]
      └─ [extension][email protected]

Dependency chain leading to [extension][email protected]:
└─ [app][email protected]
   └─ [extension][email protected]
      └─ [extension][email protected]
```

leoadonia · 2024-11-05T09:02:19Z

上述 case 1 中, tman install 多次执行结果不同的原因是, 内部使用 HashSet 存储 candidates 的, 导致每次遍历生成 input.lp 时, 同一个 pkg 的多个 candidates 的顺序是不同的, 生成的 input.lp 中的多个 depends_on_declared 的写入顺序不同. 如:
```
depends_on_declared("app", "ten_agent", "0.4.0", "extension", "agora_rtm", "0.1.0").
depends_on_declared("app", "ten_agent", "0.4.0", "extension", "agora_rtm", "0.1.1").
depends_on_declared("app", "ten_agent", "0.4.0", "extension", "agora_rtm", "0.1.2").
```
也就是, 并不是每次生成的顺序都是 0.1.0 -> 0.1.2.

clingo 在生成 answer model 时, 会按照顺序尝试, 如果已经尝试了高版本, 发现找不到 answer, 就不会再尝试低版本. 如: 如果 0.1.1 定义在前, 那生成的 answer model 列表中只会包含对 0.1.1 和 0.1.2 的尝试, 0.1.0 会忽略.

同时, 无论怎么排序, 最高版本的 0.1.2 总是会尝试, 所以, 只要取 answer model 中的最后一个, 就可以保证每次获取到的信息是一致的.
case 2 中的问题是构建 denpendency chain 时, 不应该将 dep pkg 作为 leaf node, 而应该是 introducer. 因为比如 A 和 B 因为 ten_runtime 产生了冲突, 可能整个依赖树中还有 C 依赖了 ten_runtime. 如果根据 ten_runtime 查找依赖链路, 可能会找到 C.

tomasliu-agora added the bug Something isn't working label Oct 29, 2024

halajohn assigned leoadonia Oct 29, 2024

leoadonia added the tman label Oct 30, 2024

halajohn closed this as completed Nov 4, 2024

halajohn reopened this Nov 4, 2024

leoadonia linked a pull request Nov 5, 2024 that will close this issue

fix: conflicted error msg #235

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] No error for conflict extensions in manifest.json. #217

[BUG] No error for conflict extensions in manifest.json. #217

tomasliu-agora commented Oct 29, 2024

tomasliu-agora commented Oct 30, 2024

leoadonia commented Oct 30, 2024

leoadonia commented Oct 30, 2024

leoadonia commented Oct 31, 2024 •

edited by halajohn

Loading

halajohn commented Nov 3, 2024

leoadonia commented Nov 4, 2024

leoadonia commented Nov 4, 2024

leoadonia commented Nov 5, 2024

[BUG] No error for conflict extensions in manifest.json. #217

[BUG] No error for conflict extensions in manifest.json. #217

Comments

tomasliu-agora commented Oct 29, 2024

Description

Environment

Steps to reproduce

Expected behavior

Actual behavior

Version

Severity

Additional Information

tomasliu-agora commented Oct 30, 2024

leoadonia commented Oct 30, 2024

leoadonia commented Oct 30, 2024

leoadonia commented Oct 31, 2024 • edited by halajohn Loading

现状

目标

其他包管理工具的实现

cargo

实现

halajohn commented Nov 3, 2024

leoadonia commented Nov 4, 2024

leoadonia commented Nov 4, 2024

leoadonia commented Nov 5, 2024

leoadonia commented Oct 31, 2024 •

edited by halajohn

Loading