Languages plugin no longer works

[Nixinova]

Today's metrics run deleted all languages data and doesn't add it back even in a following run. No errors are logged in the Action run.

Nixinova/Nixinova@6339db0

The only difference between today and yesterday is Linguist v7.16 came out a few hours ago; maybe this had something to do with it?

[lowlighter]

Hello,
Indeed, I have the same problem too 😕

It could come from the linguist release since the docker file is not version fixed, but the @main branch didn't got any docker image rebuild so it should have retained the previous version

I'll investigate later today, thanks for reporting

[Nixinova]

Run

Spammed with the following as the second debug statement:

#424 DEBUG: TypeError: files.map is not a function
    at file:///metrics/source/plugins/languages/analyzers.mjs:148:166
    at Array.flatMap (<anonymous>)
    at analyze (file:///metrics/source/plugins/languages/analyzers.mjs:148:133)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async indepth (file:///metrics/source/plugins/languages/analyzers.mjs:33:7)
    at async Object.default [as languages] (file:///metrics/source/plugins/languages/index.mjs:64:32)
    at async file:///metrics/source/plugins/core/index.mjs:50:30
    at async Promise.all (index 4)
    at async metrics (file:///metrics/source/app/metrics/index.mjs:61:22)

First debug is showing that the commit data is being fetched:

// line 2551
// #424 DEBUG: [
// 
[
    {
      sha: '274d05e67e8ca9fbca75ba3c0e02f52bad2a9804',
      filename: 'lib/linguist/languages.yml',
      status: 'modified',
      additions: 1,
      deletions: 0,
      changes: 1,
      blob_url: 'https://github.com/Nixinova/fork_linguist/blob/d49b21cd977093a3359f37b976e039e270d085a4/lib/linguist/languages.yml',
      raw_url: 'https://github.com/Nixinova/fork_linguist/raw/d49b21cd977093a3359f37b976e039e270d085a4/lib/linguist/languages.yml',
      contents_url: 'https://api.github.com/repos/Nixinova/fork_linguist/contents/lib/linguist/languages.yml?ref=d49b21cd977093a3359f37b976e039e270d085a4',
      patch: '@@ -6345,6 +6345,7 @@ Wget Config:\n' +
        '   language_id: 668457123\n' +
        ' Wikitext:\n' +
        '   type: prose\n' +
        '+  color: "#fc5757"\n' +
        '   wrap: true\n' +
        '   aliases:\n' +
        '   - mediawiki'
    }
  ]
// line 13036
//   ... 16 more items
// ]

[Nixinova]

Ah, I think may have I found the problem

metrics/source/plugins/languages/analyzers.mjs

Line 146 in d258f58

const files = Object.fromEntries(Object.entries(JSON.parse(await imports.run("github-linguist --json", {cwd:path}, {log:false}))).flatMap(([lang, files]) => files.map(file => [file, lang])))

await imports.run("github-linguist --json", {cwd:path}, {log:false}))).flatMap(([lang, files]) => //...

Linguist 7.16 did overhaul the output of --json:

• Add filesize metrics to github-linguist output github-linguist/linguist#5464
• github-linguist/linguist@b980593#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R122-R135

$ github-linguist --json
{"Dockerfile":{"size":1212,"percentage":"0.31"},"Ruby":{"size":264519,"percentage":"66.84"},"C":{"size":97685,"percentage":"24.68"},"Lex":{"size":5098,"percentage":"1.29"},"Shell":{"size":1257,"percentage":"0.32"},"Go":{"size":25999,"percentage":"6.57"}}

[lowlighter]

Thanks, greatly appreciate the help for debug 👍

Maybe it would be a good time to try to switch to Nixinova/Linguist and have a full JS implementation of linguist ?
It's always hasazardous to rely on spawn process, it may be more stable in the future

[Nixinova]

Yeah, I'll prioritise developing my Linguist further; currently it doesn't check gitignore or heuristics but I'll add those for v1.3.

[lowlighter]

Yeah, I'll prioritise developing my Linguist further; currently it doesn't check gitignore or heuristics but I'll add those for v1.3.

Ok, I'll try to rollback on previous linguist version in the meantime to try fix this issue as it's pretty big

[Nixinova]

Rollback to 7.15 worked, you can remove the two debug statements.

[Nixinova]

Closing since it's working now. If you want to use github-linguist@latest again you'll have to tweak that line to fit the new output format.

[Nixinova]

Yeah, I'll prioritise developing my Linguist further; currently it doesn't check gitignore or heuristics but I'll add those for v1.3.

Version 1.4.0 is out and supports both of these now. It should be good to use.

[Nixinova]

+ const linguist = require('linguist-js')
- const files = Object.fromEntries(Object.entries(JSON.parse(await imports.run("github-linguist --json", {cwd:path}, {log:false}))).flatMap(([lang, files]) => files.map(file => [file, lang])))
+ const {results:files} = await linguist(path, {})

https://github.com/Nixinova/fork_metrics/tree/replace-linguist