Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make the crawler even more generic #298

Open
bfabio opened this issue Aug 5, 2022 · 4 comments
Open

Make the crawler even more generic #298

bfabio opened this issue Aug 5, 2022 · 4 comments
Labels
enhancement Something improving existing features

Comments

@bfabio
Copy link
Member

bfabio commented Aug 5, 2022

Make the crawler even more generic by removing or parameterizing the leftover places where we are still italian specific.
@bfabio bfabio added the enhancement Something improving existing features label Feb 14, 2023
@bfabio
Copy link
Member Author

bfabio commented Feb 14, 2023

It would make sense to move publishers.thirdparty.yml to https://github.com/italia/developers.italia.it:

  • It's data
  • It's italian data concerning the italian catalog
  • It's closer to where the expected results of adding an entry are gonna end (i.e. the site)

We'll need to change:

@bfabio
Copy link
Member Author

bfabio commented Feb 14, 2023

We should also move the daily GitHub action that runs the new crawler to the developers.italia.it repo.

@bfabio bfabio mentioned this issue Mar 1, 2023
Merged
bfabio added a commit to italia/developers.italia.it that referenced this issue Mar 1, 2023
@bfabio
Merge pull request #1368 from italia/publishers
ed6bdaa 
Move publishers.thirdparty.yml from italia/publiccode-crawler

Also, edited the PR number in the commit logs to point to the original repo.

See italia/publiccode-crawler#298 (comment)
@bfabio
Copy link
Member Author

bfabio commented Mar 1, 2023

It would make sense to move publishers.thirdparty.yml to https://github.com/italia/developers.italia.it:

Done in italia/developers.italia.it#1368

bfabio added a commit that referenced this issue Mar 1, 2023
@bfabio
chore: remove publishers.thirdparty.yml
644ad8d 
Moved to the italia/developers.italia.it repo, see
* italia/developers.italia.it#1368
* #298 (comment)
@bfabio bfabio mentioned this issue Mar 1, 2023
Merged
bfabio added a commit that referenced this issue Mar 1, 2023
@bfabio
chore: remove publishers.thirdparty.yml (#333)
1d2990e 
Moved to the italia/developers.italia.it repo, see
* italia/developers.italia.it#1368
* #298 (comment)
@bfabio
Copy link
Member Author

bfabio commented Mar 20, 2023

We should also move the daily GitHub action that runs the new crawler to the developers.italia.it repo.

Done in italia/developers.italia.it#1376

bfabio added a commit to bfabio/developers.italia.it that referenced this issue Apr 1, 2023
@bfabio
chore: import publishers.yml from publiccode-crawler
66d3b35 
Import publishers.yml into this repo from
https://github.com/italia/publiccode-crawler/blob/1113650a8dc429cb072d0d7eb1b058821617c564/publishers.yml

See also ed6bdaa and
italia/publiccode-crawler#298
@bfabio bfabio mentioned this issue Apr 1, 2023
Merged
bfabio added a commit to italia/developers.italia.it that referenced this issue Apr 1, 2023
@bfabio
chore: import publishers.yml from publiccode-crawler (#1383)
1d0950f 
Import publishers.yml into this repo from
https://github.com/italia/publiccode-crawler/blob/1113650a8dc429cb072d0d7eb1b058821617c564/publishers.yml

See also ed6bdaa and
italia/publiccode-crawler#298
bfabio added a commit to bfabio/publiccode-crawler that referenced this issue Apr 1, 2023
@bfabio
chore(docker): get publishers.*.yml from italia/developers.italia.it
2a05fd7 
Get the publishers from the repo at italia/developers.italia.it
when building the Docker image.

italia#298
@bfabio bfabio mentioned this issue Apr 1, 2023
Merged
bfabio added a commit to bfabio/publiccode-crawler that referenced this issue Apr 1, 2023
@bfabio
chore(docker): get publishers.*.yml from italia/developers.italia.it
4d243a2 
Get the publishers from the repo at italia/developers.italia.it
when building the Docker image.

italia#298
bfabio added a commit that referenced this issue Apr 1, 2023
@bfabio
chore(docker): get publishers.*.yml from italia/developers.italia.it (#…
ff1df53 
…354)

Get the publishers from the repo at italia/developers.italia.it
when building the Docker image.

#298
bfabio added a commit to bfabio/developers.italia.it that referenced this issue Apr 1, 2023
@bfabio
content: update link to publishers.thirdparty.yml
fa86030 
publishers.thirdparty.yml is now in this repo
(italia/publiccode-crawler#298 (comment))
bfabio added a commit to bfabio/developers.italia.it that referenced this issue Apr 1, 2023
@bfabio
content: update link to publishers.thirdparty.yml
928ed1a 
publishers.thirdparty.yml is now in this repo
(italia/publiccode-crawler#298 (comment))
@bfabio bfabio mentioned this issue Apr 1, 2023
Merged
bfabio added a commit to italia/developers.italia.it that referenced this issue Apr 1, 2023
@bfabio
content: update link to publishers.thirdparty.yml (#1384)
8fe13c8 
publishers.thirdparty.yml is now in this repo
(italia/publiccode-crawler#298 (comment))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Something improving existing features
Projects
Developers Italia-next
Status: 📋 Backlog
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bfabio