Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for verifying downloaded file's SHA-512 hash #241

Open
EricRahm opened this issue Jan 29, 2015 · 25 comments
Open

Add support for verifying downloaded file's SHA-512 hash #241

EricRahm opened this issue Jan 29, 2015 · 25 comments
Labels

Comments

@EricRahm
Copy link
Contributor

Each build has an associated <product>...<locale><platform>.checksums file. The SHA-512 hash contained in the file could be used to verify the hash of the download.

Proposal:

  • Add a |--verify| flag
  • Download the checksums file and verify the the SHA-512 hash for the download matches
  • If the hash fails follow the standard retry logic already in place
@whimboo
Copy link
Contributor

whimboo commented Feb 9, 2015

@EricRahm I assume this hash only applies to the installer binaries to download, but not all the other files including the .zip file for Windows.

@EricRahm
Copy link
Contributor Author

EricRahm commented Feb 9, 2015

@whimboo There's a hash for every build file. For example: https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/01/2015-01-29-03-02-02-mozilla-central/firefox-38.0a1.en-US.win64-x86_64.checksums includes both the installer and the zip.

@whimboo
Copy link
Contributor

whimboo commented Feb 10, 2015

Ah, great. Thanks. Sounds like a good improvement.

@parthbakshi
Copy link

I would like to work on this bug, whats the best way to work on it.

@EricRahm
Copy link
Contributor Author

EricRahm commented May 6, 2015

@whimboo Do you have any notes for @parthbakshi?

@whimboo
Copy link
Contributor

whimboo commented May 6, 2015

Hi @parthbakshi. Great to hear that you have interests to work on this issue! Sorry for the late response but I was away for a full month.

So the best way is to fork the repository and create a local clone of it. Then you should make yourself familiar with the different kinds of Firefox builds (Nightly, Aurora, Beta, Release, candidate builds, tinderbox builds) which are supported by mozdownload. Once done it would be good to know how this checksum file looks like for those different kinds of builds. After that the download() method of the scraper would have to be extended to being able to also get the checksum file, parse the entries, and do the checks if specified via the command line (--verify).

Let me know if you have further questions. You can also reach me on the moznet IRC network in the #automation channel.

@parthbakshi
Copy link

Ok i have forked the repo, Is there a doc where i can read about the various builds and how the checksums/SHA-512 actually look?

@whimboo
Copy link
Contributor

whimboo commented May 8, 2015

The different builds are also called channels. You can get short introductions on MDN. Maybe by following the links you will get more comprehensive information. For the checksum you might want to have a look at the hashlib module documentation of Python.

@parthbakshi
Copy link

Whenever i run the mozdownload on my machine im getting an error executing it, is there anything that i am missing?

(venv)pbakshi@parthbakshi:~/personal-projects/mozilla/mozdownload$ ../venv/bin/mozdownload --version=latest
Traceback (most recent call last):
  File "../venv/bin/mozdownload", line 9, in <module>
    load_entry_point('mozdownload==1.14', 'console_scripts', 'mozdownload')()
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/pkg_resources/__init__.py", line 546, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/pkg_resources/__init__.py", line 2666, in load_entry_point
    return ep.load()
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/pkg_resources/__init__.py", line 2339, in load
    return self.resolve()
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/pkg_resources/__init__.py", line 2345, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozdownload-1.14-py2.6.egg/mozdownload/__init__.py", line 5, in <module>
    from scraper import *
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozdownload-1.14-py2.6.egg/mozdownload/scraper.py", line 19, in <module>
    import mozlog
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/__init__.py", line 19, in <module>
    import structured
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/structured/__init__.py", line 5, in <module>
    import commandline
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/structured/commandline.py", line 10, in <module>
    from structuredlog import StructuredLogger, set_default_logger
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/structured/structuredlog.py", line 14, in <module>
    from logtypes import Unicode, TestId, Status, SubStatus, Dict, List, Int, Any
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/structured/logtypes.py", line 97
    known_kwargs = {name: value for name, value in kwargs.iteritems()
                                  ^
SyntaxError: invalid syntax

@whimboo
Copy link
Contributor

whimboo commented May 11, 2015

As mentioned on IRC this is most likely Python version 2.6 which is causing those problems. Please upgrade to latest 2.7 and it should work.

@whimboo
Copy link
Contributor

whimboo commented Oct 29, 2015

@parthbakshi, given the amount of time which has been passed by here, I assume you will not work on this issue anymore? It's somewhat important for us now given that in the last days we had a couple of corrupted downloads without seeing a failure from mozdownload.

@parthbakshi
Copy link

Sorry work got in the way you can reassign it to someone else.

On Thu, Oct 29, 2015 at 6:03 PM Henrik Skupin [email protected]
wrote:

@parthbakshi https://github.com/parthbakshi, given the amount of time
which has been passed by here, I assume you will not work on this issue
anymore? It's somewhat important for us now given that in the last days we
had a couple of corrupted downloads without seeing a failure from
mozdownload.


Reply to this email directly or view it on GitHub
#241 (comment)
.

@whimboo
Copy link
Contributor

whimboo commented Oct 30, 2015

Thanks for letting me know!

@haouech
Copy link

haouech commented Oct 15, 2016

Hello,
I'm wondering, we need to pass a checksum as a parameter for the download() method and after the _download() has finished we need to generate and compare hashes?

@whimboo
Copy link
Contributor

whimboo commented Oct 17, 2016

So for each platform we have a checksums file located right next to the builds. This file contains checksums for all files related to the platform. So we should check first if such a file is present, download it, and then we can generate the hash locally and compare it with the one from the checksums file.

@haouech
Copy link

haouech commented Oct 17, 2016

I would like to work on this issue

@whimboo
Copy link
Contributor

whimboo commented Oct 18, 2016

Sure, go ahead and let us know how you proceed and where you need help. Don't hesitate to ask your questions on IRC. You can find me in the #automation channel. Thanks.

@Hyperion101010
Copy link

is it still open i would like to work on it

@whimboo
Copy link
Contributor

whimboo commented Oct 16, 2018

The active PR for this issue is #454. But we haven't gotten a reply from @haouech for a long time. Maybe you could reply so we can open up this issue again for someone else to work on? Thanks.

@Hyperion101010
Copy link

hi there @whimboo for the #454 issue as you said :you were going to distribute the code in pieces
like write it systematically
i would be happy to help
just two things
1)can you give me the names of files responsible for check_sum in the mozilla build ,
also a short intro on what are you planning like a short set-up will be quite helpful
2)and yeah open the issue let's work together (crytography has always been a fantasy)

@sayanmondal2098
Copy link

I would like to work on this issue

@shanudjn
Copy link

shanudjn commented Nov 5, 2018

I am new here . I want to work on this bug . Looking for mentorship to get started. Thank you.

Hyperion101010 pushed a commit to Hyperion101010/mozdownload that referenced this issue Dec 13, 2018
  After waiting for a long time , i have got time to work on this bug
  I saw the person before me hasn't commited any changes since last year(2017).
  Thus i am taking over the bug for submitting the patch .
  I tried with the patch but it is still incomplete
@m-agboola
Copy link

@whimboo Is this issue still open? If Yes, I'd like to work on it

@whimboo whimboo removed the mentored label Jan 8, 2019
@whimboo
Copy link
Contributor

whimboo commented Jan 8, 2019

It turned out that this particular issue was too complicated to get fixed in the past as mentored issue. As such I will remove the mentored flag.

@kaushiksk
Copy link
Contributor

Hey @whimboo I'd like to take a dig at this. Seemed like @rei2hu got most of the things right from what I can see in his PR. I have one question, are all checksum file names similar to what was mentioned earlier in this thread? i.e. extension of the release file replaced with .checksum or is there a specific format followed for each release? Any resource to help me with this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

9 participants