Raw data from stats page

[sdosemagen]

For the https://publiclab.org/stats page I'd like to either be able to download the raw data as a .csv (privacy considerations here?) and/or to potentially hover over a graph and be able to see the numbers per week.

[jywarren]

OK -- i'll begin breaking this out into sub-issues that'd have to be done before we could achieve this; is there a timeline you're hoping for, or can we enter this in the software-outreach queue for newcomers to tackle?

---

Currently, data is presented inline in https://github.com/publiclab/plots2/blob/master/app/views/notes/stats.html.erb#L84-L86, like:

    var notes = [[0,16],[1,16],[2,12],[3,25],[4,10],[5,8],[6,9],[7,9],[8,10],[9,22],[10,14],[11,12],[12,16],[13,29],[14,8],[15,14],[16,13],[17,13],[18,11],[19,12],[20,21],[21,22],[22,19],[23,21],[24,21],[25,16],[26,9],[27,9],[28,14],[29,13],[30,21],[31,4],[32,12],[33,16],[34,9],[35,7],[36,11],[37,10],[38,14],[39,19],[40,13],[41,15],[42,13],[43,19],[44,21],[45,17],[46,20],[47,57],[48,22],[49,11],[50,14],[51,11],[52,0]]
    var wikis = [[0,9],[1,1],[2,7],[3,1],[4,0],[5,0],[6,7],[7,4],[8,2],[9,8],[10,9],[11,8],[12,8],[13,2],[14,2],[15,0],[16,0],[17,1],[18,0],[19,1],[20,2],[21,2],[22,0],[23,0],[24,1],[25,2],[26,0],[27,2],[28,5],[29,3],[30,2],[31,1],[32,0],[33,0],[34,2],[35,1],[36,1],[37,2],[38,2],[39,4],[40,2],[41,2],[42,1],[43,0],[44,5],[45,2],[46,0],[47,4],[48,2],[49,2],[50,0],[51,0],[52,0]]
    var comments = [[0,21],[1,65],[2,69],[3,62],[4,46],[5,19],[6,70],[7,43],[8,23],[9,72],[10,39],[11,107],[12,80],[13,92],[14,36],[15,69],[16,113],[17,54],[18,35],[19,54],[20,100],[21,127],[22,96],[23,78],[24,48],[25,51],[26,35],[27,20],[28,35],[29,28],[30,34],[31,30],[32,41],[33,63],[34,56],[35,49],[36,33],[37,21],[38,64],[39,40],[40,55],[41,59],[42,46],[43,77],[44,53],[45,37],[46,63],[47,45],[48,57],[49,25],[50,20],[51,50],[52,0]]

Steps would be:

• move the collection of data into its own controller method such as def compile_stats using these three methods: Solved in stats controller, tests, and new ranges function, closes #964 #1103
• then make a new method which displays those as a .json file at, for example, https://publiclab.org/stats.json to which you pass a time parameter: stats.json?time=xxxxxx.
• consider breaking it up into three separate methods, such as /stats/notes.json, /stats/wikis.json in its own stats_controller.rb
• add a link to the stats.html.erb template that goes directly to the stats.json file so it can be downloaded.
• figure out a CSV presentation of this data. This could be manually, as adapted here: https://github.com/publiclab/spectral-workbench/blob/master/app/helpers/spectrums_helper.rb#L3-L17 or maybe there's a more standard ruby way to do this, with a gem

[jywarren]

@ebarry - can you offer some input on prioritization here, so we can consider bumping this up or putting it on the back burner? Thanks! 🎈

[jywarren]

Made an initial stab at this, but it'd need specifics on exactly what data would need to be downloadable: https://github.com/jywarren/plots2/tree/stats-export

@ebarry - we could alternatively seek to replace the graphs here with something from Chart.js, which would include hovers; http://www.chartjs.org/

[ebarry]

hi! @sdosemagen and I wanted to comment that this issue is not currently a priority, but it might be nice in the future to have easy access to the raw data.

[ebarry]

This is something still relevant to the evaluation project but not urgent for 2018 (in relation to other more urgent priorities)

[ghost]

Hi @jywarren ! I would like to work on this issue as to participate in this outreachy round. I think It would take me some time to figure out the solution as I'm very new to open source and in this project. It would be great if you guide me where I should start :)

[jywarren]

Hi! I think you might look at this query, and we could think about how to generate CSV output for different variables in there, and what URL we'd make them available at... what do you think? https://github.com/publiclab/plots2/blob/2a238ae5f326c9f91781395287e6e03ea87eef2f/app/controllers/stats_controller.rb#L11-L32 I.e. maybe https://publiclab.org/stats/range/1-1-2010/31-12-2018/ could break down into: https://publiclab.org/stats/range/1-1-2010/31-12-2018/people.csv https://publiclab.org/stats/range/1-1-2010/31-12-2018/notes.csv https://publiclab.org/stats/range/1-1-2010/31-12-2018/wikis.csv and so on! The routes are here: https://github.com/publiclab/plots2/blob/master/config/routes.rb and each route typically corresponds to a controller function. Does this help a bit?

…

On Tue, Oct 2, 2018 at 7:30 AM Kanika Shrivastav ***@***.***> wrote: Hi @jywarren <https://github.com/jywarren> ! I would like to work on this issue as to participate in this outreachy round. I think It would take me some time to figure out the solution as I'm very new to open source and in this project. It would be great if you guide me where I should start :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#963 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ6X2YimiOh5YegwJgtbLREl7B2K4ks5ug04dgaJpZM4KrUgL> .

[Radhikadua123]

Hi @jywarren and @kanikashrivastav

Can I take this task ? I have given thought to it about how to implement it.

  get 'stats' => 'stats#index'
  get 'stats/:type(/raw)' => 'stats#type'
  get 'stats/range/:start/:end(/raw)' => 'stats#range'
  get 'stats/:type/range/:start/:end(/raw)' => 'stats#range_with_type'

I think these routes should cover up everything. Optional parameter raw would give the exact same data in raw format which has been used in the charts. Though I'm not sure if it should be csv or json ?

:type can be subscriptions, people, notes, wikis, questions, comments, contributors etc - pretty much anything, making it generic enough.

What do you think @jywarren ?

[ghost]

@Radhikadua123 thanks for your interest, sorry but i am working on it , can you please look on any other issue available.

Thanks
Kanika

[Radhikadua123]

Sure, no worries! :)

[grvsachdeva]

As mentioned in the issue by @sdosemagen that privacy needs to be considered so whoever is/will be solving this issue, please take note of it. Although, I think only moderators and admin should be given privileges for downloading data.Thanks!

cc: @publiclab/community-reps

[Radhikadua123]

Hi @gauravano,

Even though I'm not working on this issue, it's part of outreachy project. So, thanks for reminding us about privacy concerns. I'll think about it how we can address this issue and change my proposal.

Thanks!

[cesswairimu]

@gauravano @SidharthBansal @jywarren @sagarpreet-chadha I am currently working on this.

[sagarpreet-chadha]

Awesome 😄 !

[jywarren]

Awesome!!

[SidharthBansal]

@cess Wairimu <[email protected]> It will be great if you can create a milestone and put all your project related work in that milestone. Changes looks good to me. Thanks

…

On Tue, Dec 4, 2018 at 8:26 PM Jeffrey Warren ***@***.***> wrote: Awesome!! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#963 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AUACQ3J4Y6-kPNBm85lKJEcK7YcWGh5qks5u1o0jgaJpZM4KrUgL> .

[cesswairimu]

Thanks @SidharthBansal, on it.

[SidharthBansal]

@cesswairimu as per Outreachy rules you need to write Blog post each week throughout the Outreachy period on the Public lab official website to tell us your progress.
Please mention what you have till now achieved, what are you currently working on and thinking to work upon.
Be thorough in writing the blogs. Be descriptive so that new folks next summer can read it and plan for Outreachy.
I hope this makes sense for you.

[SidharthBansal]

Can you please tell me the link to your Milestone so that I can have a look?
Thanks

[cesswairimu]

Yes it does thanks. Here is the link https://github.com/publiclab/plots2/milestone/20 @SidharthBansal, I am still adding issues

[SidharthBansal]

Please also add the work related to Outreachy which is merged.

Once you will create the blog please tag the mentors so that we can review.
Thanks

[SidharthBansal]

Thanks for creating the Milestone.
Looks good to me!

[jywarren]

Hi Cess -- I like Sidharth's point of blogging -- you should feel free to blog at PublicLab.org -- how does that sound? Thanks!

…

On Thu, Dec 6, 2018 at 10:52 AM Sidharth Bansal ***@***.***> wrote: Thanks for creating the Milestone. Looks good to me! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#963 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJxBEOFk5ve4AIv9vpTAnGgWGFQ9oks5u2T01gaJpZM4KrUgL> .

[cesswairimu]

@jywarren sounds good. Thanks.