Stats downloading returns "Page does not exist" for dates prior to early 2013 #5490

skilfullycurled · 2019-04-15T23:19:08Z

I was trying to download stats from the beginning of the site (which was sometime in 2010) until July of 2016 and I received a "Page does not exist" error. I did some experimentation and while I'm not sure of the exact date before which this occurs, I can say it is sometime between 01-01-2013 and 01-07-2013 (in stats page date format DD-MM-YYYY). For what it's worth to the diagnosis of the problem, I was "only" downloading up until July of 2016.

Let me know if there's anything else I can tell you (FF 66.0.2, macOS 10.12.6).

Oh, a possible reference where this could be merged: Raw data from stats page, #4654

NOTE for high impact URLs which could slow the main website down, if you want to test them out, please use stable.publiclab.org instead of publiclab.org and you'll only slow or break the stable test server, which should have very similar code and data. (note by @jywarren)

The text was updated successfully, but these errors were encountered:

cesswairimu · 2019-04-17T17:10:18Z

🤔 Will take a look at why this is happening thanks @skilfullycurled

cesswairimu · 2019-04-17T18:55:52Z

@skilfullycurled when did you receive the error when submitting the date ranges or the data was returned fine and this occurred when you were trying to "download as"?

skilfullycurled · 2019-04-17T18:59:22Z

No, @cesswairimu, it was the search itself that returned the error.

skilfullycurled · 2019-04-17T20:04:07Z

Oh, I forgot about that date. @cesswairimu, we're never supposed mention that date or even speak about what happened on it. It's sort of the Voldemort of dates.

But seriously folks.

Very interesting.

I'll download around it.

cesswairimu · 2019-04-17T20:05:26Z

Aha gotcha sorry will delete the comment

cesswairimu · 2019-04-17T20:09:19Z

@skilfullycurled maybe we can close this now since its not a code issue?

jywarren · 2019-04-17T23:16:13Z

lol are you folks having too much fun in here? Voldemort dates? 🔮 🤐 🙊 I hope Cess has seen Harry Potter and knows you're joking? Cess, @skilfullycurled has a strange sense of humor please don't hold it against him.

No but for real, thanks for looking into this. I'm sure it was a tough one to track down esp. given the mysterious date.

cesswairimu · 2019-04-17T23:21:31Z

😆 😆

skilfullycurled · 2019-04-17T23:24:01Z

Oh my gosh. @cesswairimu, I'm so sorry. So sorry. I feel terrible. I was hoping to convey I was kidding when I said "but seriously folks". PS: @jywarren, I'll have you know that all the people who consistently laugh at my jokes think I have a great sense of humor.

skilfullycurled · 2019-04-17T23:26:54Z

Actually, on a legitimately serious note, I'll try to be a little more clear humor. I've seen enough furrowed brows to readily admit that my sense of humor can be confusing even in person because I would have said that with a completely straight face.

cesswairimu · 2019-04-17T23:29:01Z

@skilfullycurled no its fine, blame it on having english as my second language 😆

skilfullycurled · 2019-04-17T23:38:40Z

I appreciate that. In the meantime, I should probably learn to chill out just a bit when I first meet people.

It looks like you actually deleted the comment with the date, so for the record and people seeing this trying to track down a bug, it was April 25, 2013 I think...? Also, I'd still like to know if there's a reason! What I really thought was funny was that there would be one specific day prior to, and after, everything would work fine, but just not that day. So please if anyone knows, do end the mystery!

jywarren · 2019-04-17T23:45:50Z

OK, we are synced up now I feel. Thank you Cess for being understanding and thank you Benjamin for the same! I do feel that date must be haunted! And, no worries Benjamin, i love your sense of humor!

…

On Wed, Apr 17, 2019 at 7:39 PM skilfullycurled ***@***.***> wrote: I appreciate that. In the meantime, I should probably learn to chill out just a bit when I first meet people. It looks like you actually deleted the comment with the date, so for the record and people seeing this trying to track down a bug, it was April 25, 2013 I think...? Also, I'd still like to know if there's a reason! What I really thought was funny was that there would be one specific day prior to, and after, everything would work fine, but just not that day. So please if anyone knows, do end the mystery! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5490 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAF6J3YZ4KFPI5NMKHK7DLPQ6YIDANCNFSM4HGEHCFA> .

skilfullycurled · 2019-04-17T23:49:03Z

And I am a great admirer of yours, too!

skilfullycurled · 2019-04-18T17:54:21Z

Well, not great news.

I got the "That page does not exist" for a new date set: 01-01-2012 - 01-01-2013.

skilfullycurled · 2019-04-18T18:03:43Z

Oh, also, the users.csv download for 01-01-2013 until 25-04-2013 is returning both incorrect data and incomplete data. It returns 154 users and those users are random from UID 1 to 58354.

skilfullycurled · 2019-04-22T03:56:11Z

Hey all, let me know if there is anything that I can do to try and diagnose the situation beyond simply telling you what dates aren't work. I just figured since I'm using the interface, I might as well be of some use!

skilfullycurled · 2019-05-12T21:12:48Z

Hi, returning to this conversation since I'm trying to plan a bit for the summer. Can I be of help on this? And if so, what would be the most systematic way to figure out which dates are causing the problem. And also, how can I avoid taking down the whole site? Do people use unstable?

jywarren · 2019-05-14T00:47:15Z

No, unstable and stable are both fine to hammer on as much as you'd like!

As to how to debug, i'm not sure... we could try to pull logs for when it happens. Wait - let me link some Sentry issues and see if they shine any light?

sentry-io · 2019-05-14T00:47:52Z

Sentry issue: PLOTS2-6H

skilfullycurled · 2019-05-14T00:59:22Z

@jywarren, not sure if I did this right, but I just signed up for Sentry and requested access. I wasn't sure what permissions to request so I just requested the default ones that sentry provided for me. Let me know if I need to change anything.

skilfullycurled · 2019-05-14T00:59:47Z

(also, thanks!)

jywarren · 2019-05-14T01:01:46Z

Hmm, weird that you can't see the error log, it's supposed to be public! But, ok here it is:

ActionController::UnknownFormat: ActionController::UnknownFormat
  from action_controller/metal/mime_responds.rb:205:in `respond_to'
  from app/controllers/stats_controller.rb:141:in `format'
  from app/controllers/stats_controller.rb:110:in `tags'
  from action_controller/metal/basic_implicit_render.rb:6:in `send_action'

But i'm not convinced this is for the same page: It says this is for:

https://publiclab.org/stats/tags

skilfullycurled · 2019-05-14T01:15:15Z

Just accepted the invite, this is awesome, thank you!

skilfullycurled · 2019-05-14T03:46:02Z

Well, I've spent a fair amount of time with Sentry and and Skylight and I can't seem to find an error that is registered when I recreate the error on the site with the dates in question. I even turned on a VPN so that I could be confident that I would see the correct IP address but nothing appears in the events log in Sentry. Something might be happening in Skylight but I can't figure out how to see the timestamp of the errors. Thoughts?

ebarry · 2019-05-30T19:26:57Z

I have also encountered this error when attempting to guestimate the start date for our website so that i could view data "For All Time".

I'm wondering if it's really a bug or perhaps maybe it could be worked around by people not having to guess when we started logging data for our website?

If it's the latter, here's an idea:
On the "Choose a Start Date" interface, could we add the exact start date of our website as a "quick pick" OR perhaps make an option to choose "for all time" as the period of inquiry?

jywarren · 2019-06-18T03:01:33Z

https://publiclab.org/stats?start=01-01-2012&end=01-01-2013&commit=Go now works!

jywarren · 2019-06-18T03:03:20Z

Now, we are DD-MM-YYYY, so this should work: https://publiclab.org/stats?start=01-01-2012&end=25-04-2013

jywarren · 2019-06-18T03:05:01Z

But it doesn't. Something similar perhaps:

[0b0b480c-084e-45a5-9805-ef990116fd68] Completed 404 Not Found in 4838ms (ActiveRecord: 1788.7ms)
[0b0b480c-084e-45a5-9805-ef990116fd68]   
[0b0b480c-084e-45a5-9805-ef990116fd68] ActiveRecord::RecordNotFound (Couldn't find Tag with 'tid'=3015):
[0b0b480c-084e-45a5-9805-ef990116fd68]   
[0b0b480c-084e-45a5-9805-ef990116fd68] app/models/tag.rb:65:in `block in nodes_frequency'
[0b0b480c-084e-45a5-9805-ef990116fd68] app/models/tag.rb:65:in `map'
[0b0b480c-084e-45a5-9805-ef990116fd68] app/models/tag.rb:65:in `nodes_frequency'
[0b0b480c-084e-45a5-9805-ef990116fd68] app/controllers/stats_controller.rb:36:in `block in range'
[0b0b480c-084e-45a5-9805-ef990116fd68] app/controllers/stats_controller.rb:19:in `range'
[0b0b480c-084e-45a5-9805-ef990116fd68] app/controllers/stats_controller.rb:41:in `index'

I think these range stats queries are just good at finding lonesome db records, because they try to gather ALL such records across huge swaths.

jywarren · 2019-06-18T03:05:59Z

I'll find and kill that NodeTag record too...

jywarren · 2019-06-18T03:31:21Z

OK, deleted lonesome NodeTag records pointing at non-existent tids of 3015 and 3016, and https://publiclab.org/stats?start=01-01-2012&end=25-04-2013 now works. Is that it, then? 🕵️‍♀️ 🕵

skilfullycurled · 2019-06-18T03:34:38Z

Should be for this issue...PS: where in Sentry can you see the 404's? I thought they did not show them?

skilfullycurled · 2019-06-18T03:37:02Z

oh, and thanks @jywarren! This is been like a splinter in my data finger. Thanks for tweez-ing it out!

jywarren · 2019-06-18T03:37:22Z

Oh, i think the Sentry was for a different error - i guess for the "date out of range" error on Date.parse given that i'd reversed 25-04-2013 to 04-25-2013 -- so trying for the 25th month.

…

On Mon, Jun 17, 2019 at 11:35 PM Benjamin Sugar ***@***.***> wrote: Should be for this issue...PS: where in Sentry can you see the 404's? I thought they did not show them? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5490?email_source=notifications&email_token=AAAF6J7MTTKKI56EZAM2JE3P3BJVBA5CNFSM4HGEHCFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX5CGRI#issuecomment-502932293>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAF6J7JRUXODUDXCO4KL3TP3BJVBANCNFSM4HGEHCFA> .

skilfullycurled · 2019-06-18T04:01:01Z

Ah, I see, the 404's are still logged somewhere though. Good to know!

jywarren · 2019-06-18T04:19:26Z

Thanks to you and Cess for chipping away at this so assiduously! It wouldn't have been nearly as solvable without the thorough documentation and investigation.

…

On Tue, Jun 18, 2019, 12:01 AM Benjamin Sugar ***@***.***> wrote: Ah, I see, the 404's are still logged somewhere though. Good to know! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5490?email_source=notifications&email_token=AAAF6J4XWGTGQ6HYPGEQT23P3BMX3A5CNFSM4HGEHCFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX5DI3I#issuecomment-502936685>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAF6J6RRNRLDJCI3TJZU63P3BMX3ANCNFSM4HGEHCFA> .

jywarren · 2020-10-20T15:19:22Z

https://stable.publiclab.org/stats?start=20-07-2010&end=20-10-2020 is still returning a 404 Page does not exist error, so i will leave this open for reference! Thanks!

jywarren · 2020-10-20T15:27:17Z

And noting @skilfullycurled's note on a "workaround date range" in #5904 (comment) -

this in #6050 means that the "all time" option will be only going back to Jan 1, 2014.

Next steps summarized by @skilfullycurled here:

#6050 (comment)

jywarren · 2020-10-20T15:36:01Z

OK, just an update in looking for NodeTag records with no associated Tag record as @skilfullycurled mentioned in #6050, I found these tids to look at:

irb(main):010:0> NodeTag.where('date > 1366851600 AND date < 1366990345').size
=> 70
irb(main):011:0> NodeTag.where('date > 1366851600 AND date < 1366990345').collect(&:tid)
=> [1, 14, 125, 446, 578, 578, 579, 579, 1316, 2421, 3049, 3049, 3049, 3049, 3049, 3049, 3050, 3050, 3050, 3050, 3050, 3050, 3051, 3051, 3051, 3051, 3051, 3051, 3051, 3051, 3052, 3053, 3054, 3055, 3057, 3057, 3058, 3059, 3060, 3061, 3062, 3063, 3064, 3070, 3071, 3072, 3082, 3088, 3089, 3091, 3092, 3093, 3094, 3095, 3097, 3097, 3098, 3098, 3099, 3100, 3101, 3102, 3103, 3104, 3105, 3106, 3109, 3111, 3112, 3114]
irb(main):012:0> NodeTag.where('date > 1366851600 AND date < 1366990345').collect(&:tid).size
=> 70
irb(main):013:0> NodeTag.where('date > 1366851600 AND date < 1366990345').collect(&:tid).uniq.size
=> 48
irb(main):014:0> NodeTag.where('date > 1366851600 AND date < 1366990345').collect(&:tid).uniq
=> [1, 14, 125, 446, 578, 579, 1316, 2421, 3049, 3050, 3051, 3052, 3053, 3054, 3055, 3057, 3058, 3059, 3060, 3061, 3062, 3063, 3064, 3070, 3071, 3072, 3082, 3088, 3089, 3091, 3092, 3093, 3094, 3095, 3097, 3098, 3099, 3100, 3101, 3102, 3103, 3104, 3105, 3106, 3109, 3111, 3112, 3114]

indeed:

ActiveRecord::RecordNotFound (Couldn't find all Tags with 'tid': (1, 14, 125, 446, 578, 579, 1316, 2421, 3049, 3050, 3051, 3052, 3053, 3054, 3055, 3057, 3058, 3059, 3060, 3061, 3062, 3063, 3064, 3070, 3071, 3072, 3082, 3088, 3089, 3091, 3092, 3093, 3094, 3095, 3097, 3098, 3099, 3100, 3101, 3102, 3103, 3104, 3105, 3106, 3109, 3111, 3112, 3114) (found 46 results, but was looking for 48).)

I'll try figuring out which is missing. OK - only these two:

ActiveRecord::RecordNotFound (Couldn't find Tag with 'tid'=3088)
ActiveRecord::RecordNotFound (Couldn't find Tag with 'tid'=3089)

Deleting them.

jywarren · 2020-10-20T16:16:51Z

https://stable.publiclab.org/stats?start=20-07-2010&end=20-10-2020 still shows a 404, so i'll keep looking for extra NodeTag records with no associated Tag record.

jywarren · 2020-10-20T21:21:37Z

Hmm, i tried a bunch of ways like https://stackoverflow.com/questions/5319400/want-to-find-records-with-no-associated-records-in-rails, but didn't have much success --

NodeTag.includes(:tag).where(term_data: {name: nil})
=> #<ActiveRecord::Relation []>

I also just tried collecting ALL NodeTag tids and subtracting all valid Tag tids, which took a while:

nodetag_tids = NodeTag.select(&:tid).collect(&:tid).uniq
tag_tids = Tag.select(&:tid).collect(&:tid).uniq
tids_missing = nodetag_tids - tag_tids
=> []

So, doesn't that mean all NodeTags have valid tids?

Maybe we should look in Sentry for another kind of error now?

https://stable.publiclab.org/stats?start=20-01-2013&end=20-01-2014 shows 404
https://stable.publiclab.org/stats?start=20-01-2014&end=20-01-2015 loads fine

I searched the logs... didn't find much. Maybe this?

[ba658853-5c5b-4514-b69b-bbde1cf920c5] Processing by StatsController#index as */*
[ba658853-5c5b-4514-b69b-bbde1cf920c5]   Parameters: {"utf8"=>"✓", "options"=>"Week"}
[ba658853-5c5b-4514-b69b-bbde1cf920c5] Completed 500 Internal Server Error in 5ms (ActiveRecord: 0.0ms)
[ba658853-5c5b-4514-b69b-bbde1cf920c5] Sending event 79acc5f697a249f7aaddacfa69966bcf to Sentry
[ba658853-5c5b-4514-b69b-bbde1cf920c5]   
[ba658853-5c5b-4514-b69b-bbde1cf920c5] NoMethodError (undefined method `downcase' for nil:NilClass):
[ba658853-5c5b-4514-b69b-bbde1cf920c5]   
[ba658853-5c5b-4514-b69b-bbde1cf920c5] app/controllers/stats_controller.rb:150:in `to_keyword'
[ba658853-5c5b-4514-b69b-bbde1cf920c5] app/controllers/stats_controller.rb:20:in `range'
[ba658853-5c5b-4514-b69b-bbde1cf920c5] app/controllers/stats_controller.rb:48:in `index'

But that was on Oct 14th, six days ago. I can't seem to find the errors for today... strange.

jywarren · 2020-10-20T21:27:19Z

Ah! was looking in the wrong directory! Got it!

[2c6a71d3-e104-44b7-98ab-44263e879b22] Processing by StatsController#index as HTML
[2c6a71d3-e104-44b7-98ab-44263e879b22]   Parameters: {"start"=>"20-01-2013", "end"=>"20-01-2014"}
[2c6a71d3-e104-44b7-98ab-44263e879b22] Completed 404 Not Found in 11475ms (ActiveRecord: 9149.4ms)
[2c6a71d3-e104-44b7-98ab-44263e879b22]   
[2c6a71d3-e104-44b7-98ab-44263e879b22] ActiveRecord::RecordNotFound (Couldn't find Tag with 'tid'=3088):
[2c6a71d3-e104-44b7-98ab-44263e879b22]   
[2c6a71d3-e104-44b7-98ab-44263e879b22] app/models/tag.rb:55:in `block in nodes_frequency'
[2c6a71d3-e104-44b7-98ab-44263e879b22] app/models/tag.rb:55:in `map'
[2c6a71d3-e104-44b7-98ab-44263e879b22] app/models/tag.rb:55:in `nodes_frequency'
[2c6a71d3-e104-44b7-98ab-44263e879b22] app/controllers/stats_controller.rb:40:in `block in range'
[2c6a71d3-e104-44b7-98ab-44263e879b22] app/controllers/stats_controller.rb:27:in `range'
[2c6a71d3-e104-44b7-98ab-44263e879b22] app/controllers/stats_controller.rb:55:in `index'

jywarren · 2020-10-20T21:29:21Z

https://stable.publiclab.org/stats?start=20-01-2013&end=20-01-2014 now works!

jywarren · 2020-10-20T21:30:28Z

https://publiclab.org/stats?start=20-01-2013&end=20-01-2014 now works too!

https://stable.publiclab.org/stats?start=20-01-2010&end=20-01-2020 ... loading it... it'll take a while...

jywarren · 2020-10-20T21:31:37Z

Yes, https://stable.publiclab.org/stats?start=20-01-2010&end=20-01-2020 now works too. Thanks, all - and should we make "all time" now really go back to 2010, @cesswairimu ?

#6050 - maybe we could modify this as an FTO and close this issue now?

cesswairimu · 2020-10-21T11:08:00Z

This is great 🎉 yeah that will be awesome. Creating an fto. Thanks Jeff.

cesswairimu · 2020-10-21T12:07:04Z

I've created an fto #8652.

I also think it would be a much faster to do .size on all data when getting 'all time' stats instead of using the where range clause (where timestamp btn 2010..NOW(). I will do a follow-up for that after the fto is done. Thanks everyone, closing this

jywarren · 2020-10-27T15:07:48Z

Awesome, thanks Cess!

skilfullycurled mentioned this issue Apr 17, 2019

Planning for expanded community stats system #3498

Closed

16 tasks

skilfullycurled mentioned this issue Apr 18, 2019

Stats Download And Site Overload #5524

Open

grvsachdeva added bug the issue is regarding one of our programs which faces problems when a certain task is executed help wanted requires help by anyone willing to contribute labels Apr 28, 2019

skilfullycurled closed this as completed Jun 18, 2019

skilfullycurled mentioned this issue Jun 18, 2019

Memory issues (leak?) investigation #5817

Closed

cesswairimu mentioned this issue Jul 25, 2019

Add all time stats option filter #6050

Merged

5 tasks

ebarry added this to the Metrics milestone Nov 16, 2019

jywarren reopened this Oct 20, 2020

cesswairimu closed this as completed Oct 21, 2020

cesswairimu mentioned this issue Oct 21, 2020

Count all-time stats without where clause #8654

Open

skilfullycurled mentioned this issue Jan 13, 2021

Stats page overload mitigation, passwording next steps discussion #9002

Closed

Stats downloading returns "Page does not exist" for dates prior to early 2013 #5490

Stats downloading returns "Page does not exist" for dates prior to early 2013 #5490

Comments

skilfullycurled commented Apr 15, 2019 • edited by jywarren Loading

cesswairimu commented Apr 17, 2019 • edited Loading

cesswairimu commented Apr 17, 2019

skilfullycurled commented Apr 17, 2019

skilfullycurled commented Apr 17, 2019

cesswairimu commented Apr 17, 2019

cesswairimu commented Apr 17, 2019

jywarren commented Apr 17, 2019

cesswairimu commented Apr 17, 2019

skilfullycurled commented Apr 17, 2019

skilfullycurled commented Apr 17, 2019

cesswairimu commented Apr 17, 2019 • edited Loading

skilfullycurled commented Apr 17, 2019

jywarren commented Apr 17, 2019 via email

skilfullycurled commented Apr 17, 2019

skilfullycurled commented Apr 18, 2019 • edited Loading

skilfullycurled commented Apr 18, 2019

skilfullycurled commented Apr 22, 2019

skilfullycurled commented May 12, 2019

jywarren commented May 14, 2019

sentry-io bot commented May 14, 2019

skilfullycurled commented May 14, 2019

skilfullycurled commented May 14, 2019

jywarren commented May 14, 2019

skilfullycurled commented May 14, 2019

skilfullycurled commented May 14, 2019

ebarry commented May 30, 2019 • edited Loading

jywarren commented Jun 18, 2019

jywarren commented Jun 18, 2019

jywarren commented Jun 18, 2019

jywarren commented Jun 18, 2019

jywarren commented Jun 18, 2019

skilfullycurled commented Jun 18, 2019

skilfullycurled commented Jun 18, 2019

jywarren commented Jun 18, 2019 via email

skilfullycurled commented Jun 18, 2019

jywarren commented Jun 18, 2019 via email

jywarren commented Oct 20, 2020

jywarren commented Oct 20, 2020

jywarren commented Oct 20, 2020 • edited Loading

jywarren commented Oct 20, 2020

jywarren commented Oct 20, 2020

jywarren commented Oct 20, 2020

jywarren commented Oct 20, 2020

jywarren commented Oct 20, 2020

jywarren commented Oct 20, 2020

cesswairimu commented Oct 21, 2020

cesswairimu commented Oct 21, 2020 • edited Loading

jywarren commented Oct 27, 2020

skilfullycurled commented Apr 15, 2019 •

edited by jywarren

Loading

cesswairimu commented Apr 17, 2019 •

edited

Loading

cesswairimu commented Apr 17, 2019 •

edited

Loading

skilfullycurled commented Apr 18, 2019 •

edited

Loading

ebarry commented May 30, 2019 •

edited

Loading

jywarren commented Oct 20, 2020 •

edited

Loading

cesswairimu commented Oct 21, 2020 •

edited

Loading