Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

311 Data Weekly Engineering Meeting #1720

Open
ryanfchase opened this issue May 9, 2024 · 25 comments
Open

311 Data Weekly Engineering Meeting #1720

ryanfchase opened this issue May 9, 2024 · 25 comments
Assignees
@ryanfchase @traycn
Labels
Feature: agenda Role: Frontend React front end work Size: 1pt Can be done in 6 hours
Milestone
03 - Project Mana...

Comments

@ryanfchase
Copy link
Member

ryanfchase commented May 9, 2024 •
edited
Loading

Overview

Meeting agendas for the weekly 311 Data Engineering member meetings, useful links and other resources

Agenda Quick Links

2024-05-08

Resources/Instructions

Engineering Onboard/Offboard Issue

Template for weekly meeting's comment.

## YYYY-MM-DD - 311 Data Weekly Engineering Meeting
_[Back to Top](#)_

### Quick Links
- [Open PRs](https://github.com/hackforla/311-data/pulls)
- [Ready For Dev Lead](https://github.com/hackforla/311-data/issues?q=is%3Aissue+is%3Aopen+label%3A%22ready+for+dev+lead%22+-label%3Adraft+label%3A%22Role%3A+Frontend%22)
- [Bug Hunt Live-Forever Ticket](https://github.com/hackforla/311-data/issues/1726)
- [Frontend Project Board](https://github.com/orgs/hackforla/projects/63/views/1?filterQuery=label%3A%22Role%3A+Frontend%22)
  - [Issues Ready To Be Worked On](https://github.com/orgs/hackforla/projects/63/views/1?filterQuery=label%3A%22Role%3A+Frontend%22+no%3Aassignee+status%3A%22Prioritized+Backlog%22)
  - [What's Being Worked On (Frontend)](https://github.com/orgs/hackforla/projects/63/views/1?filterQuery=has%3Aassignee+label%3A%22Role%3A+Frontend%22)
  - [Developer Self Test](https://docs.google.com/presentation/d/1RUQk4mOrv4G1fja_D3GMWMtCyCqsdj64fNPlrIDJhxg/edit#slide=id.g114a9db5138_0_0)
- [Github Pages Dev Site](https://hackforla.github.io/311-data/#/map)
- [Engineering Onboarding Issue](https://github.com/hackforla/311-data/issues/1578)
- [311 Roster](https://docs.google.com/spreadsheets/d/1OsLDl7Ciwj7WjtzbgDz38g4kpOMNlUhdjlgzQxtQTvM/edit#gid=0)

### Topic requests/questions:

### Action Items:

### Resources / Links:

### Attendees

- [ ] Tracy
- [ ] Ryan C
- [ ] Shannan
- [ ] Billy Phan
- [ ] Danielle
- [ ] Kelly
- [ ] Andrew
- [ ] Brandon
- [ ] Dorian
- [ ] Mrudula
@ryanfchase ryanfchase added this to the 03 - Project Management milestone May 9, 2024
@ryanfchase
Copy link
Member Author

ryanfchase commented May 9, 2024 •
edited
Loading

2024-05-08 - 311 Data Weekly Engineering Meeting

Back to Top

Weekly Announcement Volunteer

Quick Links

Topic requests/questions:

  • Weekly Engineering agenda
  • Dev team announcements person
  • Create open role on Engineering CoP
  • Bug-hunt ticket
  • Contact Form
  • Discuss new ticket goals

Action Items

  • brainstorm documentation issues (3 or 4 updates / new pages)

Resources / Links:

More info on today's Topic Requests 

1. Weekly Engineering agenda
  - we can briefly list all open tickets here, list PRs here, list goals here

2. Dev team announcements person: 
  - designated member to announce PRs, open tickets, etc (weekly rotation)
  - assign this person at the weekly meeting

3. Our plan to get new members:
  - create an Open Role on Engineering CoP
  - to justify this, we should make sure we have available tickets, and good onboarding + documentation

4. Bug-hunt & Code Quality tickets
  - a live forever ticket that gets reassigned every week, challenge to break the code
  - person assigned can do a bug writeup
  - can be a good first issue
  - Can document proposed solutions to these bugs and prioritize them for later

5. Contact Form
  - Create a Github access token for 311 Data dev

6. New Ticket Goals
  - try and maintain a number of Good First Issues (e.g. 3-5)
  - The number of available tickets should roughly match number of active devs

Attendees

  • Ryan C
  • Johnny
  • Shannan
  • Billy Phan
  • Drake

@ryanfchase
Copy link
Member Author

ryanfchase commented May 16, 2024 •
edited
Loading

2024-05-18 - 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

Quick Links

Topic requests/questions:

  • live forever bug reporting ticket

Action Items:

Resources / Links:

Attendees

  • Ryan C
  • Johnny
  • Shannan
  • Billy Phan
  • Drake

@ryanfchase
Copy link
Member Author

2024-05-22 - 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

Quick Links

Topic requests/questions:

Action Items:

  • review Engineering Documentation
    • CONTRIBUTING.md
    • README.md

Resources / Links:

Attendees

  • Ryan C
  • Johnny
  • Shannan
  • Billy Phan
  • Drake

@ryanfchase ryanfchase mentioned this issue May 23, 2024
Open
@ryanfchase
Copy link
Member Author

ryanfchase commented May 24, 2024 •
edited
Loading

Posting our meeting notes from Wednesday...

Notes on Readme

  • 311-Data v2 --> 311-Data v1.2
  • update the screenshot of the app
  • remove Matplotlib, Polars, Numpy
  • remove Google Colab
  • remove Observable (tbd)

Notes on contributing.md

  • if you run into issues, post onto #311-data-engineering, (not open new issue -- we dont have infrastructure for this)
  • feature branching format:
    [
    issueId-Title,
    issueId-MinimalDescription
    ]

TODO

  • prune branches on 311-Data
  • see if we want to implement any linting rules, PR rules for github actions
  • Getting started, it would be nice if there was a directory tree with main components, and some descriptions of what they do

@ryanfchase
Copy link
Member Author

ryanfchase commented Jun 4, 2024 •
edited
Loading

2024-06-12- 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

  • (replace with @ github handle)

Quick Links

Self-Intros

  • Returning members (1-3 minutes)
  1. If we have new members present, brief introduction about yourself.
  2. What have you completed since last Tuesday?
  3. What do you plan to complete before next Tuesday?
  4. If you have a blocker, leave a short summary in the chat so we can help you resolve it later in the meeting
  • New members (1-3 minutes)
  1. Where are you from?
  2. What are you looking to get out of volunteering with Hack for LA?
  3. What is your experience with coding / open source contribution?

Topic requests/questions:

Review Assigned Tickets

Questions from new members:

  • What is the primary purpose of the application
  • What's our roadmap for Dashboard related features
  • What documentation do we have for high-level data flow
  • What's our plan for frontend testing coverage
  • Where are user flows documented (fwd from Andrew's comment here). Answer: Figma Page. Do we need to update a base level of documentation for this?

RC, review with team

Action Items:

RC

I've moved these action items to the PM team agenda for tracking / updating PM issues

  • 311-Data bot by next week (OR at Bonnie 1 on 1 interview)
    • PMs come up with a human name (bot names might get flagged by github)
    • Admin makes a hackforla email address
    • PM team adds credentials to 1Password
    • PM team creates github account
    • Bonnie/Admin adds bot's github to HackForLA org
    • PMs to add team permissions to this bot
    • finish automation implementation, as outlined in this dev ticket
  • Scheduling a meeting with Bonnie for the Story of 311 Data page
    • use the Bonnie scheduling sheet to pick a time
    • pick one PM to conduct the interview

Resources / Links:

Attendees

  • Ryan C
  • Johnny
  • Shannan
  • Billy Phan
  • Drake
  • Kelly
  • Andrew
  • Danielle

This was referenced Jun 5, 2024
Open
Open
@ryanfchase ryanfchase mentioned this issue Jun 13, 2024
Open
3 tasks
@ryanfchase
Copy link
Member Author

ryanfchase commented Jun 15, 2024 •
edited by aqandrew
Loading

2024-06-19 - 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

  • (replace with @ github handle)

Quick Links

Topic requests/questions:

Action Items:

Resources / Links:

Attendees

  • Ryan C
  • Johnny
  • Shannan
  • Billy Phan
  • Drake
  • Danielle
  • Kelly
  • Andrew

@ryanfchase
Copy link
Member Author

ryanfchase commented Jun 26, 2024 •
edited
Loading

Leftover Action Items for @ryanfchase

Bugfix needed for popup being covered by address bar

image

@ryanfchase
Copy link
Member Author

ryanfchase commented Jun 26, 2024 •
edited
Loading

2024-06-26 - 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

  • (replace with @ github handle)

Quick Links

Accountability Checks

Details

Topic requests/questions:

Action Items:

Resources / Links:

Attendees

  • Ryan C
  • Johnny
  • Shannan
  • Billy Phan
  • Drake
  • Danielle
  • Kelly
  • Andrew

@ryanfchase
Copy link
Member Author

ryanfchase commented Jun 27, 2024 •
edited
Loading

RC TODO:

@ryanfchase
Copy link
Member Author

ryanfchase commented Aug 22, 2024 •
edited
Loading

2024-08-24 - 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

  • (replace with @ github handle)

Quick Links

Topic requests/questions:

  • Complete Onboarding
    • Brandon Yu
    • Dorian Deptuch

Documentation Items

  • Frontend
    • Mapbox (Billy)
    • React
    • MUI (Shannon?)
  • Backend
    • DuckDb
    • HuggingFace
    • python data transform + Github Actions
    • DbProvider + Context
    • Data Export Queries
    • Bot Email + Support Ticket Automation (Robleh)

Action Items:

Resources / Links:

Attendees

  • Ryan C
  • Johnny
  • Shannan
  • Billy Phan
  • Drake
  • Danielle
  • Kelly
  • Andrew
  • Brandon
  • Dorian
  • Mradula

@ryanfchase ryanfchase added the Size: 13pt Epic task (consider breaking into sub-tasks) label Aug 30, 2024
@ryanfchase
Copy link
Member Author

ryanfchase commented Sep 1, 2024 •
edited
Loading

Meeting Notes from 8/31/24

  • users thought calendar would expand after clicking the date
    • allow date dropdown to appear when clicking on date
    • clicking on "start date" should open month that contains "start date"
    • clicking on "end date" should open month that contains "end date"
    • OR clicking on anywhere on the text area emulates behavior of clicking on Calendar box
  • users want to be able to enter the date
    • this may require much more work from design+dev to cover all "error states"
    • might be simpler to move forward with clickable textbox, then consider typed-out-dates

@traycn traycn self-assigned this Sep 6, 2024
@ryanfchase
Copy link
Member Author

ryanfchase commented Sep 6, 2024 •
edited
Loading

2024-09-07- 311 Data Weekly Engineering Meeting

Back to Top

Quick Links

Topic requests/questions:

  • Tracy stepping into new dev lead role

Action Items:

Resources / Links:

Attendees

  • Ryan C
  • Johnny
  • Shannan
  • Billy Phan
  • Drake
  • Danielle
  • Kelly
  • Andrew
  • Brandon
  • Dorian
  • Tracy

@ryanfchase
Copy link
Member Author

ryanfchase commented Sep 7, 2024 •
edited
Loading

Borrowing from Product Planning new tickets comment.

New tickets checklist for eng:

Blocking ticket for #1708:

Other tickets

@ryanfchase
Copy link
Member Author

ryanfchase commented Sep 14, 2024 •
edited
Loading

2024-09-14 - 311 Data Weekly Engineering Meeting

Back to Top

AGENDA

Timezones listed as Pacific Time.

  • 10:00AM 5 min - Check-in
  • 10:05AM 5 min - Team announcements + updates
  • 10:10AM 20 min - Topic Requests
  • 10:30AM 30 min - Breakout Rooms
  • 11:00AM finished

Topic requests/questions:

RC

Action Items:

Resources / Links:

Attendees

  • Ryan C
  • Johnny
  • Shannan
  • Billy Phan
  • Drake
  • Danielle
  • Kelly
  • Andrew
  • Brandon
  • Dorian
  • Tracy
  • Mrudula

@ryanfchase ryanfchase removed Size: 13pt Epic task (consider breaking into sub-tasks) Size: Missing labels Sep 14, 2024
@ryanfchase ryanfchase added the Size: 1pt Can be done in 6 hours label Sep 14, 2024
@ryanfchase
Copy link
Member Author

ryanfchase commented Sep 14, 2024 •
edited
Loading

2024-09-14 Notes

  • Pair programming
    • rubber ducky discussion
    • co-working zoom and/or slack --> set one up for all of HackForLA
      • can we pilot this in #311-data-engineering ?
      • we'd need a way to set up a meeting link that doesn't expire / doesn't rely on host to be there
      • Slack huddles doesn't work because we have a Free account. It currently only supports 2 people in a huddle.
      • HackForLA discord server? easier to do co-working groups
  • unselected / updateUnselectedCouncils
    • this makes sense if we were trying to propagate a history of unselected councils
    • but sometimes we end up using the entire councils prop (see L270 in Map.jsx)
    • we should see if unselected used to be used for something useful, otherwise try to remove it
  • make this ticket an epic: Update dep styling @mui/styles with @mui/system #1820
    • consider doing a few major components first as an example
    • then start provisioning different issues to tackle groups of making this change
    • consider good first issue
    • consider good pair programming issues
      • TODO: make this a label
  • discussed project board automations
  • discussed Storybook

@ryanfchase
Copy link
Member Author

ryanfchase commented Sep 21, 2024 •
edited
Loading

2024-09-21 - 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

  • (replace with @ github handle)

Quick Links

Topic requests/questions:

  • Discuss Vite
    • consider merging this after "launch"
    • consider recreating our build process, compare to new vite process (use a simplified example)

Action Items:

Resources / Links:

Attendees

  • Tracy
  • Ryan C
  • Shannan
  • Billy Phan
  • Danielle
  • Kelly
  • Andrew
  • Brandon
  • Dorian
  • Mrudula

@ryanfchase ryanfchase mentioned this issue Sep 25, 2024
Open
1 task
@ryanfchase
Copy link
Member Author

ryanfchase commented Sep 28, 2024 •
edited
Loading

2024-09-28 - 311 Data Weekly Engineering Meeting

Back to Top

Quick Links

Topic requests/questions:

  • Review Wednesday meeting's agenda
    • discussed Contact Us page new tx upcoming (see R2.1)
  • Leetcode

Action Items:

Resources / Links:

  1. Links used in Leetcode
    1.1 How many bytes in a javascript string
    1.2 How many bytes in a javascript integer
  2. Discussing Contact Page
    2.1 What is PII

Attendees

  • Tracy
  • Ryan C
  • Shannan
  • Billy Phan
  • Danielle
  • Kelly
  • Andrew
  • Brandon
  • Dorian
  • Mrudula

@traycn
Copy link
Member

traycn commented Oct 5, 2024 •
edited by ryanfchase
Loading

2024-10-05 - 311 Data Weekly Engineering Meeting

Back to Top

Quick Links

Topic requests/questions:

Action Items:

Resources / Links:

Attendees

  • Tracy
  • Ryan C
  • Shannan
  • Billy Phan
  • Danielle
  • Kelly
  • Andrew
  • Brandon
  • Dorian
  • Mrudula

@ryanfchase
Copy link
Member Author

For next agenda

Need to discuss Vite build issue from VRMS (see if there is any overlap with our project)

@ryanfchase
Copy link
Member Author

2024-10-12 - 311 Data Weekly Engineering Meeting

Back to Top

Quick Links

Topic requests/questions:

Action Items:

Resources / Links:

Attendees

  • Tracy
  • Ryan C
  • Shannan
  • Billy Phan
  • Danielle
  • Kelly
  • Andrew
  • Brandon
  • Dorian
  • Mrudula

@ryanfchase
Copy link
Member Author

ryanfchase commented Oct 12, 2024 •
edited
Loading

2024-10-12 Extended Session (WIP)

Notes

In our meeting we covered...

Performance Profiling

  • @bphan002 and @ryanfchase attempted to use the Chrome DevTools profiler to record how much memory our application was incurring.
    • Ryan Profile 1: profile crashed, reported running out of memory, when rendering 3 months of data w/ all request types (8GB RAM, Chrome, while screen sharing)
    • Ryan Profile 2: profile experienced some slowdowns, but did not crash, rendering 3 months of data w/ 1 request types (same conditions)
    • Billy Profile 1: profile ran fine while rendering 3 months of data w/ 1 request type(16GB RAM, Chrome, needs confirmation)
    • Billy Profile 2: profile ran fine whiel rendering 3 months of data w/ 2 request types, however the application failed to respond when attempting a 3rd request type (bulky items, electronic waste, homeless encampment?)

After the meeting, continuing to look into profilers...

  • Ryan Profile 3: 3 months of data, one SR type at a time, up to 3 SRs
    • I was seeing 2GB - 2.5GB of heap size for the JS VM
    • Task Manager reported Chrome at 80~85% Memory usage
      • I could identify which Chrome sub-process was requiring 2 - 3GB of RAM
    • site did not crash, but I did experience some stutters and skipped inputs (also my music, playing in youtube, stopped for a brief moment)
    • I eventually enabled ALL SRs, application did not immediately crash.
      • it wasn't until I started zooming in/out on the map that I received this exception (see dropdown)
      • This exception doesn't necessarily indicate that we ran out of memory. Running out of memory looks different (see dropdown)
      • this.expression.evaluate is a method used while mapbox is determining how to render styles and layout of its data. I noticed that while I was profiling, the boundary fills stopped rendering, as if mapbox forgot to do it / skipped it. One possibility is that mapbox attempted to execute an expression but some resource or object was missing. Perhaps we're seeing a memory leak (probably more precisely, a memory management problem)
Profile 3: exception and stack trace

image

Out of memory icon

image

Memory Profiler: Looking at Memory Snapshots rendering 3 months of data w/ ALL service request types:

NOTE: I'm actually not sure if I'm interpreting these tables correctly. Will need to continue looking into them.

Object: Possible duplication of memory

These are listed under `Object`, I assume that means these are just JS objects living in memory. All other entries in thist list were 2MB or less

SR Requests being stored in React: ~45MB
image

SR Requests being stored in Mapbox: ~45MB
image

Top 6 Biggest Object data structures
image

MapContainer

MapContainer: ~366MB, not sure if there's much to do here. Unsure of how to slim this, or if it needs to be slimmed.
image

Action Items

  • How much data gets loaded on page-load?
    • pretty sure we just go to hugging face and look at how much data is in the 2024 data set: ~1.12MB for 2024
    • But is this what's causing the application to crash?
      • Guess 1: No.
        • It takes a long time to load the data and render the SRs, but this does not inherently crash the site
        • site will crash as soon as you start zooming in and out
        • we observe that there are MANY api calls to mapbox that occur as you zoom in/out of the map. Some are fetching tilesets, some are api calls to events

Resources

@ryanfchase
Copy link
Member Author

ryanfchase commented Oct 19, 2024 •
edited
Loading

2024-10-19 - 311 Data Weekly Engineering Meeting

Back to Top

Quick Links

Topic requests/questions:

Action Items:

Resources / Links:

Attendees

  • Tracy
  • Ryan C
  • Shannan
  • Billy Phan
  • Danielle
  • Kelly
  • Andrew
  • Brandon
  • Dorian
  • Mrudula

@ryanfchase
Copy link
Member Author

ryanfchase commented Oct 19, 2024 •
edited
Loading

Notes from 10/19

@ryanfchase
Copy link
Member Author

ryanfchase commented Oct 20, 2024 •
edited
Loading

Performance Notes (Cont)

I decided to try and see exactly how we were pulling 2023 data (since this isn't pulled on page-load). I noticed that it was actually only capable of pulling data from hugging face in 16.8MB sized chunks. See picture. This suggests that if we are to split up the data into manageable chunks, that 16.8MB is sort of the high end of resource size.

I noticed there was 4.2MB in the first chunk, proceeded by seven 16.8MB chunks, and then 8.3MB in the last chunk. That's about 130MB of data. And if you go to our Huggingface repo, you'll see that the 2023.parquet is 135MB large, so that's pretty close to what I expected.

2023 parquet being organized into chunks

image

@ryanfchase
Copy link
Member Author

ryanfchase commented Nov 9, 2024 •
edited
Loading

2024-11-09 - 311 Data Weekly Engineering Meeting

Back to Top

Quick Links

Topic requests/questions:

  • 311 Ryan
    • Recreating our map
    • Recreating our data fetching system
  • community resource sharing

Action Items:

Resources / Links:

Attendees

  • Tracy
  • Ryan C
  • Shannan
  • Billy Phan
  • Danielle
  • Kelly
  • Andrew
  • Brandon
  • Dorian
  • Mrudula

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ryanfchase ryanfchase

@traycn traycn

Labels
Feature: agenda Role: Frontend React front end work Size: 1pt Can be done in 6 hours
Projects
P: 311: Project Board
Status: Start Here
Milestone
03 - Project Management
Development

No branches or pull requests
4 participants
@ryanfchase @traycn @Skydodle @bberhane