Planning Issue: Expand, Coordinate, and Test API

[david-days]

Goals

GSoC 2016 introduced RESTful services to the plots2 system. Since then, there have been several fundamental changes to the site, as well as new projects and goals for GSoC 2017. This issue is intended to cover cleanup of work from 2016, as well as add new capabilities for further development.

Additionally, the existence and usage of the RESTful services (current and future) is rather opaque, at the moment. Developers do not have a centralized point of reference in which to become familiar with existing code. Because the RESTful subsystem is based on a particular technology, consumers and developers of the services also have to surmount a rather steep learning curve before any practical usage.

---

Phase 1: Documentation and Testing UI

Addition of a test and documentation interface for developers and consumers of existing RESTful services.

• Documentation of the web services
• Testing interface for potential consumers of RESTful services, as well as developer testing before deployment/upgrade

Phase 2: Full Coverage of Current Services

Full coverage of the Jasmine Tests for existing services and JS scripts, such as Issue #812.

• Full test coverage of existing services
• Supporting documentation, design patterns, and helper classes for future tests

Phase 3: User Authentication

This phase covers integration of RESTful services with plots2 user authentication.

The current implementations are read-only services that are deemed "safe", in that they do not require various levels of user authentication to separate anonymous calls from standard users from administrators. Before the development of more powerful RESTful services can occur, an authentication system must be designed and implemented. While there are standard patterns available, actual implementation of any service requires tight integration and testing with the specific authentication/authorization implementation.

Once this phase is complete, future developers should have a standardized approach (via helper classes, system configurations, or design patterns) to apply restrictions to unsafe web service calls that will developed in the future.

• Standardized login and user role mechanism to be applied to protected services
• Role-level access controls for RESTful service calls
• Testing for this system
• Standardized test patterns and test classes for additional services.

Phase 4: Additional RESTful Services

This phase is more generalized. As various projects move forward, the opportunity to add new services will occur, and decisions need to be made for both the location and capabilities of those services. As each of the various projects are evaluated, the following checklist of items should be considered:

• List potentially useful RESTful services that can be created, with a focus on aiding current and future usage of the project's data and capabilities
  • What data should be made available?
  • What project-specific computations and logic should be made available?
  • With respect to the previous items, what supporting data structures should be created or modified?
• Evaluate locations and access controls for the services
  • Which services are "safe" (read-only and publicly available, not requiring access controls for a logged in user?)
  • Which services require access controls?
  • Should the services be coded within the sub-project, or within the plots2 umbrella project
  • Required testing and evaluation for the same
• Documentation and Testing interface for non-RESTful consumers. Options are below:
  • Use the primary plots2 system from Phase 1
  • Use a similiar system via clone or helper classes
  • Use a unique UI designed around the specific project's logic and goals.

---

As I said, this is a generalized issue to try to bolster the current RESTful API system, as well as roll out new capabilities in a standardized manner. Phase 4 (Additional RESTful Services) may occur with no new services from GSoC 2017. However, the patterns and work from the previous phases should establish a baseline for any future work.

[david-days]

@jywarren, I'm already working on the Phase 1 part, and should be submitting that soon. Fortunately, the work that @ujithaperera and I did last year already laid the foundation. Once that part is finished, I'll be able to move fairly quickly through the next 2 phases. My overall goal is to work with @StlMaris123 in particular on the email subsystem, while keeping an eye on the rest of the work from this year and last year to apply for Phase 4.

Comments and criticism are always welcome.

[ujithaperera]

hi @david-days , thank you very much for re organizing to make this RESTful services to bring valuable end. I really appreciate it. give me some time to catch up the things you are planing ahead. I am still finding a place to begin my contribution. Thank you.

[ananyo2012]

Hey guys I am in for help too! Since the Bot Project would need the API I would really want to be a part of this. I will look in how the API is set up and Swagger documentation.

[jywarren]

This is so great. The only request I'd make is that we attempt to work as much as possible in small PRs which we merge to master in discrete, tested chunks. Each PR should be small enough that reviewers can read through it, understand the single function or improvement it adds, and see how the tests confirm this. That'll enable us to closely track the master branch. David, do you think that's feasible?

…

On Tue, Jun 6, 2017 at 2:04 PM, Ananyo Maiti ***@***.***> wrote: Hey guys I am in for help too! Since the Bot Project would need the API I would really want to be a part of this. I will look in how the API is set up and Swagger documentation. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1449 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ6UUkP-yozH015BRu5b1iLwRXzF3ks5sBZSqgaJpZM4NvKA1> .

[david-days]

I think it's feasible and desirable. I've got a plan (subject to better ideas, changes, or just "huh...never thought of that") that I can talk through.

I already have a PR (Pull Request #1450 ) for adding the Swagger front end. It already shows the documentation from the work that Ujitha and I did last year. This would go a long way towards getting everyone on the same page.

For some reason, it's failing Travis CI on the DB setup--i didn't change anything there, but I did a rebase/upstream-merge and might have screwed that up somehow.

[jywarren]

Great, i left some comments over there and look forward to this!

[jywarren]

Also be aware that Jasmine cannot test the Rails responses, the way it's set up (and the convention) is that we decide what the API boundary between the front and backends are, and we use static or mocked responses to test each of the two, rather than testing both in an integrated way. This makes sense because it holds both front and backend to the strict promised API by testing each against a static version. It also ensures 2 things aren't changing at the same time and we're missing something between.

[jywarren]

Hi, David! After #1450, I'd like to propose starting with unit tests for SearchService and TypeaheadService:

• https://github.com/publiclab/plots2/blob/master/app/services/search_service.rb
• https://github.com/publiclab/plots2/blob/master/app/services/typeahead_service.rb

Not every sub-method, but the public ones, like:

• textSearch_all
• textSearch_tags
• textSearch_notes
• etc

Since these make use of the other methods (like find_notes and notes), that should give us full coverage. Actually the remainder could become private and not need to be tested.

Later, we can set up functional tests for:

• https://github.com/publiclab/plots2/blob/master/app/api/srch/typeahead.rb
• https://github.com/publiclab/plots2/blob/master/app/api/srch/search.rb

How does that sound? Once unit tests are in place for SearchService and TypeaheadService, we can work on refining them as described in the TODOs in the comments on each one, without worrying about messing up their functionality (the tests will ensure this for us).

[david-days]

@jywarren ,

Sounds like exactly what we need. I'm going to try to make the changes to #1450 tonight, and then work on the test coverage (also updating comments that will appear in the Swagger UI, add references to the implemented JS clients, etc).

Sounds like we have a plan moving forward.

[jywarren]

Awesome -- as much as possible, let's do this in separate PRs! Good luck -- i'm hammering away at Solr config, but making progress!

…

On Thu, Jun 15, 2017 at 4:42 PM, David C Days ***@***.***> wrote: @jywarren <https://github.com/jywarren> , Sounds like exactly what we need. I'm going to try to make the changes to #1450 <#1450> tonight, and then work on the test coverage (also updating comments that will appear in the Swagger UI, add references to the implemented JS clients, etc). Sounds like we have a plan moving forward. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1449 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ362n9ofgFe_bXCBWH5lugPl7Apgks5sEZcxgaJpZM4NvKA1> .

[jywarren]

OK, - this is still a great plan for the API work, though we'll need to update and adapt some sections due to recent work. I also put more links to existing API methods here: https://github.com/publiclab/plots2/pull/2017/files

[jywarren]

Also need updates regarding the token system #1513 for the write-API

[stefannibrasil]

the API has changed a lot, more updated details at #3520