Guide for Developers

0. Development Environment Setup

Please follow the Getting Started Guide to install the required packages and obtain the codebase.

1. Setup Backend Development.

Import the Amber project into IntelliJ

Before you import the project, you need to have "Scala", and "SBT Executor" plugins installed in Intellij.

In Intellij, open File -> New -> Project From Existing Source, then choose the amber folder inside core.
In the next window, select Import Project from external model, then select sbt.
In the next window, make sure Project JDK is set. Click OK.
IntelliJ should import and build this Scala project. Wait until the indexing and importing is completed.
When IntelliJ prompts "Scalafmt configuration detected in this project" in the bottom right corner, select "Use scalafmt formatted". If you missed the IntelliJ prompt, you can check the Event Log on the bottom right

Your PR should pass scalafix check (lint) and scalafmt check.

To check lint, under core/amber run command sbt "scalafixAll --check"; to fix lint issues, run sbt scalafixAll.
To check format, under core/amber run command sbt scalafmtCheckAll; to fix format, run sbt scalafmtAll.
When you need to execute both, scalafmt is supposed to be executed after scalafix.

Run the backend engine in IntelliJ

The easiest way to run the webserver is in IntelliJ. Amber is a distributed engine. We need to run the controller process, and at least one worker process.

To run the controller process, go to src/main/scala/edu/uci/ics/texera/web/TexeraWebApplication.scala, then run the main function by pressing on the green run button.

Wait for the process to start up. If you can find the following messages, then the controller process is successfully started.

[main] [akka.remote.Remoting] Remoting now listens on addresses:
org.eclipse.jetty.server.Server: Started

If IntelliJ displays CreateProcess error=206, The filename or extension is too long : add the -Didea.dynamic.classpath=true in Help | Edit Custom VM Options and restart the IDE

To run the worker process, go to src/main/scala/edu/uci/ics/texera/web/TexeraRunWorker.scala, then run the main function. Wait for it to connect to the same cluster. If you see the following prompt, then it means you have successfully started the Texera server based on the Amber engine.

---------Now we have 1 node in the cluster---------

Run the backend engine in the command line (optional)

An alternative to run the backend engine is to run it in the command line. Navigate to core folder in terminal window, run scripts/server.sh, and then open a separate tab and run scripts/worker.sh.

Testing the backend

The test framework is scalatest, tests are located under core/amber/src/test. You can find unit tests and e2e tests.
To execute it, navigate to core/amber directory in the command line and execute sbt test.
If using IntelliJ to execute the test cases please make sure to change the working directory to core/amber.

2. Develop Frontend

This is for developers that work on the frontend part of the project. This step is NOT needed if you develop the backend only.

Install Angular CLI

Recommend using nodejs@18 LTS and required to use [email protected]. Install yarn:

npm install -g yarn
corepack enable && corepack prepare [email protected] --activate && yarn --cwd core/gui set version 4.5.1

You need to install Angular CLI to build and run the new GUI.

yarn global add @angular/cli
yarn install

Ignore those warnings (warnings are usually marked in yellow color or start with WARN).

Develop Frontend in IntelliJ

In Intellij, open File -> Open, then choose the gui folder inside core.
IntelliJ should import this project. Wait until the indexing and importing is completed.
Click on the Green Run button next to the Angular CLI Server.
Wait for some time and the server will get started. Open a browser and access http://localhost:4200. You should see the Texera UI with a canvas.

Every time you save the changes to the frontend code, the browser will automatically refresh to show the latest UI.

Testing the frontend

Before merging your code to the master branch, you need to pass the existing unit tests first.

Open a command line. Navigate to the core/gui directory.
Start the test:

ng test --watch=false

Wait for some time and the test will get started. You should also write some unit tests to cover your code. When others need to change your code, they will have to pass these unit tests so that you can keep your features safe. The unit tests should be written inside .spec.ts file.

Deploy the frontend to the production environment

Run the following command

yarn run build

This command will optimize the frontend code to make it run faster. This step will take a while. After that, start the backend engine in IntelliJ and use your browser to access http://localhost:8080.

Formatting frontend code

Run the following command

yarn format:fix

This command will fix the formatting of the frontend code.

3. Python UDF

Install [email protected], it's recommended to create a virtualenv to get a new copy of Python. Note: if you are using Apple's new M1 chip, please install Python through Anaconda.
Obtain Python executable path, for example, run which python or where python, and copy the returned path.
Fill the Python executable path into core/amber/src/main/resources/udf.conf, under the path key.
Install dependencies: pip install -r core/amber/requirements.txt -r core/amber/operator-requirements.txt -r core/amber/r-requirements.txt.
For formatting python files: black core/amber/src/main/python.

4. User Dashboard (Optional)

Install and use MySQL@8+ and create texera_db with core/scripts/sql/texera_ddl.sql.
Edit core/gui/src/environments/environment.default.ts, change userSystemEnabled to true.
Edit core/amber/src/main/resources/application.conf
1. Change user-sys.enabled to true.
2. Change jdbc.username and jdbc.password to your MySQL user and password to access texera_db that you just created.
Optional: Add googleClientId to the same file to enable google login.
Restart frontend and backend. You should see the homepage. You can register or login there.

5. Email Notification (Optional)

Set clientId and clientSecret in core/amber/src/main/resources/application.conf.
Login to Texera with an admin account.
Open the Gmail dashboard under the admin tab.
Authorize a Google account for sending emails.
Send a test email.

6. Misc (Optional)

This part is optional, you only need to do this if you are working on a specific task.

To enable MongoDB storage for sink

Install MongoDB on your development environment(4.4 works for Ubuntu, 5.0 works for Mac/Windows).
Start MongoDB with the default configuration.
Edit core/amber/src/main/resources/application.conf, change storage.mode to "mongodb".
Start Texera, all the results of sink operators will be saved into MongoDB if applicable.

To create a new database table and write queries using Java through Jooq

Create the needed new table in MySQL and update core/scripts/sql/texera_ddl.sql to include the new table.
Run core/util/src/main/java/edu/uci/ics/util/RunCodegen.java to generate the classes for the new table.
Create a helper class under core/amber/src/main/scala/edu/uci/ics/texera/web/resource/dashboard.

Note: Jooq creates DAO for simple operations if the requested SQL query is complex, then the developer can use the generated Table classes to implement the operation

Disable password login

Edit core/gui/src/environments/environment.default.ts, change localLogin to false.

Enforce invite only

Edit core/gui/src/environments/environment.default.ts, change inviteOnly to true.

Backend endpoints Role Annotation

There are two types of permissions for the backend endpoints:

@RolesAllowed(Array("Role"))
@PermitAll Please don't leave the permission setting blank. If the permission is missing for an endpoint, it will be @PermitAll by default.