class: center, middle
Helge Pfeiffer, Associate Professor,
Research Center for Government IT,
IT University of Copenhagen, Denmark
[email protected]
class: center, middle
- While working, take notes of what you are doing.
- A text file in your workspace might serve you well as a notebook.
- Keep a record of when you did what.
- Note down what went wrong, where you found a solution, and keep a link for that.
- Commit these notes to your repository. One DevOps principle is to Make Work Visible.
Are you done grouping or are there students not having a group yet?
The task description for this week contained the following:
Please share your repositories with us. Send a GitHub pull-request on the file
repositories.pyin which you replace, e.g.,https://github.com/<gh_id>/<proj_id>with the repository URL of your group. In case you use more than one repository, please add them to the corresponding list too.
But did all groups sent a respective pull request linking their repositories? Please send them today!
Boiled down:
- BSc students: In their project work they can do and choose whatever they like.
- MSc students: In their project work they have to be more reflective and argue for why certain choices are made, see e.g., today's task description.
- What did you have to change?
- Which files did you modify and how?
- Did you create a release with your refactored solution?
- Did you send a pull request to repositories.py?
When refactoring control.sh, which shebang did you set?
#!/usr/bin/env bash#!/bin/bash#!/bin/sh#!/bin/sh -
See
- https://stackoverflow.com/questions/10376206/what-is-the-preferred-bash-shebang
- https://en.wikipedia.org/wiki/Shebang_%28Unix%29
class: center, middle
- How to move artifacts around?
- How to package and distribute software?
- How to refactor our projects?
Software artifact: A software artifact is a tangible machine-readable document created during software development. Examples are requirement specification documents, design documents, source code and executables.
We transfered the ITU-MiniTwit application from the server to your computers via scp.
--
cd itu-minitwit
git init
git add .
git commit -m"Code as resurected from server"
git remote add origin [email protected]:<your_gh_id>/itu-minitwit.git
git push -u origin master
git tag v0.1.0
git push origin v0.1.0Contents of a repository with lightweight tags (see https://git-scm.com/book/en/v2/Git-Basics-Tagging) can be retrieved as zipped archives from GitHub.
That could be used to package and distribute an application.
wget https://github.com/<your_gh_id>/itu-minitwit/archive/refs/tags/v0.1.0.zip
unzip v0.1.0.zip
--
- When doing that, do we transfer our entire application?
- What were the steps that you executed during your refactoring homework?
--
We are missing quite some dependencies of our application when distributing it via scp directly or as a zipped archive.
-
Map the artifacts and all dependencies of the ITU-MiniTwit (the Python 2 version) that you took over last week.
-
That is, start with
minitwit.py,flag_tool.c, andcontrol.shand map all of their respective dependencies. -
Draw a dependency graph for all artifacts, and discuss with your group fellows if you have covered all artifacts.
-
Once you have a dependency graph, paste an image of it to our Teams chat.
-
You might want to use GraphViz (http://www.webgraphviz.com/) to draw your graph quickly and declared as code.
-
Dependency graph? See for example https://en.wikipedia.org/wiki/Dependency_graph
- How does it really look like?
- On Ubuntu Trusty (14.04)
We are standing on the shoulders of giants. Seemingly simple applications are facilitated by a plethora of software that is needed to build/compile our application, to run it, to test it, etc.
--
- Where do all these artifacts come from?
- Who knows about all of them?
--
- When developing how do you get the required dependencies?
- In production, i.e., after deployment on a (potentially other) machine how do you get the required dependencies?
--
- Who is responsible for setting up/configuring all the dependencies?
- How often do you have to do this?
We could build packages for the respective programming language. In this case Python packages, which can be shared and installed either directly or via a central index like PyPI or a self-hosted alternative.
--
- What is the advantage of that?
- What might be a drawback?
We could build packages for the respective Linux distribution.
In our case this would be .deb packages, which would be handled via apt on Debian-based distributions.
Linux packages can be shared and installed either directly or via a central repository (might be self-hosted), see for example, /etc/apt/sources.list to figure out to which repositories you are using.
--
- What is the advantage of that?
- What might be a drawback?
If you want deeper information on and examples of how to build Python and Debian packages respectively, study for example chapter 5 from Python for DevOps :
We could build application containers for the respective container engine. We will study Docker in the remainder of this session. Docker containers can be instantiated from container images, which are distributed via a central registry (DockerHub, GitHub Container Registry, etc.).
--
- What is the advantage of that?
- What might be a drawback?
When containerizing an application, we package all runtime dependencies and we provide a setup that allows to run an application without interfering with its environment.
--
What does that mean?
Run the following command in your terminal and wait for the download to finish (will likely take long in lecture rooms at ITU).
docker run -d -p5000:5000 helgecph/itu_minitwit_trusty:v0.1.0Once the download completed, navigate your web-browser to http://localhost:5000/ and appreciate that you received a running instance of our ITU-MiniTwit application with a single command.
We study Docker in this session. However, it is not the only tool to containerize applications, it only appears to be the most popular at the moment.
Alternative tools are:
In this lecture, we focus on applying Docker. That is, how to use them to package and distribute applications. To get an introduction to how these technologies work, ask in the operating systems course or check the links below:
- Liz Rice's talk builds a basic container engine in Go: https://www.youtube.com/watch?v=HPuvDm8IC-4
- https://github.com/janoszen/demo-container-runtime
- https://github.com/p8952/bocker/blob/master/README.md
- https://www.infoq.com/articles/build-a-container-golang/
- https://github.com/Zakaria-Ben/Pocker
Recreate the Docker examples from today's exercises https://github.com/itu-devops/flask-minitwit-mongodb/tree/Containerize, to work with any of the follow alternative container engines:
A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings.
...
Containers isolate software from its surroundings, for example differences between development and staging environments and help reduce conflicts between teams running different software on the same infrastructure. (https://www.docker.com/what-container)
Source: https://www.docker.com/resources/what-container/
class: center, middle
$ docker run --rm hello-worldThe command above downloaded the image hello-world from the Docker Hub, instantiated a container from that image, ran the application within this container, and finally deleted the container (--rm).
$ docker run --rm hello-worldUnable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:aa0cc8055b82dc2509bed2e19b275c8f463506616377219d9642221ab53cf9fe
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
Check on Docker Hub, there are images for many different flavors of Linux and for many packaged applications.
$ docker run -it --rm alpine:latest sh--
What does that do? It tells Docker to run a container with the latest version of Alpine Linux (a small Linux Distribution), connect to the shell process sh, run it interactively (-it) so that you can type in commands and see the results, and finally, to remove the container (--rm) after the sh process finishes.
You can mount directories (volumes) from your host to a container using the -v flag.
$ docker run -it -v $(pwd):/host alpine:latest /bin/shLet's build a simple webserver in Go. To not mess with our development machine we could use a Docker container, which has the Go compiler readily installed.
Find the following basic_http_server.go file in the directory ./webserver.
package main
import (
"fmt"
"log"
"net/http"
)
func helloWorldHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, "Hej verden!\n")
}
func main() {
port := 8080
http.HandleFunc("/", helloWorldHandler)
log.Printf("Server starting on port %v\n", port)
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%v", port), nil))
}To containerize that program, or better to run that program in a container without installing the compiler to our machines directly, you could run:
$ docker run -it --rm \
--name myserver \
-v $(pwd)/webserver:/src \
-p 8080:8080 \
-w /src \
golang:jessie go run basic_http_server.go--
That command:
- instantiates a container of the image
golang:jessie(an Ubuntu Linux with Go and some other necessary tools readily installed) - shares our local code in
./webserverwith the container, where it is mounted to the/srcdirectory - changes the current working directory in that container to
/src(-w) - binds the port
8080from the container to the same port number on our host (-p 8080:8080)- The port number in front of
:specifies the port on the host, which gets bound to the port of the container (number after:).
- The port number in front of
- builds and runs the program within the container
go run basic_http_server.go
Note, you could also build the program in the container and run the resulting binary.
$ docker run -it --rm \
--name myserver \
-v $(pwd)/webserver:/src \
-p 8080:8080 \
-w /src \
golang:jessie bash -c "go build basic_http_server.go; ./basic_http_server"Now, you can access the webserver on your host machine on http://127.0.0.1:8080. When you point your browser to that address you should see the following:
Alternatively, you could run curl on your host machine to see that our server is working correctly.
$ curl -s http://127.0.0.1:8080Unfortunately, many operating systems do not come with the curl program installed.
But there is a dockerized version of this program.
When you run:
$ docker run --rm \
--link myserver \
appropriate/curl:latest curl -s http://myserver:8080
Hej verden!then you download an image with a small Linux and with curl installed. However, the command above also allows the curl client to see our webserver --link myserver. Try to run the command above without that flag and see what happens.
--
To find more Docker images and dockerized programs have a look at https://hub.docker.com.
Dockerfiles describe a precise configuration of a container.
These configurations are stored as slices on top of each other.
Let's have a look on an example application.
It will consist of a webserver and of a simple client.
The webserver serves a static HTTP message on port 8080 and the client is just an HTTP GET query receiving this message via curl.
The following UML deployment diagram illustrates this setup:
Let's have a look at the Dockerfile that specifies our webserver.
FROM golang:jessie
# Install any needed dependencies...
# RUN go get ...
# Set the working directory
WORKDIR /src
# Copy the server code into the container
COPY basic_http_server.go /src/basic_http_server.go
# Make port 8080 available to the host
EXPOSE 8080
# Build and run the server when the container is started
RUN go build /src/basic_http_server.go
ENTRYPOINT ./basic_http_serverAs you can see from the above configuration, the Dockerfile is similar to everything described in our earlier CLI command:
$ docker run -it --rm \
-v $(pwd)/webserver:/src \
-p 8080:8080 \
-w /src \
golang:jessie bash -c "go build basic_http_server.go; ./basic_http_server"Keywords in Dockerfiles are FROM, MAINTAINER, LABEL, RUN, CMD, EXPOSE, ENV, ADD or COPY, ENTRYPOINT, VOLUME, USER, WORKDIR, ONBUILD. You can read more on them in the documentation: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/.
To use containers with our webserver, we first have to build a corresponding image. If you have the above Dockerfile stored in a directory webserver you can do so as in the following:
$ cd webserver
$ docker build -t <your_id>/myserver .--
The -t flag tells Docker to build an image with the given name <your_id>/myserver. The . says: build the image with the Dockerfile in this directory.
After building your image, you can verify that it is now accessible on your machine.
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
your_id/myserver latest a5fe35de13d2 8 seconds ago 704MB
appropriate/curl latest f73fee23ac74 3 weeks ago 5.35MB
golang jessie 6ce094895555 4 weeks ago 699MB$ docker run --name webserver -p 8080:8080 <your_id>/myserver$ docker stop webserver$ docker start webserverFROM appropriate/curl:latest
ENTRYPOINT curl -s http://webserver:8080$ cd client
$ docker build -t <your_id>/myclient .$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
<your_id>/myclient latest 3714e67fa75a 4 seconds ago 5.35MB
<your_id>/webserver latest a5fe35de13d2 About an hour ago 704MB
appropriate/curl latest f73fee23ac74 3 weeks ago 5.35MB
golang jessie 6ce094895555 4 weeks ago 699MB$ docker run --name client --link webserver <your_id>/myclient
Hej verden!--
That is nice, is not it? We just built a small application consisting of a webserver and a client, both deployed in their own containers, and we did not have to install any dependencies on our host machine manually.
However, starting the server and the client by hand with the docker run ... command is quite tedious.
Furthermore, it is not really in line with the infrastructure as code paradigm.
Therefore, we can automate even further using docker compose.
To describe an application of interconnected services, we can use a docker-compose.yml file.
It specifies the components of an application, how they shall be started, and how they interact.
--
version: '3'
services:
webserver:
image: your_id/myserver
ports:
- "8080:8080"
clidownload:
image: appropriate/curl
links:
- webserver
entrypoint: sh -c "sleep 5 && curl -s http://webserver:8080"(Note, replace your_id with your name ID before running docker compose.)
$ docker compose up
[+] Running 2/0
✔ Container session_02-webserver-1 Created 0.0s
✔ Container session_02-clidownload-1 Created 0.0s
Attaching to session_02-clidownload-1, session_02-webserver-1
session_02-webserver-1 | 2024/02/08 10:01:37 Server starting on port 8080
session_02-clidownload-1 | Hej verden!
session_02-clidownload-1 exited with code 0
^CGracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
[+] Stopping 2/2
✔ Container session_02-clidownload-1 Stopped 0.0s
✔ Container session_02-webserver-1 Stopped 10.2s
canceled--
Finally, to clean up:
$ docker compose rm -v- To prepare for your project work, practice with the exercises
- Do the project work until the end of the week
- And prepare for the next session
Strategies for evolution in order of recommendation:
- Replace
- Refactor
- Rewrite
--
See e.g., Re-Engineering Legacy Software :
For our course project replacement with a standard system is not an option, since we need ITU-MiniTwit as a vehicle for learning and you should feel complete ownership for it. In reality, this is likely the option you want to check first: Is there a ready off-the-shelf solution that covers our requirements? If yes go for it, because it likely decreases your maintenance burden.
--
Rewriting software completely is risky and takes long. In reality, this should rarely be done since legacy systems implement a lot of domain-knowledge and implicit business rules, which are very hard to unearth only from implementation and documentation.
Do that only after you demonstrated that you cannot refactor.
You will be doing this in your project work. Do it systematically. That is, implement one feature after another and run the test suite continuously. Start with implementing an endpoint/view for public timelines. Reuse the existing SQLite 3 database and schema.
That is, since you are not doing a big rewrite, you cannot re-architect the ITU-MiniTwit project yet. Instead, try to implement a one-to-one "copy" of the current ITU-MiniTwit Python/Flask application.
Consequently, you will likely build your application in one file, with one function/method per endpoint
For that you have to find a suitable micro-web-framework for your language of choice and a suitable template engine.
You continue to build a server-side rendered application. That is, for now, do not use fancy front-end technology like React, Blazor, etc.
Example: Refactoring to ASP.NET with Scriban templates
/// <summary>
/// Shows a user's timeline.
/// This timeline shows the user's messages as well as all the messages of followed users.
/// If no user is logged in it will redirect to the public timeline.
/// </summary>
IResult timeline(HttpRequest request, HttpContext context)
{
var userIDFromSession = context.Session.GetString("user_id");
if (userIDFromSession == null)
return Results.Redirect("/public");
var query = @"select message.*, user.* from message, user
where message.author_id = user.user_id and (
user.user_id = ($userID) or
user.user_id in (select whom_id from follower
where who_id = ($userID)))
order by message.pub_date desc limit ($perPage)";
var parameters = new Dictionary<string, object>()
{
["$userID"] = userIDFromSession,
["$perPage"] = perPage,
};
var data = new Dictionary<string, object>()
{
["cheeps"] = queryDB(query, parameters),
...
};
return Results.Extensions.Html(renderTemplate(ChirpTemplates.Timeline, data));
};public static class ChirpTemplates
{
private const string top = @"<!DOCTYPE html>
<html>
<head>
<title>Chirp!</title>
<link rel=""stylesheet"" type=""text/css"" href=""/css/style.css"">
<link rel=""icon"" type=""image/x-icon"" href=""/icon/favicon.ico"">
</head>
<body>
<div class=""page"">
<div>
<h1><img src=""images/icon1.png""/>Chirp!</h1>
</div>
<div class=""navigation"">
{{ if isLoggedIn }}
<a href=""/"">my timeline</a> |
<a href=""/public"">public timeline</a> |
<a href=""/logout"">sign out [{{-profileUsername-}}]</a>
{{ else }}
<a href=""/public"">public timeline</a> |
<a href=""/register"">sign up</a> |
<a href=""/login"">sign in</a>
{{ end }}
</div>
{{ if message != """" }}
<div class=message>{{ message }}</div>
{{ end }}
{{ if errorMessage != """" }}
<div class=error><strong>Error:</strong> {{ errorMessage }}</div>
{{ end }}
<div class=""body"">
";
...
}To implement a one-to-one clone one could - amongst others - choose:
- Java Spark with jinjava
- C♯ ASP.Net minimal web application with Scriban
- C♯ ASP.Net minimal web application with Razor Pages
- Ruby with Sinatra
- Crystal with Kemal
- Python with Pyramid or Bottle and Jinja2
- Go with only the standard library or with Gorilla
- Nim with Jester
- Elixir with Phoenix ...
Do not choose full-webframeworks like Django, Blazor, Spring MVC, Ruby on Rails. They all would require a big rewrite.
- To prepare for your project work, practice with the exercises
- Do the project work until the end of the week
- And prepare for the next session