Week 1: Introduction

Welcome to Computer Graphics! This course serves as an introduction to the area of graphics. We will learn many of the basics of modeling and rendering from a modern OpenGL approach. This course will feature lots of programming, and lots of math, but the math is not super complex. We will design many interesting projects, and you will have a chance to explore your own final project. If you are expecting to design a full fledged 3D game, or the next CGI movie, this course will not meet your expectations. This course is designed around understanding core concepts and preparing you to explore more advanced topics in vast field of computer graphics.

A Birds-eye view of Computer Graphics

Computer Graphics aims to compute realistic-looking representations of geometric models and scenes. The view through our eyes or the lens of a camera is a complex interaction of material surfaces, light scattering of photons, and the optics of lenses. While the underlying physics is well understood, modeling these interactions exactly on a single computer is computationally infeasible. The field of computer graphics seeks to find efficient ways to approximate these interactions and create ever more realistic scenes.

For some state of the art rendering, check out the short demo of the Marbles game unveiled during the 2020 GPU Technology Conference. Work like this requires teams of artists and engineers along with powerful hardware to produce. We’ll have considerably more modest goals, but we’ll be learning many of the core techniques used by professionals.

We will begin our journey using a modern OpenGL approach, which consists of the following key steps.

  1. Modeling: Defining 3D objects with geometric primitives including points, lines and triangles.

  2. Scene Creation: Arranging objects in a virtual world using linear algebra transformations.

  3. Projections: Setting a viewpoint for visualizing the world, and converting the scene into a standard volume.

  4. Rasterizing: Transforming the projected 3D scene into a 2D image of pixels/fragments on the computer screen or other output device.

We will initially see how to perform these basic steps, and we will then gradually add more advanced features such as texturing, lighting, and noise to improve the realism.

Later we will explore Ray Tracing or Ray Marching as an entirely different way of modeling the interaction of light and surfaces to create a scene. Both the Triangle→Tranform→Rasterize and Ray Tracing methods are widely used in modern graphics.

Connections to other Courses

Some courses explore topics related to those in this course, but approach various problems from a different perspective or starting point:

  • Computer Vision: Starts with images, ends with models

  • Animation/Game Design: Extends core concepts with advanced applications, models of motion, game AI, level design.

  • Build OpenGL: Start with C and 2D arrays, build something like OpenGL from scratch.

Tools

See Remote Tools for many of tools we’ll be using for remote learning in this course, and the Resources section for technical tools.

  • OpenGL/WebGL2: the primary graphics framework for understanding core low level concepts

    • GLSL: the shading language for OpenGL

    • TWGL: a thin helper library around WebGL2 to simplify the really boring stuff, but still highlight the core

    • Three.js: a higher level library on top of WebGL2 once we know the basics and want to expand

  • JavaScript: Our language for develop apps we can view through locally on a browser

  • Git: version control, sharing

  • CUDA?: GPGPU programming

Math

Linear Algebra

  • Matrices (4x4)

  • Vectors (up to 4x1)

  • Dot product

  • Cross product

  • Matrix/Vector product

  • Affine Transformation

  • Frames, Orthogonal Bases

Trigonomety

  • Sine

  • Cosine

  • Radians/Degrees

Hardware

This course will mostly focus on the software side of computer graphics, but realism has greatly increased since the advent of dedicated Graphics Processing Units (GPUs). These hardware devices have some notable properties compared to general purpose CPUs

  • optimized for vector math

  • highly parallel (1000-2000 cores)

  • SIMD

  • programmable via shaders or CUDA

See NVIDIA CEO Jensen Huang introduce a new GPU from his kitchen below:

Additionally, large format displays, multi panel displays, 3D displays, and 3D printers have connections to many computer graphics concepts

Overview of a WebGL2 program

Today we’ll start going over a complete WebGL2 program that draws a single triangle. Along the way we’ll be pointing out the key components in all OpenGL programs. Even though this demo will look overly complex, we will see it is extremely flexible and we will be extending and adding to this basic framework to create more realistic and more complex scenes throughout the course.

The starting version of our demos will be hosted online at the CS40 demos page.

You can also clone a copy of these demos for experimentation on your own

cd cs40
git clone git@github.swarthmore.edu:CS40-F20/demos.git
ln -s /usr/local/stow/CS40-F20/lib/cs40lib

Then you can cd into the demos/w01-intro folder and launch the python3 webserver and ngrok as we did in lab1. This repo is read only. You will be able to pull, but not push.

We’ll be looking at the first* files today.

Setting up

Web browsers know how to display web pages, so we need a template web page. This in our first.html file. It’s fairly small because most of the work is going to be done elsewhere, in an element called the canvas. An html canvas is a container for drawing graphics. The actual drawing of the graphics in the canvas is done in a separate language called JavaScript.

HTML describes the content and structure, but not appearance. That’s yet another language, CSS. HTML and CSS will not have a strong focus in the course. You can see a small first.css file that says to make our canvas as big as the HTML window. This is actually a bit annoying as it tends to add scroll bars if we really make it full size. Try replacing 100 with 95 or 98 and see if the scroll bars go away on your browser.

The script that draws to our canvas is in first.js

JavaScript setup

JavaScript is a web language for creating and modifying dynamic web content. It can do lots of things besides draw images. We will primarily be using it to draw graphics in our canvas element. To get started, we need a context

const gl = document.querySelector("#demo").getContext("webgl2");

This tells us that we want select the html element with id demo and create a webgl2 context so that we can start sending webgl2 commands to this canvas to draw an image. WebGL2 is a variant of OpenGL, a low level API for interacting with the graphics hardware. These APIs define some core functions and expected behavior for interacting with the GPU. OpenGL originally was implemented as a C library, but there are now variants in C++, python, and through WebGL2, JavaScript. The syntax of the commands will vary slightly across languages, but learning OpenGL/WebGL2 concepts in one language should allow you to transfer the OpenGL part to an other language relatively smoothly.

As you will see, OpenGL is responsible for talking to the GPU and not much else. It does not have functions for creating buttons, or windows, or handling using interaction. That is usually done through the parent language (JavaScript in our case), and other libraries.

We divided first.js into two primary functions, init which only needs to run once, and render which draws one frame of our scene. Currently, this is a static image, but we will soon be able to modify the render function to animate the scene and we do not need to redo the steps in the init function.

The core of an OpenGL program is to first set up the shaders (what do you want to do?) and the input data (what do you want the shaders to process?). Once we send the shader program and the geometry data to the GPU, we give one small command gl.drawArrays to actually do the drawing.

Friday

  • Continue working on lab1, ask questions on Slack, attend office hours today 1:30-2:30pm.

  • Set Partner preferences for lab2.

Today’s goals:

  • Understand GPU Inputs/Outputs in a WebGL2 program

  • Full WebGL2 walkthrough of a complete program.

  • Basics of Vertex/Fragment shader programs

  • in

  • out

  • uniform

  • requestAnimationFrame, render, and the animation loop

Init

  • loading, compiling, linking shaders

    • We write shaders in GLSL, e.g., first_vs.js and first_fs.js.

    • We send the shader source to the GPU to be compiled and linked into a shader program. Each program must have one vertex shader and one fragment shader.

      programInfo = twgl.createProgramInfo(gl, [vshader, fshader]);

  • creating geometry data on the CPU

    • Our models start on the CPU side, but eventually we will copy the data to the GPU.

    • For a simple model like our triangle, we can define all the geometry directly in init().

      const triangle = {
  position: {numComponents: 2,
             data: [-1, -1, 1, -1, 0, 1] }
};

  • creating GPU buffer

    • Next, we request a buffer from the GPU, allocate space for our geometry on the GPU, and copy the vertex data from the CPU to the GPU. The twgl library does all of this in one step.

      bufferInfo = twgl.createBufferInfoFromArrays(gl, triangle);

  • setup VAO

    • The vertex shader will read data from a buffer to get its inputs. We need to describe exactly how to map the array of data in the buffer to the inputs in the vertex shader. The VAO helps with this mapping.

    • Note that we called the array of vertices for the triangle position. There is a input variable a_position in the vertex shader. Earlier in init() we called the helper function twgl.setAttributePrefix("a_"). With this information, twgl can infer that we want to connect the elements in the position array from our CPU buffer to the a_position variable in the vertex shader program.

    • It is not required that these names match up, but if they do not, you will need to do extra work to describe how to connect the buffer to the shader. It’s just a triangle for now, so simple defaults are good enough.

       vao = twgl.createVAOFromBufferInfo(
    gl, programInfo, bufferInfo);

  • set OpenGL flags, initial state

    • OpenGL is simply a scaffold. To draw anything, it is essential that you provide shaders and models. But there is also additional state you can set in the init() function. One example is setting the background clear color.

      gl.clearColor(0.9,0.9,0.9,1.);

Most of OpenGL is just setting state. For most of the functions, none of the effects are directly visible. You are simply asking OpenGL to set some internal variable and remember it for later use. When it comes time to actually draw, the draw command will be relatively simple, but it will leverage all the current state you set with prior OpenGL function calls. This may be a new way of thinking about things, but we will continue to emphasize this point throughout the course.

Render

  • Adjust OpenGL context to fit current window

    • Occasionally, the user might resize the window and we’d like OpenGL to adjust the rendering context to fit the new size.

      twgl.resizeCanvasToDisplaySize(gl.canvas);
gl.viewport(0, 0, gl.canvas.width, gl.canvas.height);

  • Erase old data

    • OpenGL will write output color fragments to a buffer and display this buffer in the current context. If you are rendering multiple frames, you must explicitly erase the old frame using gl.clear. This will fill the output buffer with the current clear color that you most recently set with the gl.clearColor function.

      gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);

  • Make program, vao active

    • We are now ready to draw. It’s a good idea to ensure your program and buffers are active. Since there is only one program/buffer, this step probably could be done once in init, but for complex scenes, we may toggle between multiple programs and buffers and having this process in render is a good idea.

      gl.useProgram(programInfo.program);
gl.bindVertexArray(vao);

  • Draw!

    • All the prep work is done. To actually get things going on the GPU and see an output image, we have to call a draw function.

      gl.drawArrays(gl.TRIANGLES, 0, 3);

      The syntax for drawArrays is drawArrays(primitive, offset, count). This function instructs OpenGL to start the vertex shader and start drawing the specified primitive gl.TRIANGLES using the data in the current buffer starting at the provided offset 0 and using count(=3) total vertices. In this example, we have just instructed OpenGL to draw one triangle. That seemed like a lot of work!

TWGL vs Real WebGL2

You may have notices that some functions begin with gl. while others begin with twgl.. The gl. functions are core WebGL2 functions, and you would usually see them in OpenGL APIs for other languages. twgl is a 3rd party tool that provides a thin wrapper that simplifies some of the gl. commands. OpenGL by itself is very verbose. You can see a similar demo in straight WebGL2 that has a lot of extra gl. calls. But you should be able to identify the init and render steps above.

WebGL2 Fundamentals also has very nice post on how the individual vertex shaders and fragment shaders get called on a sample triangle. Their example does a few extra steps and transforms that we haven’t yet discussed, but in terms of seeing multiple calls to each shader, it’s a good description. Keep in mind that to keep the illustrations clear, the steps are shown sequentially, but in reality, OpenGL and the GPU processes geometry and fragments in parallel, leading to greater efficiency.