diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index dbca4b5..dccffe0 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,3 +1,5 @@ + + ## What changes are you trying to make? (e.g. Adding or removing code, refactoring existing code, adding reports) ## What did you learn from the changes you have made? diff --git a/.github/workflows/autograder.yml b/.github/workflows/autograder.yml new file mode 100644 index 0000000..2956274 --- /dev/null +++ b/.github/workflows/autograder.yml @@ -0,0 +1,48 @@ +name: Assignment autograder +on: + pull_request: + branches: + - main + types: [opened, synchronize, reopened] + + # Allows you to run this workflow manually from the Actions tab + workflow_dispatch: + +permissions: + # Read repository contents for checkout and grading. + contents: read + # Pull request comments use the Issues API. + issues: write + # Required to create REQUEST_CHANGES reviews when checks fail. + pull-requests: write + +jobs: + autograder: + name: Assignment autograder + if: startsWith(github.head_ref, 'assignment') && github.repository_owner != 'UofT-DSI' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: astral-sh/setup-uv@v5 + - name: Setup working directory for script + run: | + UUID=$(uuidgen) + echo "UUID=$UUID" >> $GITHUB_ENV + mkdir /tmp/$UUID + cp $GITHUB_WORKSPACE/02_activities/assignments/assignment_script.sh /tmp/$UUID + - name: Run assignment script + run: | + bash -c "bash assignment_script.sh 2>&1 | tee /tmp/${{ env.UUID }}_output.txt" + working-directory: /tmp/${{ env.UUID }} + - name: Grade responses + id: grade + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + REPO_OWNER: ${{ github.repository_owner }} + REPO_NAME: ${{ github.event.repository.name }} + PR_NUMBER: ${{ github.event.pull_request.number }} + REPO_BRANCH: ${{ github.event.pull_request.head.ref }} + WORKING_DIR: /tmp/${{ env.UUID }} + run: | + wget -O /tmp/autograder.py https://github.com/UofT-DSI/shell/raw/refs/heads/main/03_instructional_team/autograder/autograder.py + uv run --python 3.12 --with pandas,tabulate,requests /tmp/autograder.py diff --git a/.github/workflows/deploy_github_pages.yml b/.github/workflows/deploy_github_pages.yml index a18df3c..507b571 100644 --- a/.github/workflows/deploy_github_pages.yml +++ b/.github/workflows/deploy_github_pages.yml @@ -2,9 +2,7 @@ name: Deploy static content to Pages on: - # Runs on pushes targeting the default branch push: - branches: ["main"] # Allows you to run this workflow manually from the Actions tab workflow_dispatch: @@ -24,6 +22,9 @@ concurrency: jobs: # Single deploy job since we're just deploying deploy: + if: | + (github.repository == 'UofT-DSI/shell' && github.ref == 'refs/heads/main') || + (github.repository != 'UofT-DSI/shell' && github.ref == 'refs/heads/githubpages') environment: name: github-pages url: ${{ steps.deployment.outputs.page_url }} diff --git a/.gitignore b/.gitignore index 72a2ccb..e478052 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,29 @@ .DS_Store .vscode/ -/03_instructional_team/node_modules -.venv \ No newline at end of file +03_instructional_team/lessons/pdf/* +03_instructional_team/lessons/html/* + +# local node modules +node_modules/ +package-lock.json +package.json + +### Running the assignment script unzips a large number of files into the tree +# Ignore all files in shell/02_activities/assignments/ +02_activities/assignments/* + +# Un-ignore the specific files +!02_activities/assignments/assignment.sh +!02_activities/assignments/github_actions.png +!02_activities/assignments/assignment_instructions.md + +# Python +.venv/ +__pycache__/ +*.py[cod] +*$py.class +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ + +newproject/ diff --git a/01_materials/ShellGitMindmap.svg b/01_materials/ShellGitMindmap.svg new file mode 100644 index 0000000..7cd41d2 --- /dev/null +++ b/01_materials/ShellGitMindmap.svg @@ -0,0 +1,4 @@ + + + +
Unix
operating system
is an
Linux
is a derivative of
MacOS
is a derivative of
Shell
is a
Bash
is a type of
zsh
is a type of
Command-line interface
is a way to
interact with
user interface
are a type of
Graphical user interface
are a type of
enables access
to
terminal emulator
Windows Terminal
is a type of
MacOS Terminal
is a type of
also installs
(Windows only)
creates
Git
stores
version control
is a tool for
can be
uploaded to
Git repository
enables
project history
is a type of
Git Bash
is required for
accountability
backups
robust software
is a user interface for
github.com
\ No newline at end of file diff --git a/01_materials/bash_quick_reference.md b/01_materials/bash_quick_reference.md new file mode 100644 index 0000000..2f43154 --- /dev/null +++ b/01_materials/bash_quick_reference.md @@ -0,0 +1,84 @@ +# Bash Commands Reference + +## 📋 Commands Overview + +```bash +$ echo Rachael +Rachael + +$ date +Tue, Nov 19, 2024 6:28:10PM + +$ asdkfjhgoipwhr +Command not found + +$ man ls +# Opens the manual for `ls` +# To exit, press 'q' + +$ help +# Lists available shell commands +``` + +--- + +## 📁 File & Directory Navigation + +| Command | Description | +|------------------------|------------------------------------------------------------| +| `pwd` | Show current working directory | +| `ls` | List folders & files in the working directory | +| `cd` | Move to home directory | +| `cd Desktop` | Move into Desktop directory | +| `cd ` | Move into a specific path | +| `cd ..` | Move one level up (parent directory) | + +--- + +## 🛠 File & Directory Management + +| Command | Description | +|-------------------------------------|-----------------------------------------------------| +| `mkdir ` | Make new directories | +| `touch ` | Create a file | +| `rm ` | Remove a directory | +| `rm ` | Remove a file | +| `cp ` | Copy file from source to destination | +| `cp /* ` | Copy all files from source dir to destination dir | +| `mv ` | Rename/move file1 to file2 | +| `cat ` | Display content of the file | +| `echo "text" > file.txt` | Replace content in file with text | +| `echo "text" >> file.txt` | Append text to file | + +--- + +## ✏️ Editing Files + +| Command | Description | +|--------------------|------------------------------------| +| `nano ` | Open file in nano text editor | +| `code ` | Open file in VS Code | + +--- + +## 🧰 Useful Options + +| Option | Description | +|--------|-----------------------------------------------------------| +| `-i` | Prompt before overwriting an existing file | +| `-r` | Recursively copy directories and their contents | +| `-v` | Show informative messages during operations | +| `-f` | Force action (overrides `-i` if both are used together) | + +--- + +## 🔧 Hash-bang (Shebang) + +Place this at the top of every Bash script to indicate which shell to use: + +```bash +#!/bin/bash +``` + +--- + diff --git a/01_materials/slides/optional_unix_slides.pdf b/01_materials/slides/optional_unix_slides.pdf new file mode 100644 index 0000000..20e58a7 Binary files /dev/null and b/01_materials/slides/optional_unix_slides.pdf differ diff --git a/01_materials/slides/unix_slides.pdf b/01_materials/slides/unix_slides.pdf new file mode 100644 index 0000000..3428e15 Binary files /dev/null and b/01_materials/slides/unix_slides.pdf differ diff --git a/02_activities/assignments/assignment_instructions.md b/02_activities/assignments/assignment_instructions.md new file mode 100644 index 0000000..9015eca --- /dev/null +++ b/02_activities/assignments/assignment_instructions.md @@ -0,0 +1,104 @@ +# Shell / Git Assignment +You work in the data team at a consulting firm, and one of your team's products is helping companies optimize and manage their cloud hosting expenditures. + +Your team has an existing bash script that initializes an analysis directory for each new client. This script is shared amongst members of the team. + +You've been asked to update this script to also automate the initial organization of data files provided by clients. + +## Instructions + +### Setup +1. **Forking the Repository**: If you have not already done so, fork this Shell learning module repository following these [instructions](https://github.com/UofT-DSI/onboarding/blob/main/onboarding_documents/submissions.md#setting-up). + - Forking creates a copy of the main repository in your GitHub account. This allows you to work on your version without affecting the original repository. + +2. **Enable GitHub Actions**: Click on the **Actions** tab in your repository and enable workflows if prompted. + + +2. **Clone Your Fork To Your Computer**: To make edits and test your script. + ```bash + cd + git clone + ``` + +2. **Create a Branch for Your Work**: To keep your changes organized, create a new branch named `assignment_shell`: + ```bash + git switch -c assignment_shell + ``` + +--- + +### Part 1: Update the Data Ingest Script +1. **Modify the Script**: Using the template in ``02_activities/assignments/assignment_script.sh``, fill in the correct commands as described by the comments. + - It may help to paste your commands into the Terminal as you write your script (or vice versa) to test as you go + +2. **Test Your Script Locally**: Execute your script to ensure it works as expected. You may need to make additional tweaks and re-run it until you are satisfied with the results. + ```bash + bash assignment_script.sh + ``` + + - Check if the expected directories are created. (eg. by browsing the directory with `cd` and `ls`) + - Verify that files are moved or copied as expected. + - Ensure that files that should be deleted are no longer present. + - Ensure your script works regardless + +3. **Commit Your Changes**: As you complete parts of your script and confirmed that it is working correctly, stage and commit your changes to ensure your progress is saved. + ```bash + git add assignment_script.sh + git commit + ``` + + - Remember that the script is shared amongst your team members. Your script should run properly in any directory and should not have any hardcoded absolute paths or rely on advanced functionality not available in standard Bash. + - *Note*: Your assignment is graded solely based on `assignment_script.sh`. + - ‼️ You **must not** include any data files created during testing of your script and you **must not** change any other files. + +4. **Push Your Changes to GitHub** + ```bash + git status # check that your working directory is clean (all changes committed) + git push origin assignment_shell # push your changes + ``` + - ‼️ **Do not** push to `main` + +5. **Create a pull request** + - Open a pull request from your `assignment_shell` branch to your repository's `main` branch. + - The autograder will run automatically and post your assignment grade as a comment. + - Ensure the pull request is **NOT** made to the `UofT-DSI` organization + - ‼️ **Do not merge** your pull request until the end of the module. Your teaching team will only grade *open* pull requests. + +6. **Repeat steps 1-4 as needed**: As you make tweaks to your assignment, push new changes to your `assignment_shell` branch to rerun the autograder + - There's no need to create a new pull request + - Pushes to the `assignment_shell` branch will trigger additional runs + +--- + +### Part 2: Merge in Updates from Your Coworkers +Your coworkers have made some other changes to the script. You'll need to incorporate their updates and resolve any conflicts. +1. **Merge the Updates**: Use the following command to pull in changes from the coworker's branch: + ```bash + git pull https://github.com/UofT-DSI/shell coworker-changes --no-rebase + ``` +2. **Resolve Merge Conflicts**: If there are any conflicts, use ```git status``` to see which files are affected, resolve the conflicts manually, and then mark them as resolved. +3. **Commit the Merge**: Once all conflicts are resolved, commit the merge. +1. **Re-test Your Script**: Make sure that your script still works after merging the updates and make any changes as necessary + +--- + +## Submission Information + +🚨 **Please review our [Assignment Submission Guide](https://github.com/UofT-DSI/onboarding/blob/main/onboarding_documents/submissions.md)** 🚨 for detailed instructions on how to format, branch, and submit your work. Following these guidelines is crucial for your submissions to be evaluated correctly. + +### Grading guidelines: +This assignment is considered complete if all autograder tests pass successfully and your Pull Request is approved by the github-actions bot, and incomplete otherwise. + +### Submission Parameters: +* The branch name for your repo should be: `assignment_shell` +* What to submit for this assignment: + * One or more commits that update the `assignment_script.sh` script +* What the pull request link should look like for this assignment: `https://github.com//shell/pull/` + * Open a private window in your browser. Copy and paste the link to your pull request into the address bar. Make sure you can see your pull request properly. This helps the technical facilitator and learning support staff review your submission easily. + +Checklist: +- [ ] Create a branch called `assignment_shell`. +- [ ] Ensure that your repository is public. +- [ ] Review [the PR description guidelines](https://github.com/UofT-DSI/onboarding/blob/main/onboarding_documents/submissions.md#guidelines-for-pull-request-descriptions) and adhere to them. +- [ ] Verify that your link is accessible in a private browser window. +- [ ] Verify the autograder has run and approved your pull request diff --git a/02_activities/assignments/assignment_script.sh b/02_activities/assignments/assignment_script.sh new file mode 100644 index 0000000..c10b794 --- /dev/null +++ b/02_activities/assignments/assignment_script.sh @@ -0,0 +1,50 @@ +#!/bin/bash +set -x + +############################################ +# DSI CONSULTING INC. Project setup script # +############################################ +# This script creates standard analysis and output directories +# for a new project. It also creates a README file with the +# project name and a brief description of the project. +# Then it unzips the raw data provided by the client. + +if [ -d newproject ]; then + echo "Recreating the newproject directory" + rm -rf newproject +fi +mkdir newproject +cd newproject + +mkdir analysis output +touch README.md +echo "# Project Name: DSI Consulting Inc." > README.md +touch analysis/main.py + +# download client data +curl -Lo rawdata.zip https://github.com/UofT-DSI/shell/raw/refs/heads/main/02_activities/assignments/rawdata.zip +unzip -q rawdata.zip + + +########################################### +# Complete assignment here + +mkdir data +mv rawdata data/raw +ls data/raw + +mkdir data/processed +mkdir data/processed/server_logs data/processed/user_logs data/processed/event_logs + +cp data/raw/*server*.log data/processed/server_logs/ +cp data/raw/*user_log_*.log data/processed/user_logs/ +cp data/raw/*event_log_*.log data/processed/event_logs/ + +rm data/raw/*ipaddr* +rm data/processed/user_logs/*ipaddr* + +find data/processed -type f > data/inventory.txt + +########################################### + +echo "Project setup is complete!" diff --git a/02_activities/assignments/github_actions.png b/02_activities/assignments/github_actions.png new file mode 100644 index 0000000..9f717e2 Binary files /dev/null and b/02_activities/assignments/github_actions.png differ diff --git a/02_activities/assignments/rawdata.zip b/02_activities/assignments/rawdata.zip new file mode 100644 index 0000000..fe36b0f Binary files /dev/null and b/02_activities/assignments/rawdata.zip differ diff --git a/02_activities/practice/shell_advanced/dataset.zip b/02_activities/practice/shell_advanced/dataset.zip new file mode 100644 index 0000000..ef16cf7 Binary files /dev/null and b/02_activities/practice/shell_advanced/dataset.zip differ diff --git a/02_activities/practice/shell_advanced/shell_advanced.md b/02_activities/practice/shell_advanced/shell_advanced.md new file mode 100644 index 0000000..7132f96 --- /dev/null +++ b/02_activities/practice/shell_advanced/shell_advanced.md @@ -0,0 +1,65 @@ +# Unix Shell Advanced Practice Problems + +## Introduction +Kaylee and Simeon are biomedical engineering researchers trying to reanalyze data from an undergraduate project, where they collected electroencephalography (EEG), electromyography (EMG), and actigraphy (acceleration and rotational) recordings, and other information from subjects as they tried to keep their balance on a moving platform for a balance project. + +Unfortunately, when they were silly undergraduates, they didn't know to keep their data tidy. They'd like your help efficiently doing the following data management tasks using your newfound Bash knowledge. + +## Tasks +* Complete the listed tasks in Bash +* This problem set requires commands that were not covered in the teaching sessions. +* Feel free to use any commands or strategy that you'd like. Suggested commands are listed. + +### Task 1: Setup your environment +1. Download their undergraduate data package: [dataset.zip](dataset.zip?raw=1) +1. Unzip it into your directory + * *Hint*: Consider the `unzip` command +1. Change directory into the folder containing the data package contents + * *Hint*: Consider the `cd` command +1. To make sure we're keeping good records this time, print the current path to your working directory + * *Hint*: Consider the `pwd` command +1. Make a new folder named `tidied_data` + * *Hint*: Consider the `mkdir` command + +### Task 2: Taking inventory +1. List the contents of the data folder + * *Hint*: Consider the `ls` command +1. List all the EEG files and write this list to a text file in the tidied_data folder named `eeg_inventory.txt`. + * *Hint*: Consider the `ls` and `head` commands, and either the `>` file redirection operator or `|` pipe operator and the `tee` command + * Preview the **first 8 lines** of this file using the `head` command, does this look right? + +### Task 3: Taking inventory, part 2 +1. Let's double check: does the naming convention of the EEG files match `eeg_[subjectid]_[session].edf`? + * List the contents of the folder and inspect it visually! +1. Based on the `eeg_inventory.txt` file and the naming convention, generate a list of subject numbers and write it a file named `subject_ids_from_eeg.txt`. + * Preview the **first 8 lines** of this file. + * *Hint*: Consider the `cut` command +1. Sort the `subject_ids_from_eeg.txt` file and write the output to `subject_ids_from_eeg_sorted.txt`. + * Preview the **first 8 lines** of this file. +1. Let's double check: does each subject have multiple EEG files? Are their subject IDs duplicated? + * *Hint*: Look through the whole with the `less` command +1. Create a unique list of subject IDs and write the output to `subject_ids.txt`. + * *Hint*: Consider the `unique` command + * Preview the **first 8 lines** of this file. + +### Task 4: The life changing magic of tidying up +1. For each subject: + 1. create a folder named after the subject ID in the `tidied_data` directory + * *Hint*: Consider a `for` loop + 1. Move all files relating to that subject into their respective directories (eg. `tidied_data/[subjectid]/eeg_[subjectid]_[session].edf`) + * *Hint*: Consider the `mv` command +1. Notice that all the notes files have not been named consistently. Rename all the note files to `[subjectid]_notes.txt` within each subject folder. + * *Hint*: Consider the `mv` command inside a `for` loop or using `find ... -exec mv ...` + +### Task 5: Checking our work +1. Confirm that you've copied all the files over to the `tidied_data` directory + 1. Count the number of files copied: + 1. Generate a list of all the files within all the directories in `tidied_data` + * *Hint*: Consider the `find` command and look for files with the `-type` option + 1. Count the number of lines in the file list + * *Hint*: Consider the `wc` command. How do we make it count lines? + 1. Count the number of files in the original folder, using the same strategy as above +1. Do the file counts match? + * *Hint*: Consider using variables, an `if` statement, and `-eq` + + \ No newline at end of file diff --git a/02_activities/practice/shell_basics.md b/02_activities/practice/shell_basics.md new file mode 100644 index 0000000..888b7b6 --- /dev/null +++ b/02_activities/practice/shell_basics.md @@ -0,0 +1,4 @@ +# Unix Shell Basics Practice Problems + +Load the basic practice problems in your browser here: https://uoft-dsi.github.io/shell/interactive_problems.html#shell_basics + diff --git a/02_activities/practice/shell_moderate.md b/02_activities/practice/shell_moderate.md new file mode 100644 index 0000000..d169c2b --- /dev/null +++ b/02_activities/practice/shell_moderate.md @@ -0,0 +1,4 @@ +# Unix Shell Moderate Practice Problems + +Load the moderate practice problems in your browser here: https://uoft-dsi.github.io/shell/interactive_problems.html#shell_moderate + diff --git a/03_instructional_team/GitShell_mindmap.drawio b/03_instructional_team/GitShell_mindmap.drawio new file mode 100644 index 0000000..0789fb7 --- /dev/null +++ b/03_instructional_team/GitShell_mindmap.drawio @@ -0,0 +1,190 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/03_instructional_team/README.md b/03_instructional_team/README.md index 719baf7..d16e11b 100644 --- a/03_instructional_team/README.md +++ b/03_instructional_team/README.md @@ -1,4 +1,4 @@ -# Technical Facilitator Playbook +# Instructional Team Playbook ## How do you interact with the repo? The Technical Facilitator will deliver the content in the `/01_materials/slides` directory. You are encouraged to live code with participants during live sessions. Please ensure that live coding files are uploaded to a new directory called `/live_code` under `/04_cohort_three` in this repository using a new branch. Please open a pull request for it to be merged. @@ -13,28 +13,22 @@ The `/01_materials/slides` directory contains the live learning session slides. The `/02_activities/assignments` contains assignments participants should submit for evaluation as `complete` or `incomplete`. The assignments measure a participant's achievement of the learning outcomes, and help technical facilitators determine if a participant has successfully completed the learning module. -The `/02_activities/homework` directory contains homework participants can complete to further develop and practice the skills covered in a learning module. Homework is optional, but participants are encouraged to complete as much as they can. +The `/02_activities/homework` directory contains homework participants can complete to further develop and practice the skills covered in a learning module. Homework is optional, but participants are encouraged to complete as much as they can. ### Week 1 #### Live Learning Session -The focus of the live learning session is to introduce the learners to using `git` commands such as: -```git -git add . -git commit -m "Your message" -git push -u origin -git pull -git checkout -b -git checkout -``` +The focus of the live learning session is to introduce participants to basic bash commands such as `cd` and `pwd`, as well as creating files using `touch` and directories using `mkdir`. The module also introduces intermediate commands like `rm`, how to recursively remove files from a nested directory, and how to make copies or move files using `mv`. + +Note: If someone wants to open up the "manual", recommend they use the `man` command. #### Assignment -The assignment for this week helps participants learn more about `git` and the difference between 'git' and `GitHub`. They will practice using `git` to constantly push their answers (changes) onto `GitHub`. +The assignment for this week is to have participants find the secret password by navigating directories and files. ## How do you assign assignments? Technical Facilitators are encouraged to introduce assignments as early as possible in the learning module. The Technical Facilitator should describe the assignment to participants and explain how the topics covered in the module will equip them with the knowledge and skills to complete the assignment. -## How an assignment is expected to be completed and delivered? +## How is an assignment is expected to be completed and delivered? Participants are expected to complete the assignment by the end of the first week. They will deliver the assignment by opening a pull request on their copied repo. The participants will also add a Learning Support Staff as a reviewer indicating they delivered a completed assignment, and it is ready to be evaluated as `complete` or `incomplete`. ## What is the criteria for `complete` or `incomplete`? @@ -58,17 +52,3 @@ Feedback should be given through the pull request a participant has made. Techni ## generate_slides.sh This script is designed to convert Markdown files located in a specified folder into slide presentations using Marp CLI, allowing for the generation of either HTML or PDF formats based on user input. It includes an option to apply a custom CSS theme to the slides by specifying a theme path. The script also provides a help function detailing its usage, options, and examples for convenience. It ensures the necessary directories exist, validates the presence of Marp CLI on the system, processes each Markdown file found in the specified directory, and outputs the generated slides into a designated output folder, displaying the status of each operation and a completion message. The only configuration needed is to set where the md files are and where you would like the pdf/html files to be placed. - -Example command to generate themed PDF slides: - -```bash -cd 03_instructional_team -./generate_slides.sh --pdf --theme markdown_slides/themes/dsi_certificates_theme.css -``` - -To generate themed HTML slides instead, use: - -```bash -cd 03_instructional_team -./generate_slides.sh --html --theme markdown_slides/themes/dsi_certificates_theme.css -``` diff --git a/03_instructional_team/autograder/autograder.py b/03_instructional_team/autograder/autograder.py new file mode 100644 index 0000000..106e76d --- /dev/null +++ b/03_instructional_team/autograder/autograder.py @@ -0,0 +1,363 @@ +import pandas as pd +import os +import requests +import glob +from itertools import compress + +# get environment variables for output +if 'GITHUB_TOKEN' in os.environ: + github_token = os.environ["GITHUB_TOKEN"] + github_step_output = os.environ['GITHUB_STEP_SUMMARY'] + github_repo_owner = os.environ["REPO_OWNER"] + github_repo_name = os.environ["REPO_NAME"] + github_repo_branch = os.environ["REPO_BRANCH"] + github_pr_number = os.environ["PR_NUMBER"] +else: + github_token = None + +root_dir = os.environ["WORKING_DIR"] +working_dir = os.path.join(root_dir, 'newproject'); + +status_c = '✅' +status_i = '❌' +status_u = '⚠️' + + +# functions +def is_commit_in_branch( + owner, + repo, + branch, + other_owner, + other_repo, + other_branch, +): + headers = {} + if github_token: + headers['Authorization'] = f'token {github_token}' + + # Step 1: Get the latest commit SHA of the other repository's branch + other_commit_url = f'https://api.github.com/repos/{other_owner}/{other_repo}/commits/{other_branch}' + response = requests.get(other_commit_url, headers=headers) + response.raise_for_status() + other_commit_sha = response.json()['sha'] + print(f'Commit SHA of {other_owner}/{other_repo}/{other_branch}: {other_commit_sha}') + + # Step 2: Check if this commit exists in the given repository + commit_in_repo_url = f'https://api.github.com/repos/{owner}/{repo}/commits/{other_commit_sha}' + response = requests.get(commit_in_repo_url, headers=headers) + if response.status_code == 404: + # Commit does not exist in the given repository + print(f'Commit {other_commit_sha} not found in {owner}/{repo}') + return False + response.raise_for_status() + + # Step 3: Compare the commit with the branch in the given repository + compare_url = f'https://api.github.com/repos/{owner}/{repo}/compare/main...{branch}' + response = requests.get(compare_url, headers=headers) + response.raise_for_status() + compare_data = response.json() + + commit_shas = [commit['sha'] for commit in compare_data['commits']] + + # If the status is 'ahead' or 'identical', the commit is in the history + print(f'Commit SHAs in {owner}/{repo}/{branch}: {commit_shas}') + return other_commit_sha in commit_shas + + +def post_to_github(url, payload, headers, description): + response = requests.post(url, json=payload, headers=headers) + if response.ok: + print(f'GitHub {description} posted successfully.') + else: + print( + f'GitHub {description} failed with status {response.status_code}: ' + f'{response.text}') + return response + + +# score table +s = [] + +# load script output +with open(root_dir + '_output.txt', 'r') as f: + script_rslt = f.read() + +script_rslt = script_rslt.split('\n+') +script_rslt = [{ + 'command': x.split('\n')[0][1:].strip(), + 'output': x.split('\n')[1:] +} for x in script_rslt] + +qn = 0 + +############################################################################################################ +# Step 1: Check if 'data' directory exists +qn += 1 +if os.path.isdir(os.path.join(working_dir, 'data')): + s.append({'question': f'Part 1 - Q{qn:d}', 'status': 1}) +else: + s.append({ + 'question': f'Part 1 - Q{qn:d}', + 'status': 0, + 'comment': 'data directory does not exist' + }) + +############################################################################################################ +# Step 2: Check that 'rawdata' directory was moved to 'data/raw' +qn += 1 +if os.path.isdir(os.path.join(working_dir, 'data/raw')) and not os.path.exists( + os.path.join(working_dir, 'rawdata')): + s.append({'question': f'Part 1 - Q{qn:d}', 'status': 1}) +else: + s.append({ + 'question': f'Part 1 - Q{qn:d}', + 'status': 0, + 'comment': 'rawdata not moved to data/raw' + }) + +############################################################################################################ +# Step 3: Check that 'ls data/raw' command was run +qn += 1 +indx = [i for i, x in enumerate(script_rslt) if x['command'].startswith('ls')] +if len(indx) > 0: + if any(['data/raw' in script_rslt[i]['command'] for i in indx]): + s.append({'question': f'Part 1 - Q{qn:d}', 'status': 1}) + else: + s.append({ + 'question': f'Part 1 - Q{qn:d}', + 'status': 0, + 'comment': '`ls` command run on wrong directory' + }) +else: + s.append({'question': f'Part 1 - Q{qn:d}', 'status': 0, 'comment': '`ls` command not run'}) + +############################################################################################################ +# Step 4: Check that in 'data/processed', the directories server_logs, user_logs, and event_logs were created +qn += 1 +dirs = [ + 'data/processed/server_logs', 'data/processed/user_logs', + 'data/processed/event_logs' +] +if all([os.path.isdir(os.path.join(working_dir, d)) for d in dirs]): + s.append({'question': f'Part 1 - Q{qn:d}', 'status': 1}) +else: + missing_dirs = [ + d for d in dirs if not os.path.isdir(os.path.join(working_dir, d)) + ] + s.append({ + 'question': f'Part 1 - Q{qn:d}', + 'status': 0, + 'comment': f'Missing directories: {", ".join(missing_dirs)}' + }) + + +############################################################################################################ +# Step 5: Check that server log files were copied from 'data/raw' to 'data/processed/server_logs' +def check_logs(log_type): + raw_logs = glob.glob( + os.path.join(working_dir, f'data/raw/*{log_type}*.log')) + processed_logs = glob.glob( + os.path.join(working_dir, f'data/processed/{log_type}_logs/*')) + if len(raw_logs) == 0: + return {'status': 0, 'comment': f'No {log_type} log files in data/raw'} + else: + raw_log_files = [os.path.basename(f) for f in raw_logs] + processed_log_files = [os.path.basename(f) for f in processed_logs] + if all([f in processed_log_files for f in raw_log_files]): + return {'status': 1} + else: + return { + 'status': 0, + 'comment': f'Missing files in data/processed/{log_type}_logs' + } + + +# Check server logs +qn += 1 +result = check_logs('server') +if result['status'] == 1: + s.append({'question': f'Part 1 - Q{qn:d}', 'status': 1}) +else: + s.append({'question': f'Part 1 - Q{qn:d}', 'status': 0, 'comment': result['comment']}) + +############################################################################################################ +# Step 6: Check that user logs and event logs were copied appropriately +qn += 1 + +result_user = check_logs('user') +result_event = check_logs('event') + +if result_user['status'] == 1 and result_event['status'] == 1: + s.append({'question': f'Part 1 - Q{qn:d}', 'status': 1}) +else: + comments = [] + if result_user['status'] == 0: + comments.append(result_user['comment']) + if result_event['status'] == 0: + comments.append(result_event['comment']) + s.append({'question': f'Part 1 - Q{qn:d}', 'status': 0, 'comment': '; '.join(comments)}) + +############################################################################################################ +# Step 7: Check that files containing 'ipaddr' in the filename were removed from 'data/raw' and 'data/processed/user_logs' +qn += 1 + +ipaddr_files_raw = glob.glob(os.path.join(working_dir, 'data/raw/*ipaddr*')) +ipaddr_files_user_logs = glob.glob( + os.path.join(working_dir, 'data/processed/user_logs/*ipaddr*')) + +if not ipaddr_files_raw and not ipaddr_files_user_logs: + s.append({'question': f'Part 1 - Q{qn:d}', 'status': 1}) +else: + comments = [] + if ipaddr_files_raw: + comments.append( + 'One or more files with ipaddr in data/raw not removed.') + if ipaddr_files_user_logs: + comments.append( + 'One or more files with ipaddr in data/processed/user_logs not removed' + ) + s.append({'question': f'Part 1 - Q{qn:d}', 'status': 0, 'comment': '; '.join(comments)}) + +############################################################################################################ +# Step 8: Check that 'data/inventory.txt' was created and contains all files in 'data/processed' subfolders +qn += 1 + +if os.path.isfile(os.path.join(working_dir, 'data/inventory.txt')): + with open(os.path.join(working_dir, 'data/inventory.txt'), 'r') as f: + inventory_files = [line.strip() for line in f.readlines()] + + # Now, find all files in 'data/processed' and its subfolders + processed_files = [] + for root, dirs, files in os.walk( + os.path.join(working_dir, 'data/processed')): + for name in files: + processed_files.append(name) + + files_in_inventory = [ + any([f in x for x in inventory_files]) for f in processed_files + ] + + missing = list(compress(processed_files, [not x for x in files_in_inventory])) + if len(missing) > 0: + print('Missing files in inventory: ' + ', '.join(missing)) + + # check if data/raw is present in the inventory + data_raw_present = any(['data/raw' in x for x in inventory_files]) + + if data_raw_present: + s.append({ + 'question': + f'Part 1 - Q{qn:d}', + 'status': + 0, + 'comment': + 'data/inventory.txt appears to list the contents of data/raw' + }) + elif all(files_in_inventory): + s.append({'question': f'Part 1 - Q{qn:d}', 'status': 1}) + else: + s.append({ + 'question': + f'Part 1 - Q{qn:d}', + 'status': + 0, + 'comment': + 'data/inventory.txt does not contain all files in data/processed' + }) +else: + s.append({ + 'question': f'Part 1 - Q{qn:d}', + 'status': 0, + 'comment': 'data/inventory.txt does not exist' + }) + +############################################################################################################ +# Step 2-1: Check if the coworker's commit ID is in the commit history + +try: + # Check if commit_id is in git rev-list HEAD using grep and wc -l + result = is_commit_in_branch( + github_repo_owner, + github_repo_name, + github_repo_branch, + 'UofT-DSI', + 'shell', + 'coworker-changes', + ) + if result: + s.append({'question': 'Part 2', 'status': 1}) + else: + s.append({ + 'question': + 'Part 2', + 'status': + 0, + 'comment': + f'`coworker-changes` branch not found in commit history' + }) + +except Exception as e: + s.append({ + 'question': 'Part 2', + 'status': pd.NA, + 'comment': f'Error checking git commit history.' + }) + print(f"Error checking git commit history: {e}") + +############################################################################################################ + +### Postprocessing ### +df = pd.DataFrame(s) +if 'comment' in df.columns: + df['comment'] = df['comment'].fillna('') + +# compute percentage correct +correct = df['status'].sum() +total = df.shape[0] + +# output the score table +df['status'] = df['status'].replace({1: status_c, 0: status_i, pd.NA: status_u}) +if github_token: + df.to_markdown(github_step_output, index=False) + +# also display markdown to console +render_md = df.to_markdown(index=False) +print(render_md) + +# create GitHub comment with markdown +if github_token: + headers = { + "Authorization": f"Bearer {github_token}", + "Accept": "application/vnd.github+json" + } + comment_body = "## Autograder results\n" + render_md + if correct == total: + comment_body += "\n\nAll autograder tests passed. This submission is approved by the autograder." + + post_to_github( + f"https://api.github.com/repos/{github_repo_owner}/{github_repo_name}/issues/{github_pr_number}/comments", + {"body": comment_body}, + headers, + "comment") + + if correct != total: + # request changes to the PR + post_to_github( + f"https://api.github.com/repos/{github_repo_owner}/{github_repo_name}/pulls/{github_pr_number}/reviews", + { + "event": "REQUEST_CHANGES", + "body": "Autograder checks failed. Please address the issues listed in the results comment." + }, + headers, + "request changes review") + +else: + print("GitHub token not found. Skipping comment creation.") + +if correct == total: + print('All tests passed!') + exit(0) +else: + print(f'Only {correct}/{total} tests passed.') + exit(0) diff --git a/03_instructional_team/autograder/requirements.txt b/03_instructional_team/autograder/requirements.txt new file mode 100644 index 0000000..60edec3 --- /dev/null +++ b/03_instructional_team/autograder/requirements.txt @@ -0,0 +1,3 @@ +pandas +tabulate +requests diff --git a/03_instructional_team/bash_customizations_for_teaching.md b/03_instructional_team/bash_customizations_for_teaching.md new file mode 100644 index 0000000..a7a23be --- /dev/null +++ b/03_instructional_team/bash_customizations_for_teaching.md @@ -0,0 +1,29 @@ +## ⚙️ Optional extras for customizing bash +These commands are what I use to setup my environment for teaching and demonstrations. +* These modifications are **not recommended** for learners + +⚠️ Beware: mistakes when editing your profile script can **cause significant issues and impact your ability to use the terminal**. ⚠️ + +On Windows, these commands should be appended to `~/.bash_profile`. +On MacOS, these commands should be appended to `~/.zshrc`. + +#### Append to the shell profile for extra spacing between previous commands and the subsequent shell prompt +``` +export PS1="\n\n$PS1" +``` + +#### Append to the shell profile to track live command history +``` +export PROMPT_COMMAND="history -a; $PROMPT_COMMAND" +alias showhistory="clear; tail -n0 -F ~/.bash_history | nl" +``` +* clear: clear the terminal +* tail: command for displaying the last few lines of a file +* -n0: display no lines currently in the file +* -F: watch the file for changes and display any new lines +* ~/.bash_history: command history file for bash +* | nl: line numbering + +#### To display live updated command history, run this in a separate window: +showhistory + diff --git a/03_instructional_team/githubpages/README for interactive problems.md b/03_instructional_team/githubpages/README for interactive problems.md new file mode 100644 index 0000000..5bd9fd6 --- /dev/null +++ b/03_instructional_team/githubpages/README for interactive problems.md @@ -0,0 +1,42 @@ +# Interactive Problems Rendering System + +The interactive problem set rendering system was developed by Simeon Wong for the Data Sciences Institute. + +The page is served through GitHub pages, and loads problem sets encoded in a YAML file with the following format: +```yaml +title: Problem Set Title +Problems + - title: Problem 1 + description: Solve this problem + questiontype: freeform + solution_template: `example code` + solutions: Long form solution with explanations + + - title: Problem 2 + description: Solve this problem + questiontype: parsons + options: + - code line 1 + - code line 2 + - code line 3 + - code line 4 + correct_options: [0, [1, 2], 3] + solutions: Long form solution + + - title: Problem 3 + description: Solve this problem + questiontype: multiplechoice + options: + - option 1 + - option 2 + correct_options: [0] + solutions: Long form solution +``` + +This system provides instructors with a convenient way to setup self-contained problem sets hosted entirely within a GitHub repository. + +### Setup +1. Copy `03_instructional_team/githubpages/interactive_problems.html` and `.github/workflows/static.yml` into your repository +2. Enable [GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#managing-github-actions-permissions-for-your-repository) and [GitHub Pages](https://docs.github.com/en/pages/getting-started-with-github-pages/configuring-a-publishing-source-for-your-github-pages-site#publishing-with-a-custom-github-actions-workflow) +3. Create a YAML file with your problems [[Example]](/03_instructional_team/githubpages/shell_basics.yml) +4. Create a Markdown file in your `02_activities/homework` folder linking to the interactive problems page [[Example]](02_activities/homework/shell_basics.md) diff --git a/03_instructional_team/githubpages/shell_basics.yml b/03_instructional_team/githubpages/shell_basics.yml new file mode 100644 index 0000000..7506e89 --- /dev/null +++ b/03_instructional_team/githubpages/shell_basics.yml @@ -0,0 +1,321 @@ +title: Shell Basics Practice Problems +problems: + - title: "Creating Directories" + description: | + Which command would you use to create a directory named `dir1`? + questiontype: multiplechoice + options: + - "`mkdir dir1`" + - "`touch dir1`" + - "`cd dir1`" + - "`ls dir1`" + correct_options: [0] + solution: | + The correct command to create a directory named `dir1` is: + ```bash + mkdir dir1 + ``` + + * `touch dir1` would create a file named `dir1`, not a directory. + * `cd dir1` would change into a directory named `dir1`, but it doesn't create it. + * `ls dir1` would list the contents of a directory named `dir1`, but it doesn't create it. + + - title: "Creating Directories" + description: | + How would you create directories named `dir2` and `dir3`, within a new directory named `dir1`? + questiontype: multiplechoice + options: + - | + ``` + mkdir dir1/dir2 dir1/dir3 + ``` + - | + ``` + mkdir dir1 + mkdir dir1/dir2 dir1/dir3 + ``` + - | + ``` + mkdir dir1 + mkdir dir2 dir3 + ``` + - | + ``` + cd dir1 + mkdir dir2 dir3 + ``` + - | + ``` + mkdir dir1 + cd dir2 + mkdir dir3 + ``` + correct_options: [1] + solution: | + First, we must create `dir1`, and then we can create `dir2` and `dir3` inside it. + + Alternatively, we can use the `mkdir -p` command to create the parent directory and subdirectories in one go: + ```bash + mkdir -p dir1/dir2 dir1/dir3 + ``` + This command would creates `dir1` and its subdirectories `dir2` and `dir3` in a single step. + + + The other options are incorrect. + ``` + mkdir dir1 + mkdir dir2 dir3 + ``` + * This would create `dir2` and `dir3` in the current directory, not inside `dir1`. + + + ``` + cd dir1 + mkdir dir2 dir3 + ``` + * This would fail because `dir1` does not exist yet. + + + ``` + mkdir dir1 + cd dir2 + mkdir dir3 + ``` + * This would fail because `dir2` does not exist yet. + + + - title: "Verifying Directory Creation" + description: | + You've just created the directories above. + + For reference, this is your current terminal session: + ```bash + user@computer:~$ pwd + /home/user + user@computer:~$ + ``` + + How might you verify the directories were created successfully? + questiontype: multiplechoice + options: + - "`ls`" + - "`ls -r`" + - "`ls dir1`" + - "`ls dir2 dir3`" + - "`find dir1 dir2 dir3`" + - "`cat dir1/*`" + - "`less dir1 dir1/dir2 dir1/dir3`" + + correct_options: [1, 2] + solution: | + To verify the directories were created successfully, you can use: + `ls dir1` or `ls -r` + - `ls dir1` lists the contents of `dir1`, which should include `dir2` and `dir3`. + - `ls -r` lists all files and directories in the current directory and its subdirectories. + + The other options are incorrect because: + * `ls` would list the contents of the current directory, but not specifically check for `dir2`, or `dir3` which are inside `dir1`. + * `ls dir2 dir3` would fail because `dir2` and `dir3` are inside `dir1` and do not exist in the current directory. + * `find dir1 dir2 dir3` is not a valid command. + * `cat dir1/*` would attempt to display the contents of files in `dir1`, but there are no files yet. + * `less dir1 dir1/dir2 dir1/dir3` would attempt to display the contents of `dir1`, `dir1/dir2`, and `dir1/dir3`, but these are directories, not files. + + - title: "Creating Text Files" + description: | + After verifying you've created the directories, how would you create two text files named `file1.txt` and `file2.txt` inside a folder named `text_files`? + + This is your current terminal session: + ```bash + user@computer:~$ pwd + /home/user + user@computer:~$ ls + dir1 + user@computer:~$ + ``` + questiontype: parsons + options: + - "`mkdir text_files`" + - "`cd text_files`" + - "`touch file1.txt file2.txt`" + - "`touch file1 file2`" + - "`new file1.txt file2.txt`" + - "`echo file1.txt file2.txt`" + - "`mv file1.txt file2.txt`" + - "`cp file1.txt file2.txt`" + - "`less file1.txt file2.txt`" + - "`cat file1.txt file2.txt`" + correct_options: [0, 1, 2] + solution: | + To create the text files, you can follow these steps: + ```bash + mkdir text_files + cd text_files + touch file1.txt file2.txt + ``` + This creates a new directory named `text_files`, navigates into it, and creates two empty text files named `file1.txt` and `file2.txt`. + + The other options are incorrect because: + * `touch file1 file2` would create files without the `.txt` extension. + * `new file1.txt file2.txt` is not a valid command. + * `echo file1.txt file2.txt` would just print `file1.txt file2.txt` to the terminal, not create files. + * `mv file1.txt file2.txt` would move files, but they don't exist yet. + * `cp file1.txt file2.txt` would copy files, but they don't exist yet. + * `less file1.txt file2.txt` would attempt to display the contents of non-existent files. + * `cat file1.txt file2.txt` would attempt to display the contents of non-existent files. + + - title: "Appending Text to a File" + description: | + Suppose your terminal session looks like this: + ```bash + user@computer:~$ pwd + /home/user + user@computer:~$ ls text_files + file1.txt file2.txt + user@computer:~$ + ``` + You want to *append* the text "Bash is hard!" to `file1.txt`. + questiontype: parsons + options: + - "`cd text_files`" + - "`echo 'Bash is hard!' > file1.txt`" + - "`echo 'Bash is hard!' >> file1.txt`" + - "`cat file1.txt`" + - "`less file1.txt`" + - "`mv file1.txt file2.txt`" + - "`cp file1.txt file2.txt`" + - "`rm file1.txt`" + correct_options: [0, 2] + solution: | + To append text to `file1.txt`, you can use: + ```bash + cd text_files + echo 'Bash is hard!' >> file1.txt + ``` + This command navigates to the `text_files` directory and appends the text "Bash is hard!" to `file1.txt`. + + The other options are incorrect because: + * `echo 'Bash is hard!' > file1.txt` would overwrite the contents of `file1.txt`, not append to it. + * `cat file1.txt` and `less file1.txt` would display the contents of the file but not modify it. + * `mv file1.txt file2.txt` would rename or move the file, not append text. + * `cp file1.txt file2.txt` would copy the file, not append text. + * `rm file1.txt` would delete the file. + + - title: "Verifying Appended Text" + description: | + Suppose your terminal session looks like this: + ```bash + user@computer:~/text_files$ + ``` + After appending text to a file, how would you verify that your text has been successfully added to `file1.txt`? + questiontype: parsons + options: + - "`cat file1.txt`" + - "`echo file1.txt`" + - "`rm file1.txt`" + - "`ls file1.txt`" + - "`cd text_files`" + correct_options: [0] + solution: | + To verify that the text has been successfully added to `file1.txt`, you can use: + ```bash + cat file1.txt + ``` + This command displays the contents of `file1.txt`, allowing you to confirm that the text "Bash is hard!" is present. + + The other options are incorrect because: + * `cd text_files` is not necessary since you are already in the `text_files` directory. + * `echo file1.txt` would just print the filename, not its contents. + * `rm file1.txt` would delete the file. + * `ls file1.txt` would list the file, but not its contents. + + - title: "Deleting Files and Globs" + description: | + Suppose you are inside `dir2`. + This is your current terminal prompt: + ``` + user@computer:~$ ls dir1 + file1.txt file2.txt dir2/ dir3/ + user@computer:~$ cd dir1/dir2 + /home/user/dir1/dir2 + user@computer:~/dir1/dir2$ + ``` + How would you delete all `.txt` files within `dir1`? + + questiontype: multiplechoice + options: + - "`rm .txt`" + - "`rm dir1/*.txt`" + - "`rm dir1/dir2/.txt`" + - "`rm ../*.txt`" + correct_options: [3] + solution: | + To delete all `.txt` files within `dir1`, you can use: + ```bash + rm ../*.txt + ``` + This command removes all `.txt` files in the parent directory (`dir1`) from your current location inside `dir2`. + + The other options are incorrect because: + * `rm .txt` would not match any files. + * `rm dir1/*.txt` and `rm dir1/dir2/.txt` would fail because you are currently in `dir2` + + - title: "File and Directory Management: Navigating to the Parent Directory" + description: | + This is your current terminal session: + ```bash + user@computer:~/dir1/dir2$ pwd + /home/user/dir1/dir2 + user@computer:~/dir1/dir2$ ls + user@computer:~/dir1/dir2$ + ``` + After working inside one of the directories, how would you navigate back to your home directory? + questiontype: multiplechoice + options: + - "`cd ..`" + - "`cd ../..`" + - "`cd`" + - "`cd /home/user`" + - "`cd dir1`" + correct_options: [1, 2, 3] + solution: | + To navigate back to your home directory, you can use: + - `cd` - This command takes you directly to your home directory. + - `cd /home/user` - This command takes you to the absolute path /home/user, which is your home directory. + - `cd ../..` - This command takes you two levels up from your current directory, which would also take you to your home directory. + + These other options are incorrect because: + - The command `cd ..` would take you one level up to `dir1`, not directly to your home directory. + - The command `cd dir1` would attempt to change into the `dir1` directory which will fail because there is no `dir1` in the current directory. + + - title: "Removing Directories and Their Contents" + description: | + This is your current terminal session: + ```bash + user@computer:~$ ls + Desktop/ dir1/ Documents/ Downloads/ Music/ Pictures/ Videos/ + user@computer:~$ + ``` + How would you remove directory `dir1` and all its contents, including `dir2`, `dir3`, and their files? Then confirm that all directories and files have been deleted. + questiontype: multiplechoice + options: + - "`rm dir1`" + - "`rm -r dir1`" + - "`rm -r dir1 dir2 dir3`" + - "`ls`" + - "`ls -r dir1`" + - "`ls dir1/*`" + correct_options: [1] + solution: | + To remove `dir1` and all its contents, including `dir2`, `dir3`, and their files, you can use: + ```bash + rm -r dir1 + ``` + This command recursively removes the directory and all its contents. + + The other options are incorrect because: + * `rm dir1` would fail because `dir1` is a directory and requires the `-r` option to remove it. + * `rm -r dir1 dir2 dir3` would attempt to remove `dir1`, `dir2`, and `dir3` but would fail because they are not in the current directory. + * `ls` would list the contents of the current directory, but not confirm that `dir1` has been deleted. + * `ls -r dir1` would list the contents of `dir1`, but it doesn't confirm that it has been deleted. + * `ls dir1/*` would list the contents of `dir1`, but it doesn't confirm that it has been deleted. + diff --git a/03_instructional_team/githubpages/shell_moderate.yml b/03_instructional_team/githubpages/shell_moderate.yml new file mode 100644 index 0000000..417bbbc --- /dev/null +++ b/03_instructional_team/githubpages/shell_moderate.yml @@ -0,0 +1,248 @@ +title: Unix Shell Moderate Practice Problems +problems: + - title: "File and Directory Management 1" + description: | + You are working on a project and need to create a directory named `data` inside your project directory (`project3`) + and a file named `info.txt` inside it. + + This is your current terminal session: + ```bash + user@computer:~/project3/scripts$ pwd + /home/user/project3/scripts + + user@computer:~/project3/scripts$ + ``` + + Select and order the following commands to accomplish this task. + questiontype: parsons + options: + - "`cd ..`" + - "`mkdir data`" + - "`cd data`" + - "`touch info.txt`" + - "`ls`" + - "`echo info.txt`" + - "`cd scripts`" + correct_options: [0, 1, 2, 3] + solution: | + ```bash + user@computer:~/project3/scripts$ cd .. + user@computer:~/project3$ mkdir data + user@computer:~/project3$ cd data + user@computer:~/project3/data$ touch info.txt + ``` + + We start off in the `scripts` directory. + - First, we need to navigate to the project root directory using `cd ..` + - (`project3` is the parent directory of `scripts`) + - Then, we can create the `data` directory and navigate into it. + - Finally, we create the `info.txt` file. + + - we can also list the contents of the current directory using `ls` to check if the `data` directory and + `info.txt` file was created successfully + + + + - title: "File and Directory Management 2" + description: | + Create a directory structure for a project with the following hierarchy: + ``` + project/ + ├── src/ + │ ├── main.py + │ └── utils.py + ├── data/ + │ ├── input.csv + │ └── output.csv + └── README.md + ``` + questiontype: freeform + solution: | + To create the directory structure, you can use the following commands. + ```bash + mkdir -p project/src project/data + touch project/src/main.py project/src/utils.py project/data/input.csv project/data/output.csv project/README.md + ``` + - Since the problem asks us to create this in the current directory, we don't need to `cd` anywhere. + - `mkdir -p` creates the directories and any necessary parent directories. + - `touch` creates the files in the specified directories. + + **Alternative solution:** + ```bash + mkdir project + mkdir project/src + touch project/src/main.py + touch project/src/utils.py + mkdir project/data + touch project/data/input.csv + touch project/data/output.csv + touch project/README.md + ``` + - In this longer example, we create the `project` directory first, then create the `src` and `data` directories inside it. + - `touch` is called individually for each file + + Additionally: + - `cd` can also be used to change into each directory before creating files, but it is not necessary + + + + - title: "Output Redirection" + description: | + Which command correctly redirects the standard output of `ls` to a file named `filelist.txt` without overwriting? + questiontype: multiplechoice + options: + - "ls > filelist.txt" + - "ls < filelist.txt" + - "ls >> filelist.txt" + - "ls filelist.txt" + correct_options: [2] + solution: | + The correct command is: + ```bash + ls >> filelist.txt + ``` + This sends the output of `ls` into `filelist.txt`, without overwriting the file if it exists. + + - `ls > filelist.txt`: This command **overwrites** the contents of `filelist.txt` with the output of `ls`. + - `ls < filelist.txt`: This command tries to read from `filelist.txt` as input, which is not what we want. (*not covered in the main session*) + - `ls filelist.txt`: This command tries to list the contents of `filelist.txt`, instead of writing to it. + + + - title: "Debugging 1" + description: | + You are trying to run a script but encounter an error. The script is supposed to print the current date and time. + However, it fails with the following error: + ``` + bash: ./script.sh: No such file or directory + ``` + What is the likely cause of this error? + questiontype: multiplechoice + options: + - The script does not exist in the current directory. + - The script does not have execute permissions. + - The script is not in your PATH. + - There is a syntax error in the script. + correct_options: [0] + solution: | + The script does not exist in the current directory. + + - The script does not have execute permissions: If this were the case, the error would be `permission denied`, not `No such file or directory`. + - The script is not in your PATH: The error message indicates that the script is being called with `./`, which means it is expected to be in the current directory, not in the PATH. + - There is a syntax error in the script: A syntax error would occur after the script is found and executed, not before. + + - title: "Debugging 2" + description: | + Suppose this is the contents of a directory on your computer: + ```bash + user@computer:~$ ls ~ + myfile.txt mydata.csv myscript.sh myfolders/ + ``` + + When you type the following commands, you get an error: + ```bash + user@computer:~$ mv mydata.csv myfolder/ + mv: cannot move 'mydata.csv' to 'myfolder/': No such file or directory + user@computer:~$ mv myfile.txt myfolder/ + mv: cannot move 'myfile.txt' to 'myfolder/': No such file or directory + user@computer:~$ mv myscript.sh myfolder/ + mv: cannot move 'myscript.sh' to 'myfolder/': No such file or directory + ``` + + How can you fix this error? + questiontype: freeform + solution_template: | + mv mydata.csv myfolder/ + mv myfile.txt myfolder/ + mv myscript.sh myfolder/ + solution: | + ```bash + user@computer:~$ mv mydata.csv myfolders/ + user@computer:~$ mv myfile.txt myfolders/ + user@computer:~$ mv myscript.sh myfolders/ + ``` + + The error indicates that the directory `myfolder/` does not exist. + Ordinarily, we could create the directory using the `mkdir` command, but in this case, it looks like the directory is already created as `myfolders/`. + To fix the error, we can either create the directory `myfolder/` or move the files to `myfolders/` instead. + + Note the spelling of myfolder**s**/ + + - title: "Organizing files" + description: | + Suppose this is the contents of a folder on your computer: + ```bash + user@computer:~$ ls /data/MYPROJECT/scripts + analyze_data.py dataset1.db dataset2.db outputimg.png + outputimg2.png + ``` + + You've just opened a new terminal and want to organize the files by moving datasets and outputs in separate folders. + Select and order the following commands to accomplish this task. + questiontype: parsons + options: + - "`cd /data/MYPROJECT`" + - "`mkdir datasets`" + - "`mkdir outputs`" + - "`mv dataset* datasets/`" + - "`mv scripts/dataset* datasets/`" + - "`mv output* outputs/`" + - "`mv scripts/output* outputs/`" + - "`cd data`" + - "`cd outputs`" + - "`echo scripts/output*`" + - "`movefile dataset* datasets/`" + - "`movefile output* outputs/`" + - "`cd`" + - "`ls`" + correct_options: [0, [1, 2], [4, 6]] + + solution: | + ```bash + user@computer:~$ cd /data/MYPROJECT + user@computer:/data/MYPROJECT$ mkdir datasets + user@computer:/data/MYPROJECT$ mkdir outputs + user@computer:/data/MYPROJECT$ mv scripts/dataset* datasets/ + user@computer:/data/MYPROJECT$ mv scripts/output* outputs/ + ``` + + First, we need to navigate to the project directory, then create the `datasets` and `outputs` directories. + After that, we can move the dataset and output files into their respective directories. + + - Running `mv dataset* datasets/` won't work, because the `dataset*` files are in the `scripts` directory, not the current directory. + - The command `ls` can be run to check the contents of the current directory, but it is not necessary to accomplish the task. + + + - title: "Text files" + description: | + Suppose you have the following script: + ```bash + # preview all my data files + echo "Data file preview" + echo "=================" + cat mydata/*.txt + ``` + + When your project had fewer data files, this script worked fine. But now, the output is too long to read. + What are some approaches to fix this? + questiontype: freeform + solution: | + There are several ways to fix this issue. Here are a few options: + - Redirect the output to a file for later review with less + ```bash + cat mydata/*.txt > preview.txt + less preview.txt + ``` + + **More advanced strategies not covered in the main session:** + - Use the pipe operator to pipe the output to `less` directly. + ```bash + cat mydata/*.txt | less + ``` + - Use `head` or `tail` to preview only the first or last few lines of each file. + ```bash + head mydata/*.txt + tail mydata/*.txt + ``` + + + \ No newline at end of file diff --git a/03_instructional_team/markdown_slides/images/1-terminal.png b/03_instructional_team/markdown_slides/images/1-terminal.png new file mode 100644 index 0000000..9489e29 Binary files /dev/null and b/03_instructional_team/markdown_slides/images/1-terminal.png differ diff --git a/03_instructional_team/markdown_slides/images/2-shell.png b/03_instructional_team/markdown_slides/images/2-shell.png new file mode 100644 index 0000000..21c3562 Binary files /dev/null and b/03_instructional_team/markdown_slides/images/2-shell.png differ diff --git a/03_instructional_team/markdown_slides/images/3-gitbash.png b/03_instructional_team/markdown_slides/images/3-gitbash.png new file mode 100644 index 0000000..9b56096 Binary files /dev/null and b/03_instructional_team/markdown_slides/images/3-gitbash.png differ diff --git a/03_instructional_team/markdown_slides/images/4-windows_shells.png b/03_instructional_team/markdown_slides/images/4-windows_shells.png new file mode 100644 index 0000000..5388aa5 Binary files /dev/null and b/03_instructional_team/markdown_slides/images/4-windows_shells.png differ diff --git a/03_instructional_team/markdown_slides/images/5-windows_terminals.png b/03_instructional_team/markdown_slides/images/5-windows_terminals.png new file mode 100644 index 0000000..9284555 Binary files /dev/null and b/03_instructional_team/markdown_slides/images/5-windows_terminals.png differ diff --git a/03_instructional_team/markdown_slides/images/terminology_explainer.afdesign b/03_instructional_team/markdown_slides/images/terminology_explainer.afdesign new file mode 100644 index 0000000..97d2dc9 Binary files /dev/null and b/03_instructional_team/markdown_slides/images/terminology_explainer.afdesign differ diff --git a/03_instructional_team/markdown_slides/images/terminology_explainer.afdesign~lock~ b/03_instructional_team/markdown_slides/images/terminology_explainer.afdesign~lock~ new file mode 100644 index 0000000..cce471a Binary files /dev/null and b/03_instructional_team/markdown_slides/images/terminology_explainer.afdesign~lock~ differ diff --git a/03_instructional_team/markdown_slides/optional_unix_slides.md b/03_instructional_team/markdown_slides/optional_unix_slides.md new file mode 100644 index 0000000..5ddc4b9 --- /dev/null +++ b/03_instructional_team/markdown_slides/optional_unix_slides.md @@ -0,0 +1,1766 @@ +--- +marp: true +theme: dsi_certificates_theme +paginate: true +--- +# Advanced Unix Shell + +``` +$ echo "Data Sciences Institute" +``` + +--- +# Navigate Files / Directories + +--- +## Files + +Knowing the different types of files available helps us better +understand how to navigate and manipulate them. + +- Regular files are text files with readable characters. +- Executable files are programs that are invoked as commands. +- Shell scripts are readable executable files, in contrast to bash, which is a non-human-readable executable + +--- +## Directories + +Directories, similar to folders, contain files and subdirectories, forming a hierarchical structure. + +- We can think of the structure of directories as a tree with the + top of the tree being the root. +- All files can be named and found in relation to the root by + listing the directory names in order from the root, separated + by slashes, followed by the file's name. + +--- +## Paths + +Using `cd` and `pwd`, let's explore how to use absolute and relative pathnames. + +``` +$ cd +$ pwd +``` + +``` +$ cd Desktop +$ pwd +``` + +``` +$ cd 15 +$ pwd +``` + +--- +Let's now try move through some directories to get comfortable. +Try out lots of different paths depending on the file structures +of your computer. Try getting into different directories from +different parent directories. The tilde notation ~ in the +examples below refer to our home directory. + +--- +``` +$ cd ~/Desktop +$ pwd +``` + +``` +$ cd ~/Desktop/dir1 +$ pwd +``` + +--- +## Options and Arguments + +Options and arguments are used to write commands that can +make changes to our system. The syntax is: + +``` +$ command -—option argument +``` + +--- +Options can also be combined, a topic we'll briefly touch on now and explore in more detail later. + +--- +There are two ways to write an `--option`: + +1. Short option: one dash followed by a single character +2. Long option: two dashes followed by a word + Some examples: + `-a` or `--all` + `-d` or `--directory` + `-r` or `--reverse` + +--- +Let's try these lines of code and see what happens: + +``` +$ ls -l +``` + +``` +$ ls -lt +``` + +``` +$ ls -lt --reverse +``` + +--- +- `—l` long format +- `—t` modification time +- `—reverse` reverse the sort order + +Notice how the `-lt` command combines multiple options. + +--- +## Wildcards + +Wildcards allow us to quickly specify groups of filenames based on character patterns. Let's look at a few examples below: + +--- +- `*` > matches any character +- `?` > matches any single character +- `[characters]` -> matches any character that is in the set +- `[!'characters]` -> matches any character that is not in the set + +Some other helpful character wildcards are: + +- `[:digit:]` > matches any numeral +- `[:lower]` ~ matches any lowercase letter +- `[:upper:]` > matches any uppercase letter + +--- +Let's try a few in our terminal: + +``` +$ ls +``` + +``` +$ ls * +``` + +``` +$ ls ax.txt +``` + +``` +$ ls [abc]x +``` + +``` +$ ls [[:upper:]]x +``` + +``` +$ ls [![:digit:]]x +``` + +--- +## Input / Output + +--- +### Standard Input/Output + +Each program uses standard input, output, and error channels. We can think of the standard input default as coming from the keyboard and if we think of everything as a file, a command such as `ls` will result in a file called `standard output` and the status message to a file called `standard error`. By default, both are linked to the screen and not saved to a disk file. + +--- +### Input/Output Redirection + +Input/Output redirection allows us to change where the input +comes from and where the output goes to, such as storing the +output of a command into a file. We can do this using the +redirection operator `>`. + +``` +$ ls -l /usr/bin > ls—output.txt +``` + +--- +Here we have redirected the output of `ls -—l /usr/bin` to a `.txt` file called _ls-output.txt_. + +We can now see the details of that file and if it worked: + +``` +$ ls -l ls-output.txt +``` + +--- +By examining the details, we can see that the file was created and is a fairly large text file, indicating that content was written to it. + +If we specify a directory that does not exist, we receive the Standard error: + +``` +$ ls -l /bin/usr > ls—-output.txt +``` + +--- +Why was the standard error not written to the `.txt` file? +What happened to our _ls-output.txt_ file? + +--- +Although the standard error was not written to the `.txt` file, +the destination file is always written from the beginning, +therefore, the redirection began to write the file and once +noticed there was an error, stopped, resulting in an empty file. + +--- +So how do we append rather than rewrite? By using the +redirection operator `>>`. + +``` +$ ls -l /usr/bin >> ls—-output. txt +``` + +--- +If we want to redirect the standard error, we need to use the +redirection operator `2>` + +``` +$ ls -l /bin/usr 2> ls—-error.txt +``` + +--- +If we want to redirect both the standard output and standard +error to one file, we have two options. + +1. Use `2>&1` at the end of the command. + +``` +$ ls -l /bin/usr > ls-output.txt 2>&1 +``` + +2. Use `&>` inplace of `>` + +``` +$ ls -l /bin/usr & 1ls-output. txt +``` + +--- +### cat + +`cat` takes one or more files and copies them to standard output. Using the previously created `ls-output.txt`, we can see how this is done: + +``` +$ cat ls—output.txt +``` + +--- +We can also use it to join files togther. Let's say | have two files, `file1` and `file2` and I want to combine them into a file called `file3`: + +``` +$ cat filel file2 > file3 +``` + +--- +Now the contents of file1 and file2 should be combined. + +We can also use `cat` to add to a `.txt` file. + +``` +$ cat > new_cat.txt +``` + +--- +Now we can type the desired text into the file. To finish, use `CTRL-D` to exit. + +What is the difference between `$ cat > new_cat.txt` and `$ cat >> new_cat.txt` ? + +Finally, we can redirect the standard input from the keyboard to the file _new_cat.txt_ + +``` +$ cat < new_cat.txt +``` + +This is almost identitcal to just typing _$ cat new_cat.txt_ but we can see later how it could be more useful. + +--- +# Pipes / Filters + +--- +Pipelines read data from standard output and send it to standard input using the pipe operator `|` . This means the standard output of one command can be piped into the standard input of another. + +Several commands put together in a pipeline are often referred to as filters. Filters take an input, change it and then output it. + +--- +## Commands + +Let's learn a few more commands that will help us further +understand pipelines and filters. We'll learn: + +- extract columns from output `cut` +- sort lines of text `sort` +- report or omit repeated lines `uniq` +- print lines matching a patter `grep` +- search directories and subdirectories for files `find` +- ouput the first part of a file `head` +- output the last part of afile `tail` + +--- +### cut + +Let's look at a `csv` to see how we Can initially see our data. Because it's a `csv`, each line is separated by a comma. Let's first read that file using `cat`: + +``` +$ cat parking_data.csv +``` + +We'll see a lot of text, so let's make some sense of it using cut. + +To use cut, I need to pass a couple options: + +1. `-d` which cuts the text based on what follows. For example, `—d:` will cut based on colons or `-d" "` will cut based on a + space. + +2. `-f`, which extracts a particular field based on what follows. For example, `-f1` will take the first field or `-f2` will take the second field and so on. + +--- +In this example, I'm taking the file _parking_data_ and cutting it based on colons and then only extracting the first field. + +``` +$ cut -d, -f1 < parking_data.csv +``` + +What happens if I add another `-f` option? What does this do? + +``` +$ cut -d, -f1 -f2 < parking_data.csv +``` + +--- +How would I specify more than three fields? + +--- +### sort + +How can we make our previous example more readable? + +One answer is to use the sort feature. We can pipe this with the cut feature: + +``` +$ cut -d, -f1 < parking_data.csv | sort +``` + +--- +### uniq + +Additionally, I can make the above command even more readable by removing any duplicates with `uniq` + +``` +$ cut -d, -f1 < parking_data.csv | sort | uniq +``` + +--- +### grep + +`grep` is a powerful tool for finding patterns in text files. The syntax is: + +``` +$ grep pattern [file...] +``` + +--- +In our case, we're going to use it with our previous example and +pipe it with other commands: + +``` +$ cut -d, -f1 parking_data.csv | sort | uniq | grep FIRE +``` + +The results are all patterns of FIRE in the text file. + +--- +### find + +Another useful use for `grep` is to find files in directories. `grep` is nicely combined with `find` for this feature. + +``` +$ find ~/Desktop/dir1 | grep cat +``` + +--- +Here we're searching in the directory _dir1_ with the pattern _cat_. This would be helpful if we wanted to know if there were any files with the word cat in the filename. + +--- +### head | tail + +We can also extract the first and last part of files using `head` and `tail`. We can also add the option `-n` followed by a number to extract a certain number of lines. + +``` +$ head -n 5 ls-output.txt +``` + +``` +$ tail -n 5 ls-output.txt +``` + +--- +`head` and `tail` can also be used in pipelines: + +``` +$ cut -d, -f1 < parking_data.csv | sort | uniq | head —-n 5 +``` + +``` +$ cut -d, -f1 < parking_data.csv | sort | unig | tail -n 5 +``` + +--- +## Expansions + +--- +Expansion uses special characters to expand upon something before the shell processes it. We have learned a few expansions so far such as the tilde `~` and wildcards `*`. We've also seen some character wildcards `[characters]`. + +Expansions are another feature that help us when we're manipulating and working with files and directories. + +--- +Other examples of expansions are: + +- arithmetic expansion +- brace expansion + +--- +### Arithmetic Expansion + +Arithmetic expansion basically makes the shell a calculator. The syntax is: + +``` +$((expression)) +``` + +For example: + +``` +$ echo $((2 + 2)) +``` + +--- +Arithmetic expressions can be nested: + +``` +$ echo $(($((2 + 2)) * 3)) +``` + +--- +Just for reference, here is a list of the arithmetic operators: + +| Operator | Description | +| -------- | ---------------- | +| `+` | Addition | +| `-` | Subtraction | +| `*` | Multiplication | +| `/` | Integer division | +| `**` | Exponentiation | + +--- +### Brace Expansion + +Brace expansions allow us to create multiple text strings from a pattern containing braces. Here are a few examples: + +``` +$ echo Test-{A,B,C}-Example +``` + +``` +$ echo Number_{1..5} +``` + +``` +$ echo {Z..A} +``` + +--- +Brace expansions can also be nested: + +``` +$ echo a{A{1,2},B{3,4}}b +``` + +--- +We can use brace expansion to help make multiple directories using `mkdir`. + +``` +$ mkdir dir-{1..3} +``` + +This command makes 3 directories named _dir-1_, _dir-2_, and _dir-3_ + +--- +## Quoting / Backslashing + +Quoting suppresses unwanted expansions. We can use double +quotes, single quotes or backslashes: + +- Double quotes force special characters to lose their meaning + and are treated as ordinary characters except for `*` , `\` , and `'`. +- Single quotes suppress all expansion +- Backslashes are used to escape single characters + +--- +Many times there will be file names or directories that are named with spaces. In this case, we'll need to use double quotes so that the shell can read it. Using `touch` we can create a text file named something separated with two words: + +``` +$ touch "two words.txt" +``` + +--- +We can then see the details of the file we just created: + +``` +$ ls -l "two words.txt" +``` + +--- +If we want to rename the text, we would do as follows: + +``` +$ mv "two words.txt" two_words.txt +``` + +--- +Let's see what these three examples do in shell: + +``` +$ echo '2 * 3 > 5 is an equation’ +``` + +``` +$ echo '2 * 3 > 5' is an equation +``` + +``` +$ echo 2 \x 3 \> 5 is an equation +``` + +--- +## Command Line Editing + +Getting familiar with command line editing can save you time. Bash uses a library called Redline to use command line editing +There are many shortcuts and you don’t have to memorize them all, just use the ones that you feel are best. There are even more shortcuts that you can read about in the textbooks! + +--- +### Command Character + +| Command | Description | +| ------- | --------------------------------------- | +| CTRL-B | Move one character backwards | +| CTRL-F | Move one character forwards | +| DEL | Delete one character backwards | +| CTRL-D | Delete one character at cursor location | + +--- +### Word Commands + +| Command | Description | +| ------- | ------------------------- | +| ESC-B | Move one word backwards | +| ESC-F | Move one word forwards | +| ESC-DEL | Delete one word backwards | +| ESC-D | Delete one word forwards | +| CTRL-Y | Undo | + +--- +### Line Commands + +Command Description +| Command | Description | +| ---------- | ------------------------------------------ | +| CTRL-A | Move to the beginning of the line | +| CTRL-E | Move to the end of the line | +| CTRL-K | Delete text from the cursor to the end of the line | +| CTRL-U | Delete text from the cursor to the beginning of the line | + +--- +### History Line Commands + +| Command | Description | +| ---------- | ----------------------------------------------------- | +| CTRL-P | Move to the previous line in your history of commands | +| CTRL-N | Move to the next line in your history of commands | +| `!!` | Repeat the last command | +| `!number` | Repeat history list item number | +| `!string` | Repeat last history item starting with string | +| `!?string` | Repeat last history item containing string | + +--- +## Completion Command + +Completion commands autocomplete your command if it exists by hitting `tab`. If it does not exist, the command will not be able to complete. + +If multiple exist, the command will also not be able to complete because it will not know which one to choose. + +--- +For example, let's say we have two files called `file1` and `file2`. If would not be able to use autocomplete because the shell will not know which to choose until the last character. + +If we have two files, one called `foot.txt` and one called `file.txt`. This command would not be able to autocomplete: + +``` +$ ls f +``` + +--- +But this one will: + +``` +$ ls fil +``` + +--- +# Shell Scripts + +--- +### Shell Scripts + +Shell scripts allow us to combine several commands into one file, rather than one by one on the command line. + +The shell will read the script just as if you were to write the command on the command line. + +Most things that can be done in the shell script can be done on the command line and vice versa. + +--- +### Writing Shell Scripts + +There are three important considerations when writing the shell script + +1. **Write a script:** scripts are ordinary text files. You can use a text editor that will provide syntax highlighting (color coding elements of the script). It can help find errors but writing in TextEdit is possible. + +2. **Make a shell script executable:** set the script permissions to allow it to be executed + +3. **Put the shell script somewhere the shell can find it:** the shell script automatically searches certain directories for executable files when no explicit pathname is specified. + +--- +### Set Up + +Open either TextEdit or your text editor of choice. Some popular programs are Sublime Text, Vim, Atom, and Notepad++. +If you want to see the syntax highlighting, you might have to save your script as a `.sh` file. Without doing this, your file will just look like a regular `.txt` file. +Once you open your text editor and save it, we can begin our first script! + +--- +### Script File Format + +We must first tell the shell the name of the interpreter that should be used to execute the script. This is marked by using a shebang: `#!` +Throughout the script, you can and **should** use `#` to make comments. Comments make your code more readable and can help you understand your code when you come back to it. + +``` +#!/bin/bash +# this is our first comment +echo "This is our first script!" +``` + +--- +Here we can see we've told the shell to use `/bin/bash` using the shebang `#!` +We've also added a comment using `#` +And finally, something quite familiar, we have our first line of script using `echo` + +--- +### A Note on Commenting + +Commenting is important not just so you can understand your own work, but also so others can understand your work in collaborative projects. It also helps make your code reproducible. + +Comments can be inline: + +``` +echo "Hello World" #this is an inline comment +``` + +or as comment blocks: + +``` +#this is a comment block +echo "Hello World" +``` + +--- +### Executable File Permission + +In order to execute our file, we have to add file permissions: `chmod` helps make our script executable, `775` is used to make scripts that everyone can execute, `700` is used to make scripts that only the owner can execute + +Here, `chmod` is combined with `775` so that everyone can execute the script: + +``` +$ ls -l first_script.sh +``` + +``` +$ chmod 775 first_script.sh +``` + +--- +### Script File Location + +In order to run our script, we have to call it using `./` in front of the script filename ( `./Script` ). +File location is important to run your script. If just `Script` was written, the shell would not be able to find the script and try to read it as a command, outputting `command not found`. +Running `echo $PATH` helps us see what directories are being searched for the script. + +--- +If we want to run our script without `./`, we can create a `/bin` for our script, move our script into the bin folder and then run it. It's important to note that we have to make this bin in our home directory. If we made it on our Desktop, the script would still not be found. + +``` +$ mkdir bin +$ mv first_script.sh bin +$ first_script.sh +``` + +--- +In this block of code, we're making the bin folder using `mkdir`, +moving the script into the bin with `mv` and then running the +script without `./`. + +--- +### Good Locations for Scripts + +For personal use, a good place to put your script is `/bin`. +For everyone's access, it's better to put scripts in `/usr/local/bin`. + +--- +# Shell Functions + +--- +### Functions + +Functions are a good way to break down code into smaller, more manageable chunks. Each chunk can represent a task. + +For example, let's say your entire process is make pasta. It can be broken down into: + +1. Prepare vegetables +2. Make sauce +3. Cook pasta +4. Serve + +--- +Each of these steps can be expanded further into subprocesses. Cook pasta can be: + +1. Fill pot with water +2. Boil water +3. Measure pasta +4. Add pasta to boiling water +5. Cook for 8-12 minutes +6. Strain + +--- +Functions have two syntactic forms: + +``` +function name { + commands + return +} +``` + +``` +name () { + commands + return +} +``` + +`name` is the name of the function +`commands` are the commands contained in the function + +--- +Let's write our first function: + +``` +#!/bin/bash +function funct { + echo "Step 2" + return +} + +--- +#program starts here + +echo "Step 1" +funct +Echo “Step 3" +``` + +--- +What do you think this function will output? + +--- +Let's save and run this function in our terminal to see what happens. + +Here's a good time to recap how to save, grant permissions, and run the script. + +- `chmod` - permissions command +- `775` - grant permissions to everyone +- `700` - grant permissions to yourself +- `/bin` - where to save permissions + +--- +## Variables + +--- +### Global Variables + +Let's make our script more complex with some variables. We can first define variables directly through the terminal. + +``` +$ foo="Something cool" +$ echo $foo +``` + +--- +Notice how in order to call the variable we need to add `$` before the variable. The quotes are not necessary if the value of the variable doesn't include spaces when defining it. If we did not include the quotes here, we would receive an error. + +--- +Now let's add some global variables to our script: + +``` +#!/bin/bash + +step="Step 2" + +function funct { + echo $step + return +} + +#program starts here + +echo, step yl” +funct +echo "Step 3" +``` + +What do we think will be the output in this example? + +--- +### Local Variables + +Local variables are variables that are contained within the function. Because they're contained, they can have names that already exist in the shell globally or within other shell functions. + +``` +#!/bin/bash + +foo=@ # global variable foo + +funct_1 () { + local foo # variable foo local to funct_1 + foo=1 + + echo "funct_1: foo = $foo" +} + +funct_2 () { + local foo # variable foo local to funct_2 + foo=2 + echo "funct_2: foo = $foo" +} + +echo "global: foo = $foo" +funct_1 +echo "global: foo = $foo" +funct_2 +echo "global: foo = $foo" + +``` + +--- +What would happen if we removed `local` ? + +``` +#!/bin/bash + +foo=@ # global variable foo +funct_1 () { + foo=1 + echo "funct_1: foo = $foo" +} + +funct_2 () { + foo=2 + echo "funct_2: foo = $foo" +} + +echo "global: foo = $foo" +funct_1 +echo "global: foo = $foo" +funct_2 +echo "global: foo $foo" +``` + +--- +## Parameters + +--- +### Positional Parameters + +Positional parameters are built-in parameters that allow our programs to get access to the contents of the command line. +This is extremely valuable when we are creating scripts and then want to pass a parameter through the script from the command line. + +If our code has more than 9 positional parameters, you need to enclose the positional parameter in curly brackets `${10}` + +Let's create a script to see how this works: + +``` +#!/bin/bash + +echo " +Number of arguments: $# +\$0 = $0 +\$1 = $1 +\$2 = $2 +\$3 = $3 +\$4 = $4 +\$5 = $5 +\$6 = $6 +\$7 = $7 +\$8 = $8 +\$9 = $9 +" +``` + +--- +In the example, you may notice that we haven't given `$0` any specific value. +Let's try to run the script a couple ways through the command line to see what this means: + +1. Run the script with arguments `a` `b` `c` `d`. +2. Run the script with any arguments of your choice. + +What do we notice? + +--- +### \$\* and $@ + +`$*` —> Expands into the list of positional parameters, starting with 1. When surrounded by double quotes, it expands into a double-quoted string containing all of the positional parameters, each separated by the first character of the IFS shell variable (by default a space character). + +`$@` —> Expands into the list of positional parameters, starting with 1. When surrounded by double quotes, it expands each positional parameter into a separate word surrounded by double quotes. + +--- +Let's take a look at this code piece by piece: + +``` +print_params () { + echo "\\$1 = $1" + echo "\\$2 = $2" + echo "\\$3 = $3" + echo "\\$4 = $4" +} +pass_params () { + echo -e "\\n" '$* :'3 print_params $x + echo -e "\\n" '"$x" :'s print_params '$x" + echo -e "\\n" '$@ :'; print_params $@ + echo -e "\\n" '"$@" :'; print_params '$@"' +} +pass_params "word" "words with spaces" +``` + +--- +1. Here we have two functions: `print_params ()` and `pass_params ()`. `pass_params ()` calls on the function `print_params ()` within its function. +2. In the first function, `echo` is printing the line inside the double quotes. The `\` infront of `$1` escapes the `$`, thus losing its meaning, as we learned earlier. + +``` +print_params () { + echo "\\$1 = $1" + echo "\\$2 = $2" + echo "\\$3 = $3" + echo "\\$4 = $4" +} +``` + +--- +3. In the second function, `echo` again is printing the line inside the single quotes. `"\n"` is adding a tab at the beginning of the line for readability. It is then calling on the first function ( `print_params ()` ) with the argument `$*` . The second echo is calling the first function but with the argument `$*` in double quotes. This is repeated for `$@` + +``` +pass_params () { + echo -e "\\n" '$x :'; print_params $x + echo -e "\\n" '"$x" :'s print_params '$x" + echo -e "\\n" '$@ :'; print_params $@ + echo -e "\\n" '"$@" :'; print_params '$@"' +} +``` + +--- +4. In the final part of the code, we're calling on the `pass_params ()` function and passing two arguments: `"word"` and `"words with spaces"`. + +``` +pass_params "word" "words with spaces" +``` + +--- +Let's see what happens's when we run the script in terminal. Remember, we don't have to pass any arguments in the command line because we have done so in our script. + +--- +Let's take a look at another example. In this example we'll get a greater understanding of variables and positional parameters: + +``` +function afunc { + echo in function: $0 $1 $2 + var1="in function" + echo var1: $var1 +} +var1="outside function" + +echo var1: $var1 +echo $0: $1 $2 +afunc funcarg1 funcarg2 +echo var1: $var1 +echo $0: $1 $2 +``` + +--- +Let's break it down again: + +1. In our first function called `afunc` , using `echo` we will print + `in function:` and pass 3 positional parameters. We will + then define the variable `var1` and call it `"in function"` + and print it using echo again. + +``` +function afunc { + echo in function: $@ $1 $2 + var1="in function" + echo var1: $var1 +} +``` + +--- +2. Outside of the function, we'll create another variable also named `var1` and give it the value of `"outside function"` + +``` +var1="outside function" +``` + +--- +3. We'll then add the program. + a) `echo`, we'll print `var1` + b) Print 3 positional parameters + c) Call the function with two arguments + d) Print `var1` again + e) Print 3 positional parameters again + +``` +echo var1: $var1 +echo $0: $1 $2 +afunc funcarg1 funcarg2 +echo var1: $var1 +echo $0: $1 $2 +``` + +--- +Let's run it in our terminal without any additional arguments and see what the output is. + +- Why did `echo $@: $1 $2` only output one argument? +- Why did `var1` change the third time to `inside function` rather than `outside function` ? + +--- +Now let's change and add a few things to see what happens: + +- In our terminal, what happens if we pass two arguments by entering `ascript.sh arg1 arg2` with `ascript.sh` being the name of our script and `arg1 arg2` being two random arguments? +- What happens if we add `local` to our function? + +--- +### Parameter Expansion + +Let's discuss the difference between `$a` and `${a}` + +`$a` on its own is fine, but when placed next to another string, it can confuse the shell. For example: + +- `$a_file` the shell will try to expand a variable named + `a_file` rather than `a` +- `${a}_file` the shell will now try to expand the variable `a` + +--- +This can help us be more flexible when navigating and manipulating files and directories. + +Let's look at the code below to see how this helps us: + +``` +$ filename="myfile" +$ touch $filename +$ mv $filename ${filename}1 +``` + +--- +This block of code creates a file based on our defined variable and then renames it with the same variable but with an additional component. + +Parameter expansion also help us if our variables are unset (i.e. do not exist) or are empty. Let's take a look at a couple examples in the next few slides. + +--- +1. `${parameter:=x}` If parameter is unset or empty, expansion results in the value of `x` and the value of `x` is assigned to the parameter. If it's not empty, it results in the value of the parameter + +``` +$ foo= +$ echo ${foo:="Something else"} +$ echo $foo +$ foo=bar +$ echo ${foo:="something else"} +$ echo $foo + +``` + +Through this sequence of commands we can see that when `$foo` is empty, `:-` fills the variable with `"something else"`. Once we define the variable, `:-` results in our defined variable. + +--- +2. `${parameter:=x}` If parameter is unset or empty, expansion results in the value of x and the value of x is assigned to the parameter. If it's not empty, it results in the value of the parameter + +``` +$ foo= +$ echo ${foo:="something else"} +$ echo $foo +$ foo=bar +$ echo ${foo:="Something else"} +$ echo $foo +``` + +We can see that when `$foo` is empty, `:=` assigns the variable with `"something else"`. If we define the variable again, `:-` results in our second defined variable. + +--- +3. `${parameter:?x}` If parameter is unset or empty, this expansion causes the script to exit with an error, and the contents of `x` are sent to standard error. If parameter is not empty, the expansion results in the value of parameter. + +``` +$ foo= +$ echo ${foo:?"something else"} +$ echo $? +$ foo=bar +$ echo ${foo:?"something else"} +$ echo $? +``` + +We can see that when `$foo` is empty, `:?` gives us an error which we can see as `echo $?` outputs `1`. If we define the variable again, `:?` results in the value of our variable. + +--- +4. `${parameter:+x}` If parameter is unset or empty, the expansion results in nothing. If parameter is not empty, the value of `x` is substituted for parameter; however, the value of parameter is not changed. + +``` +$ foo= +$ echo ${foo:+"something else"} +$ echo $foo +$ foo=bar +$ echo ${foo:+"something else"} +$ echo $foo +``` + +Here, `:+` resulted in an empty output and the value of `$foo` remains empty. If we define the variable, `:+` will still output what we defined, but it will not reassign the variable permanently. + +--- +### String Operators + +String operators are extremely valuable for operations on pathnames. They can help extract parts of pathnames, especially if they follow a pattern. Many pathnames typically follow patterns, such as all extensions are preceded with `.`. +Some character expansions are: + +1. `${#parameter}` +2. `${parameter:offset}` +3. `${parameter:offset: Length}` +4. `${#parameter}` expands into the length of the string +contained by the parameter. + +``` +$ foo="Toronto needs more trees" +$ echo "'$foo' is ${#foo} characters long." +``` + +--- +With the following expansions, we can extract a portion the string contained by the parameter. + +2. `${parameter:offset}` will extract characters from _offset_ characters to the end of the string. For example, counting from the beginning of the string, the _n_ of _needs_ is 8 characters from the beginning. Because did not specify an end, `echo` will print from _needs_ onwards. + +``` +$ foo="Toronto needs more trees" +¢ echo ${foo:8} +``` + +--- +3. `${parameter:offset: length}` will specify the length that we want to extract. This length is counted not from the beginning of the string, but from the offset of the string. + +``` +$ foo="Toronto needs more trees" +$ echo ${foo:8:5} +``` + +--- +We can see that from the beginning of the string, _n_ is 8 +characters in, and from _n_, s of _needs_ is the 5th character from +_n_. Therefore, our ouput will be _needs_. + +Let's now see how to use patterns in our parameter expansions. +There are several ways we can achieve this: + +1. `${parameter#pattern}` +2. `${parameter##pattern}` +3. `${parameterxpattern}` +4. `${parameterxspattern}` + +--- + +1. `${parameter#pattern}` removes the shortest leading portion of the string contained in _parameter_ defined by the _pattern_. + +``` +$ foo=/User/name/Desktop/file.txt.zip +$ echo ${foo#/x/} +``` + +In this example, we've defined foo as a file with an extension. +The expansion matches any (`*`) pattern of `/*/` and returns the shortest leading portion. + +--- + +2. `${parameter##pattern}` is very similar to the previous expansion except it removes the longest leading portion of the string. + +``` +$ foo=/User/name/Desktop/file.txt.zip +$ echo ${foo##/x/} +``` + +Very similar to the previous example, the expansion matches any (`*`) pattern of `/*/` and returns the longest leading portion. + +--- + +3. `${parameter%spattern}` removes the shortest ending portion of the string rather than the beginning. + +``` +$ foo=/User/name/Desktop/file.txt.zip +$ echo ${fo00%.x} +``` + +--- + +4. `${parameter%spattern}` removes the longest ending portion of the string. + +``` +$ foo=/User/name/Desktop/file.txt.zip +$ echo ${fo00%%.\*} +``` + +What happens if we change our pattern to `#*_` ? + +Let's pretend a file named "rachaels_file" and we want to know its extension. How would we do that? + +What if our file was name "rachaels file" + +We can also use expansions to replace the contents of the parameter with a string based on the pattern. + +1. `${parameter/pattern/string}` replaces only the first occurence of pattern. +2. `${parameter//pattern/string}` replaces all occurances. +3. `${parameter/#pattern/string}` requires the match to occur at the beginning of the string to replace it. +4. `${parameter/%pattern/string}` requires the match to occur at the end of the string to replace it. + +Let's see how this would work: + +``` +$ foo="MP3.MP3" +``` + +``` +$ echo ${foo/MP3/mp3} +``` + +--- +``` +$ echo ${foo//MP3/mp3} +``` + +``` +$ echo ${foo/#MP3/mp3} +``` + +``` +$ echo ${foo/%MP3/mp3} +``` + +Can you think of when this might be helpful? + +--- +Let's say I have a a named "rachaels cool file". I want to rename them because spaces cause problems in filenames. How would I do this? + +--- +### Arithmetic Assignment + +We have seen assignment before with examples such as `foo=5` . This is a simple assignment but we can also add complexity to this assignment with other operators. + +- `$((parameter += x))` assigns the parameter to itself `+` x +- `$((parameter -= x))` assigns the parameter to itself `-` x +- `$((parameter *= x))` assigns the parameter to itself `*` x +- `$((parameter /= x))` assigns the parameter to itself `/` x + +We can also increase or decrease our parameters by one. + +- `$((parameter++))` increases parameter by one after the parameter is retruned +- `$((parameter--))` decreases the parameter by one after the parameter is returned +- `$((++parameter))` increases parameter by one before the parameter is returned +- `$((--parameter))` decreases parameter by one before the parameter is returned. + +--- +These are very subtle changes so let's see what we mean after and before a parameter is returned: + +``` +$ foo=1 +$ echo $((foo++)) +$ echo $foo +``` + +``` +$ foo=1 +$ echo $((++foo)) +$ echo $foo +``` + +--- +### Command Substitution + +So far we've learned how to get values into variables by using assignment statements ( `x=5` ) and positional parameters ( `x=$1` ). Another way is command substitution which allows you to use the standard output of the command as if it were a variable. + +--- +Let's say we want to assign a variable to the output of a command so that we can apply another command to that output. In this particular case, we want to make a variable equalall files beginning with _t_. We then want to apply a sort command on that variable: + +``` +$ x=$(find tx) +$ echo $x | sort +``` + +Although this seems quite simple now, we'll see how this can be +extremely powerful when we move into flow control. + +--- +# Flow Control + +Flow control allows programs to "change directions" based on the results from a given input. +Bash supports several constructs: + +- `if/else` +- `while` / `until` +- `case` +- `for` + +--- +## if / else + +`if/else` is a conditional statement that chooses whether or +not to do something based on a true or false statement. + +--- +``` +if condition; then + commands +[elseif condition; then + commands... ] +[else + commands ] +fi +``` + +--- +Here, we've assigned `x` to the value `5`. We've then written an +`if/else` statement that asks if `x` is equal to `5` than tell us +that `x` equals `5`. Otherwise (`else`), tell us that `x` does not +equal `5`. + +--- +``` +x=5 +if [ $x = 5 ]; then + echo "x equals 5." +else + echo "x does not equal 5." +fi +``` + +--- +Let's take a look at a more practical example: we want to know if there are any files in our directory that contain spaces. + +``` +#!/bin/bash +cd ~/Desktop/dir1 +if [[ -n $(find . -type f | grep " ") ]]; then + echo "A file contains a space" +else + echo "No files contain a space" +fi +``` + +--- +First we've changed our working directory to _dir1_: + +``` +cd ~/Desktop/dir1 +``` + +We then utilized command substitutions that we've just learned +by storing the output of files that contain a space. The `-n` +option checks if the length of a string is _nonzero_: + +``` +-n $(find . -type f | grep " ") +``` + +--- +By wrapping our output in an if statement, we're stating: + +1. `if` the value of `$(find . -type f | grep " ")` is nonzero, then + print (`echo`) "A file contains a space" +2. Otherwise (`else`), print (`echo`) `"No files contain a space"` + +--- +### Control Operators + +Control operators ( `&&` and `||` ) allow you to test more than one thing at a time. Their syntax is: + +``` +if command1 && command2; then + ... +fi +``` + +``` +if command1 || command2; then + ... +fi +``` + +--- +With the `&&` operator, command1 is executed and command2 is executed only if command1 is **successful**. +With the `||` operator, command1 is executed and command2 is +executed only if command1 is _unsuccessful_. + +Example of `&&` + +``` +filename=$1 +word1=$2 +word2=$3 + +if grep $word1 $filename && grep $word2 $filename; then + echo "$word1 and $word2 are both in $filename." +fi +``` + +--- +Using positional parameters that we learned earlier, what do you think will happen if we run the previous code? + +- What happens if both words exist? +- What happens if only one word exists? +- What happens if no words exist? + +--- +Example of `||` + +``` +filename=$1 +word1=$2 +word2=$3 + +if grep $word1 $filename || grep $word2 $filename; then + echo "$word1 or $word2 is in $filename." +fi +``` + +--- +Similarly, what will happen if... + +- What happens if both words exist? +- What happens if only one word exists? +- What happens if no words exist? + +--- +## While + +Using the while command, let's discuss looping. Looping allows portions of a program to repeat as long as the condition is false. +This syntax is: + +``` +while condition; do + commands +done +``` + +--- +Let's make a basic while script that displays five numbers in sequential order from 1 to 5 and then tells us when it's finished. + +``` +#!/bin/bash + +# script called while-count.sh + +count=1 + +while [ $count -le 5 ]; do + echo $count + count=$((count +1)) +done +echo "Finished." +``` + +--- +Why does the loop end? + +While loops are extremely helpful to read lines of a file and then perform some command if a line meets a certain condition. Let's explore how to read lines first: + +``` +file=file1 +while read -r line; do + echo $line +done < "$file" +``` + +--- +In this script, we're creating a variable with our file. We're then reading the file until the last line is read. In this example, we're using an input redirection that we learned earlier ( `<` ), which passes the file into the read command. We've also used `-r` so that any backslashes are escaped. + +Because line is acting as a variable, we can also nest another loop if `$file` meets a condition. Let's say we have a file and we want to know every line that has `bananas` in it. +How would we combine the while loop with an if statement? + +``` +while read -r line; do + if [[ $line == *"bananas"* ]]; then + echo $line + fi +done < "$file" +``` + +--- +Here we're reading the file line by line using the `while` loop. +We're then saying `if` our variable, `$line` equals `"banana"` , +then print the `$line`. + +1. Why have we added the wildcard `*` ? +2. What would happen if we didn't include `*` ? + +--- +## Until + +Until loops are similar to while, except unlike while loops that run as long as the condition is true, the until loop will run as long as the condition is **false** + +``` +until condition; do + commands +done +``` + +--- +Let's create a script similar to the while statement: a basic while script that displays five numbers in sequential order from 1 to 5 and then tells us when it's finished. + +``` +count=1 + +until [ $count -gt 5 ]; do + echo $count + count=$((count +1) ) +done +echo "Finished." +``` + +How is this script different to the while loop? + +--- +How might this be useful? Let's say we want to create 3 +directories labeled _dir1_, _dir2_ and _dir3_: + +``` +x=1 +until [[ $x == 4 ]]; do + echo "Creating dir$x..." + mkdir dirs~x + ((x++) ) +done +``` + +--- +Here we've created a variable `x=1` because we want our first directory to be _dir1_. We're then saying up until `x=4`, make a directory `mkdir` called _dir_ plus our variable. We've then added 1 to `x` each iteration using an arithmetic assignment. The `echo` part is just to give us some feedback on what is happening behind the scenes. + +--- +## for + +For our final flow control, we're going to learn a powerful loop called `for`. The syntax is: + +``` +for variable [in words]; do + commands +done +``` + +What we might notice is that this flow uses variables that will increment during the execution of the loop. + +How would we use `for` if we wanted to list all files and directories in a folder? + +--- +``` +for i in $(find x«); do + echo $i +done +``` + +--- +The variable `i` becomes all instances of the variable `$(find *)` . For each instance of `i`, we are then printing it. Although this seems quite basic and there more simple ways to list all files and directories ( `ls` ), this enables us to do many things with the looped variable `i` by nesting other loops. + +What other ways can we use for loops? +What other ways can we use for loops within files? + +--- +### Questions? + +- Why do we use `i`? + +--- +### Next Week: Git and Github + +- Please make sure to come with a GitHub account + +--- +## Additional Material + +--- +### Exit Status + +Commands issue a value to the system when they terminate, which is an integer in the range of O and 255 indicating the success or failure of a command's execution. + +Conventionally, zero indicates success and any other value indicates failure. + +--- +Let's list a file that we know exists on our desktop: + +``` +$ ls -d /usr/bin +$ echo $? +``` + +`-d` is an option that returns the file if it exists and is a directory. + +`$?` returns the value of the last executed command. The value being either zero for success or any other number for failure. + +--- +If we then list a file that we know does not exist in our desktop and return the value of `$?`, what do we expect to happen? + +``` +$ ls -d /bin/usr +$ echo $? +``` + +--- +### Exit Command + +The `exit` command ina script replaces the return command and accepts a single, optional argument, which becomes the scripts exit status. + +When no argument is passed, it defaults to zero. This enables our scripts to indicate an error. + +--- +If the script is a function in a larger program, we can use `return` instead of `exit` with a single, optional argument, allowing our function to indicate an error. + +--- +``` +#!/bin/bash + +# test-file: Evaluate the status of a file + +FILE=~/.bashrc + +if [ -e "$FILE" ]; then + if [ -f "$FILE" ]; then + echo "$FILE 1sva regular file." + fi + if [ -d "$FILE" ]; then + echo "$FILE is a directory." + fi +else + echo "$FILE does not exist" + exit 1 +fi + +exit + +test_file () { + # test-file: Evaluate the status of a file + + FILE=~/.bashrc + + if [ -e "$FILE" ]; then + if [ -f "$FILE" ]; then + echo "$FILE is a regular file." + fi + if [ -d "$FILE" ]; then + echo "$FILE is a directory." + fi + else + echo "$FILE does not exist" + return 1 + fi +} + +``` + +--- +`if / else` statements are most frequently used with `test` + +`test` performs a variety of checks and comparisons + +Its syntax is: + + `test expression` + + or + + `[ expression ]` + +--- +There are many expressions that are used to evaluate the status +of files. Some important **File Expressions** include: + +| Expression | Is True If | +| ---------- | ----------------------------------------------------------------------------- | +| -e file | file exists | +| -d file | file exists and is a directory | +| -f file | file exists and is a regular file | +| -r file | file exists and is readable (has readable permissions for the effective user) | +| -s file | file exists and has a length greater than zero | + +--- +#### String Expressions + +| Expression | Is True If | +| ------------------ | ----------------------------------------- | +| string | string is not null | +| -n string | the length of string is greater than zero | +| -z string | the length of string is zero | +| string1 == string2 | string1 equals string2 | +| string1 != string2 | string1 and string2 are not equal | + +--- +### Breaking Out Of A Loop + +Bash has two build-in commands that can be used to control program flow inside loops. + +- `break` command immediately terminates a loop and resumes with the next statement following the loop + +- `continue` command skips the remainder the loop that is not needed (ie. a condition has been met) and resumes with the next iteration of the loop. `continue` allows for a more efficient execution + +--- +``` +if condition; then + if condition; then + commands + continue + fi + + if condition; then + commands + continue + fi +else condition; then + command +fi +``` + +--- +If the first `if` condition is met, then the second one will be skipped and resumed with the next iteration. + +``` +if condition; then + if condition; then + commands + continue + fi + if condition; then + commands + break + fi +else condition; then + command +fi + +If the second `if` condition is met, then the break immediately +terminates the loop and resumes with the next statement. +``` diff --git a/03_instructional_team/markdown_slides/unix_slides.md b/03_instructional_team/markdown_slides/unix_slides.md new file mode 100644 index 0000000..5a4f25d --- /dev/null +++ b/03_instructional_team/markdown_slides/unix_slides.md @@ -0,0 +1,559 @@ +--- +marp: true +theme: dsi_certificates_theme +style: | + section { + font-family: Inter, -apple-system, BlinkMacSystemFont, 'Helvetica Neue', sans-serif; + } +paginate: true +--- + +# Unix Shell + +``` +$ echo "Data Sciences Institute" +``` + +--- +### What is Unix? + +Unix was a text-based operating system created in 1970. Many of its derivatives are commonly used today, including Linux and MacOS. + +Linux powers +* 90% of global cloud infrastructure +* 100% of the world's top 500 high performance computers +* 97% of embedded and IoT devices + +--- +*Bash* and similar shells +* are the primary way of interacting with most production Linux systems +* empower you to quickly and easily navigate the system, manipulate files, and automate tasks + +--- +### So what is the shell? + +A *shell* is any user interface/program that takes an input from the user, translates it into instructions that the operating system can understand, and conveys the output back to the user. + +--- +There are various types of user interfaces: + +- graphical user interfaces (GUI) +- touch screen interfaces +- command line interfaces (CLI) + +--- +### And what is Bash? + +We'll be focusing on command line interfaces (CLI), more +specifically Bash, which stands for the Bourne Again SHell. + +On Windows: open the **Git Bash** or **Windows Terminal** app +On MacOS or Linux: open the **Terminal** app + +On newer macs, the default shell is zsh, which is almost identical to Bash. + +--- +### Clearing up an abundance of terminology + + +--- +### Clearing up an abundance of terminology + + +--- +### Clearing up an abundance of terminology + + +--- +### Clearing up an abundance of terminology + + +--- +### Clearing up an abundance of terminology + + +--- +### Let's get started! + +First, we'll open our terminal. As mentioned earlier, this is most +likely called terminal and can be found by searching our +computer, which on a Mac would be through cmd + space + +--- +Let's take a look at the terminal. What do we notice? + +- last login +- name +- location +- shell + +--- +### Looking at the Shell +Let's start with a few commands and see what happens in our +terminal. + +```bash +$ echo Rachael +$ date +$ cal +$ lksjfs +``` + +--- +- What happens when we type something that does not exist? +- What happens with errors? + +--- +# Getting help and accessing documentation +Bash includes built-in documentation for all commands + +```bash +$ man ls +``` + +Retrieve the **man**ual for each commands using `man`. +* Many manuals are also available online + + +--- +View a list of commonly used Bash commands + +```bash +$ help +``` + +--- +# Navigate Files / Directories + +--- +## Directories + +Let's try three commands that help us navigate our system: + +1. When using Bash, we always have a *working directory*: + +```bash +$ pwd +``` + +`pwd` prints our current working directory. If we ever need to know +where we are, we can execute this command. + +--- +2. To know what files and folders exist in our working directory, + we can use the code below: + +```bash +$ ls +``` + +--- +3. We can change the working directory using the following command: + +```bash +$ cd +``` + +By default, `cd` changes your *working directory* to your *home +directory*. You can also use `cd` to set your *working directory* by +including the desired *pathname* + +```bash +$ cd Desktop +``` + +--- + +```bash +$ cd Desktop +``` + +Note #1: In this example +* We can change the behaviour of the `cd` command by providing *arguments* +* By default, `cd` changes the *working directory* to our *home directory* +* By adding `Desktop` after the `cd` command, we change to the `Desktop` directory instead +* Most other bash commands have their own *arguments* that allow you to modify the behaviour and effect of the command + +--- + +Let's try using a *pathname argument* with the `ls` command too! + +```bash +$ cd +$ ls Desktop +``` + +--- + +```bash +$ cd Desktop +``` + +Note #2: +* we were able to refer to the `Desktop` directory using only its directory name because it existed within our *working directory* at the time the command was *called* +* in this context, `Desktop` is a *relative pathname* +* to refer to paths outside our working directory, use the full path instead + + +If we wanted to change the working directory to a directory +outside of our working directory, we would need to specify an *absolute pathname*: + +```bash +$ cd /Users/rachellam/Documents +$ cd /c/Users/simeo/Downloads +``` + +--- +## Paths + +As we've seen, directory names separated by slashes are paths. +There are two types of paths, _absolute_ and _relative_. + +- An *absolute pathname* begins at the *root directory* and + includes each directory, separated by slashes until the + desired directory or file is reached. +- A *relative pathname* starts from the current *working directory* and + uses symbols `.` or `..` to represent relative positions in the + file tree. + +--- +Using `cd` and `pwd` let's take a look at how we can use +*absolute* and *relative pathnames*. + +```bash +$ cd /usr/bin +$ pwd +``` + +```bash +$ cd /usr +$ pwd +``` + +```bash +$ cd .. +$ pwd +``` + +--- +## Working with Files / Directories + +We're going to learn some basic commands to begin some +preliminary coding. We'll also be using these throughout the +module, so it's important to understand how they work now: + +- create directory `mkdir` +- create an empty file `touch` +- copy `cp` +- move and rename `mv` +- remove `rm` + +--- +## Commands + +--- +### mkdir + +First let's make a directory. It's important to remember what +directory you're working in currently, because that's where the +new directory will be made. Let's assume for now, we're working +on our Desktop. + +```bash +$ mkdir my_directory +``` + +We can also create multiple directories at the same time: + +```bash +$ mkdir dir1 dir2 dir3 +``` + +--- +### touch + +We can also make new empty files from the command line. +Using `touch`, we can make a new file in our +*working directory*. + +```bash +$ touch file1 +``` + +We can also create a specific file type by adding the extension: + +```bash +$ touch file1.txt +``` + +--- +### cp + +Now we're going to copy a file that we have in our *working directory*. It +can be any file but remember to include the extension or if it +has multiple characters, special characters and spaces, to wrap +it in quotes. + +```bash +$ cp file1 file2 +``` + +--- +We can also copy files or directories into a directory. + +```bash +$ cp file1 dir1/ +``` + +And all files from one directory into another using *wildcards*: + +```bash +$ cp dirl/* dir2 +``` + +--- +#### Wildcards +*Wildcards* enable us to use commands on more than one file at a time. +The `*` is a placeholder that represents zero or more characters when used in a *pathname*. + +For example: +* `./*` matches all files and folders in the current directory +* `./*.txt` matches all files and folders that have a txt extension in the current directory +* `/home/simeo/project1_*` matches all files and folders that begin with `project1_` in the `/home/simeo` directory + +--- +*Wildcards* are one of many reasons why bash so powerful! + +Many of the features in bash empower you to work with a large number of files easily. + + +--- +There are some useful *options* that accompany `cp`: +| Option | Description | +| ------ | ------------------------------------------------------------- | +| -i | Before overwriting an existing file, prompt the user for confirmation. | +| -R | Recursively copy directories and their contents. | +| -v | Display informative messages as the copy is performed. | + +--- +### mv + +The mv command enables us to move and rename files and +directories, depending on how it's used. In th example below, `mv` renames `file1` to `file2`. + +Rename a file: +```bash +$ mv file1 file2 +``` + +Here, `mv` moves `file1` from the *working directory* into `./dir1`: +```bash +$ mv file1 dir1 +``` + +--- +We can also move directories into other directories: + +```bash +$ mv dir1 dir2 +``` + +--- + +```bash +$ mv dir1 dir2 +``` + +In this case, if `dir2` exists, `dir1` will be moved into `dir2` (eg. from `./dir1` to `./dir2/dir1`). + +If `dir2` does not exist, `dir1` will be renamed to `dir2`. + +In both cases the entire directory will be affected (moved into another directory, or renamed), rather than the contents. + +--- +Let's say we're in the directory `Desktop` and we just moved +`file1` into `dir1` but now we want to put it back in `Desktop`. How would we move a file out of a directory into another one? Unfortunatly we can't just say + +```bash +$ mv file1 Desktop +``` + +because `file1` does not exist in `Desktop` any more and the command will try and rename `file1` to `Desktop`. + +--- +The answer involves using *absolute pathnames* and the tilde `~` notation: + +``` +$ mv dir1/file1 ~/Desktop +``` + +The `~` is shortform for your *home directory*. + +--- +If we just wanted to move `file1` into `dir2` (if `dir2` is in our *working directory*), we could type: + +```bash +$ mv dir1/file1 dir2 +``` + +--- +What if we want to move just the contents of dir1 to another +directory rather than the whole folder? + +HINT: it is very similar to copying (`cp`). + +--- +Move just the contents of dir1 to another +directory rather than the whole folder: + +```bash +$ mv dir1/* dir2 +``` + +This is a combination of the directory `dir1`, pathnames `/` and wildcards `*`. Here, `dir1/*` takes the all the contents of `dir1` and puts it in `dir2`. + +--- +We could also use the same technique to specify certain files to move rather than all of them. + +How would we move all txt files from `dir1` into `dir2`? + +--- +#### Questions? + +We're starting to combine our knowledge of files, directories and pathnames with some basic commands. How do we feel up to this point? + +--- +### rm + +To remove files we use the command `rm`. Because we're now +deleting files, it's important that you're sure of what you're deleting because there is no way to undo. Fortunately!! there +are ways to do this. + +```bash +$ rm file1 +``` + +Without specifying any *options*, `file1` will be deleted +without any feedback. + +--- +To ensure we want to delete something, we can use the option `-i` (interactive) that we learned earlier. + +```bash +$ rm -i file1 +``` + +This will prompt a question asking us if we want to delete `file1`. We can respond with `y` (yes) or `n` (no). + +--- +If we want to delete a directory, we need to use the option `-r` (recursive) as we did when copying (`cp`). This will recursively delete everything inside of the directory and the directory itself. + +```bash +$ rm -r dir1 +``` + +--- +If we're specifying multiple deletions and a directory does not exist, the shell will tell us. If we don't want that message, we can add the `-option`, `-f` (force). Force will override `-i` if it is included. + +1. How do you delete multiple directories? +2. What happens if you delete multiple directories with `-i`? +3. What happens if you delete multiple directories with `-i` but one does not exist? + +--- +Remember, it's extremely important to remember that you cannot undo `rm`. This means, if you start using wildcards to specify filenames and don't include `-i`, you could delete things by accident. For example, let's say you want to delete all `.txt` files in a directory: + +```bash +$ rm *.txt +``` + +If you accidently add a space between `*` and `.txt`,the `rm` command will delete all the files in the directory and then try to find a `.txt` file which does not exist because it delete everything. + + +--- +## Working with text files +--- +Output the contents of any text file using `cat` + +```bash +$ cat file1.txt +``` + +* Quick way to preview file contents +* Note, this might flood your terminal if the file is too big. + +--- +Write the output of a command to a text file using `>` or `>>` + +```bash +$ ls >> dirlist.txt +``` + +* `>` replaces the destination file (deletes existing content) +* `>>` appends to the destination file + +--- + +We can also write custom content + +```bash +$ echo "Hello, world!" > myfile.txt +``` + +--- +We can also edit text files in the command line using `nano` + +```bash +$ nano file1.txt +``` + +To exit `nano`, press Control + X. The shortcuts are also shown at the bottom of the terminal while in the `nano` editor, where `^` denotes the Control key. + +--- +For more complex edits, we can launch Visual Studio Code directly from the command line. + +```bash +$ code file1.txt +``` + +Note: MacOS users may need to setup the `code` command first. + +--- +## Bash scripts +--- +We can write down a list of commands in a *script*, useful for +* A set of commands we need to frequently *execute* together +* Sharing commands with others + +A bash *script* is a text file with commands on different lines. +* bash will *execute* each command in sequence + +--- +Lines that begin with a `#` are *comments* that are ignored by bash. +* Use *comments* to annotate your thought process + +```bash +# copy text files from temp directory +cp tempdir/*.txt data/ + +# delete the rest of the temporary files +rm tempdir/* +``` + +--- +Bash scripts usually have a special *comment* as the very first line that indicates which shell should be used to run the script. +* This is also known as the sh-bang (ha**sh** symbol + exclamation mark/**bang**) + +Typically, the shell used is bash: +```bash +#!/bin/bash + +# do stuff +cp file1 dir2/ +``` + +--- + +Execute a script by running: +```bash +$ bash myscript.sh +``` diff --git a/README.md b/README.md index 7ee9a36..1e7f4e2 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,7 @@ -# Git and GitHub +# Unix shell + +## Content -## Contents * [Description](#description) * [Learning Outcomes](#learning-outcomes) * [Activities](#activities) @@ -9,37 +10,37 @@ * [Schedule](#schedule) * [Requirements](#requirements) * [Resources](#resources) - + [Cheat sheet](#cheatsheet) + + [Cheat sheet](#cheat-sheet) + [Videos](#videos) + [How to Get Help](#how-to-get-help) * [Folder Structure](#folder-structure) ## Description -This module explores version control with Git and GitHub, and how it connects to the ethical discussions of reproducibility. Participants will set up Git and create and use repositories, including recording, viewing and undoing changes. You will also learn how to create branches and collaborate with others with shared branches. This module also introduces more advanced commands such as de-bugging and history editing. +This module introduces the Unix shell language and covers file and directory navigation and manipulation. Participants gain proficiency in various commands, script creation, and writing basic functions using pipes, filters, and loops. -Throughout the entire module, participants will learn how to problem solve through live coding. You will also learn about reproducibility and how to centre it within your work. +Participants will acquire problem-solving skills through live coding sessions. Additionally, they will explore the concept of reproducibility and its integration into their work. ## Learning Outcomes - By the end of the module, participants will be able to: -* Use Git to collaboratively save, restore, and update work through version control -* Explain the difference between Git and GitHub +* Comfortably access and navigate the terminal +* Create, modify, and delete directories and files + ## Activities This module has two types of activities. 1. Assignments are mandatory, and form part of your evaluation. -1. Practice problems are not assessed, but are provided to you for extra practice. We encourage you to attempt the practice problems and discuss solutions together during the work sessions or on the Slack! +1. Practice problems are not assessed, but are provided to you for extra practice. We encourage you to work on them together and discuss solutions during Work Periods and on Slack. ### Assignments Participants should review the [Assignment Submission Guide](https://github.com/UofT-DSI/onboarding/blob/main/onboarding_documents/submissions.md) for instructions on how to complete assignments in this module. -1. [Combined Shell/Git Assignment](https://github.com/UofT-DSI/shell/blob/main/02_activities/assignments/assignment_instructions.md) +1. [Shell script assignment](./02_activities/assignments/assignment_instructions.md) -### Practice problems -1. [Git Basics](https://uoft-dsi.github.io/git/interactive_problems.html#git_basics) -1. [Git Moderate](https://uoft-dsi.github.io/git/interactive_problems.html#git_moderate) -1. [Git Advanced](./02_activities/practice/git_advanced.md) +### Practice Problems +1. [Shell Basics](https://uoft-dsi.github.io/shell/interactive_problems.html#shell_basics) +1. [Shell Moderate](https://uoft-dsi.github.io/shell/interactive_problems.html#shell_moderate) +1. [Shell Advanced](./02_activities/practice/shell_advanced/shell_advanced.md) ## Contacts @@ -55,16 +56,16 @@ Participants should review the [Assignment Submission Guide](https://github.com/ svkhomich1@gmail.com * **Kasra Vakiloroayaei** vakiloroayaei.kasra@gmail.com - * **Dmytro Bonislavskyi** + * **Dmytro Bonislavskyi** dmytro.bonislavskyi@gmail.com -## Delivery of the Learning Module - -This module will include live learning sessions and optional, asynchronous work periods. During live learning sessions, the Technical Facilitator will introduce and explain key concepts and demonstrate core skills. Learning is facilitated during this time. Before and after each live learning session, the instructional team will be available for questions related to the core concepts of the module. Optional work periods are to be used to seek help from peers, the Learning Support team, and to work through the practice problems and assignments in the learning module, with access to live help. Content is not facilitated, but rather this time should be driven by participants. We encourage participants to come to these work periods with questions and problems to work through. -Participants are encouraged to engage actively during the learning module. They key to developing the core skills in each learning module is through practice. The more participants engage in coding along with the instructional team, and applying the skills in each module, the more likely it is that these skills will solidify. +## Delivery of the Learning Module +This module will include live learning sessions and optional, asynchronous work periods. During live learning sessions, the Technical Facilitator will introduce and explain key concepts and demonstrate core skills. Learning is facilitated during this time. Before and after each live learning session, the instructional team will be available for questions related to the core concepts of the module. Optional work periods are to be used to seek help from peers, the Learning Support team, and to work through the practice problems and assignments in the learning module, with access to live help. Content is not facilitated, but rather this time should be driven by participants. We encourage participants to come to these work periods with questions and problems to work through. +  +Participants are encouraged to engage actively during the learning module. The key to developing the core skills in each learning module is through practice. The more participants engage in coding along with the instructional team, and applying the skills in each module, the more likely it is that these skills will solidify. ## Requirements @@ -72,7 +73,7 @@ Participants are encouraged to engage actively during the learning module. They * Participants are encouraged to ask questions, and collaborate with others to enhance their learning experience. * Participants must have a computer and an internet connection to participate in online activities. * Participants must not use generative AI such as ChatGPT to generate code in order to complete assignments. It should be used as a supportive tool to seek out answers to questions you may have. -* We expect participants have completed the [onboarding repo](https://github.com/UofT-DSI/onboarding/blob/main/environment_setup/README.md). +* We expect participants to have completed the steps in the [onboarding repo](https://github.com/UofT-DSI/onboarding/blob/main/environment_setup/README.md). * We encourage participants to default to having their camera on at all times, and turning the camera off only as needed. This will greatly enhance the learning experience for all participants and provides real-time feedback for the instructional team. ## Resources @@ -80,14 +81,13 @@ Participants are encouraged to engage actively during the learning module. They Feel free to use the following as resources: ### Cheat sheet - -- [Atlassian](https://www.atlassian.com/git/tutorials/atlassian-git-cheatsheet) -- [GitHub](https://education.github.com/git-cheat-sheet-education.pdf) +- [Devhints](https://devhints.io/bash) +- [Bash-Cheat-Sheet](https://github.com/RehanSaeed/Bash-Cheat-Sheet) ### Videos -- [Most common Git commands](https://www.youtube.com/watch?v=PSJ63LULKHA) -- [Git explained in 100 seconds](https://www.youtube.com/watch?v=hwP7WQkmECE) -- [Git vs GitHub: What's the difference?](https://www.youtube.com/watch?v=wpISo9TNjfU) +- [Change Directory](https://www.youtube.com/watch?v=6U4XV4w8qtE) +- [Deleting Files and Directories](https://www.youtube.com/watch?v=-L3XeZPwj_Y) +- [Bash in 100 seconds](https://www.youtube.com/watch?v=I4EWvMFj37g) ### How to Get Help #### 1. Gather information about your problem @@ -109,8 +109,8 @@ Feel free to use the following as resources: - Describe your overall goals, the context, and the specific details of what you were trying to accomplish - Make sure to **copy and paste** your code, your error message - Copying and pasting helps: - 1. your peers and teaching team quickly try out your code - 1. others to find your question in the future + 1. Your peers and teaching team quickly try out your code + 1. Others to find your question in the future #### Great resources on how to ask good technical questions that get useful answers - [Asking for Help - The Odin Project](https://www.theodinproject.com/lessons/foundations-asking-for-help) @@ -119,7 +119,7 @@ Feel free to use the following as resources: - [How to create a minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) ## Folder Structure - +Below is an outline of the folder structure for this module: ``` . ├── .github @@ -131,13 +131,13 @@ Feel free to use the following as resources: ├── LICENSE ├── SETUP.md └── README.md -``` +``` * **.github**: Contains issue templates and pull request templates for the repository. * **materials**: Module slides and interactive notebooks (.ipynb files) used during learning sessions. * **activities**: Contains graded assignments and ungraded practice problems to apply concepts covered in the learning module. * **instructional_team**: Resources for the instructional team. -* **this_cohort**: Additional materials and resources for this cohort. +* **this_cohort**: Additional materials for this cohort. * **.gitignore**: Files to exclude from this repository, specified by the Technical Facilitator. * **LICENSE**: The license for this repository. * **SETUP.md**: Contains the steps required to set up this repo for the module.