Refactor for Lambda Function URL and working examples

.devcontainer/devcontainer.json

@@ -19,6 +19,14 @@
 			]
 		}
 	},
-	"onCreateCommand": "sudo cp .devcontainer/welcome.txt /usr/local/etc/vscode-dev-containers/first-run-notice.txt && echo 'MONGODB_URI = \"<your_mongodb_uri_here>\"\nCOURSE_ID = \"<your_course_id_here>\"\nSSID = \"<your_wifi_ssid>\"\nPASSWORD = \"<your_wifi_password>\"' > my_secrets.py",
-	"postCreateCommand": "sudo -H pip3 install -r requirements.txt"
+	// Use 'forwardPorts' to make a list of ports inside the container available locally.
+	// "forwardPorts": [],
+	// Use 'postCreateCommand' to run commands after the container is created.
+	"onCreateCommand": "sudo cp .devcontainer/welcome.txt /usr/local/etc/vscode-dev-containers/first-run-notice.txt && echo 'SSID = \"<your_ssid_here>\"\nPASSWORD = \"<your_wifi_password_here>\"\nCOURSE_ID = \"<your_course_id_here>\"\nLAMBDA_FUNCTION_URL = \"<your_lambda_function_url_here>\"\nDATABASE_NAME = \"<your_database_name_here>\"\nCOLLECTION_NAME = \"<your_collection_name_here>\"' > my_secrets.py",
+	"postCreateCommand": "sudo -H pip3 install -r requirements.txt" // alternatively, `pip3 install --user -r requirements.txt`
+	// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
+	// "remoteUser": "vscode",
+	// "features": {
+	// 	"azure-cli": "latest"
+	// }
 }

.vscode/settings.json

@@ -7,7 +7,6 @@
     "githubPullRequests.ignoredPullRequestBranches": [
         "main"
     ],
-    "python.linting.enabled": true,
     "python.languageServer": "Pylance",
     "python.analysis.typeCheckingMode": "basic",
     "python.analysis.diagnosticSeverityOverrides": {

README.md

@@ -11,12 +11,100 @@ This assignment consists of two parts:
 For this assignment, you will need to add the [GitHub repository secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions) listed below both as GitHub Actions secrets and Codespaces secrets, so that both you and the autograding scripts can access them. Please also keep a backup copy of these secrets in a secure location.
 
 **The secrets to be added to GitHub Actions and Codespaces are:**
-| Variable Name | Description |
-|--------------|-------------|
-| `COURSE_ID`  | Your student identifier for the course |
-| `MONGODB_URI`| Your MongoDB connection string |
+| Variable Name         | Description                                                                                                                                            |
+| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `COURSE_ID`           | Your student identifier for the course                                                                                                                 |
+| `DATABASE_NAME`       | The name of your database within the cluster                                                                                                           |
+| `COLLECTION_NAME`     | The name of the collection from your database                                                                                                          |
+| `LAMBDA_FUNCTION_URL` | The Lambda Function URL that acts as an intermediate layer between the microcontroller and the orchestrator                                            |
+| `ATLAS_URI`           | The MongoDB Atlas universal resource identifier (URI) for the Python orchestrator to connect to MongoDB (sometimes referred to as a connection string) |
 
-**Note:** If you encounter issues adding secrets to Codespaces, please refer to the "Alternative Methods for Handling Secrets" section at the end of this document.
+While tedious, this approach is a one-time setup that simplifies the development and autograding process while adhering to security best practices. Your `COURSE_ID` can be accessed from the main course website by referencing the corresponding quiz response from the main course website. Instructions based on step 13a from [the Build Instructions manuscript](https://doi.org/10.1016/j.xpro.2023.102329) for each of the remaining secrets and instructions on how to add secrets are provided in the following sections.
+
+### Account and Cluster Setup
+
+First, [set up your MongoDB account](https://account.mongodb.com/account/register) and [create a free-tier M0 cluster](https://www.mongodb.com/docs/guides/atlas/cluster/). Choose a cluster name* such as `test-cluster` and take note of the name (`CLUSTER_NAME`).
+
+<sub>*`CLUSTER_NAME` should be 23 characters or fewer and should not end with a hyphen `-` (based on [Atlas docs](https://www.mongodb.com/docs/manual/reference/limits/#mongodb-atlas-label-limits))</sub>
+
+### Database and Collection
+
+Follow the `Creating a MongoDB Database with the Atlas UI` instructions within the [Create a Database guide](https://www.mongodb.com/basics/create-database). Take note of the database name (`DATABASE_NAME`) and collection name (`COLLECTION_NAME`) that you choose: for example, `test-db` and `test-collection`, respectively.
+
+### Database User
+
+Next, [create a database user](https://www.mongodb.com/docs/guides/atlas/db-user/). Assign a specific privilege with the `readWrite` for the database you created (e.g., `test-db`) and **Restrict Access to Specific Clusters** for your current cluster (e.g., `test-cluster`), as shown below. Store the username and password in a secure location. Remember to click the green **Add User** button at the bottom-right.
+
+![Privileges](./readme-images/privileges.png)
+
+You will do this in place of assigning a built-in role, which typically would allow you access to any database in any cluster. NOTE: There are many ways you can customize access including [built-in roles and privileges](https://www.mongodb.com/docs/atlas/mongodb-users-roles-and-privileges/) and [custom database roles](https://www.mongodb.com/docs/atlas/security-add-mongodb-roles/).
+
+### Network Configuration
+
+Add `0.0.0.0/0` to [the network configuration](https://www.mongodb.com/docs/guides/atlas/network-connections/) to allow access from anywhere.
+
+<!-- ![Creating the database and collection](create-db-collection.gif) -->
+
+### Atlas URI
+
+You will also need to copy the [MongoDB Atlas URI](https://www.mongodb.com/docs/guides/atlas/connection-string/) for your cluster. Refer to the video below. IMPORTANT: After copying the Atlas URI (connection string), you will need to manually populate `<db_username>` and `<db_password>` with your database user credentials before adding this as a GitHub secret. For example, if the copy-pasted Atlas URI is `mongodb+srv://<db_username>:<db_password>@test-cluster.ab123.mongodb.net/?retryWrites=true&w=majority` and your MongoDB username and password are `sgbaird` and `HGzZNsQ3vBLKrXXF`, respectively, then your `ATLAS_URI` would be `mongodb+srv://sgbaird:[email protected]/?retryWrites=true&w=majority`.
+
+![Finding MongoDB Atlas URI Cluster ID](./readme-images/mongodb-connect-uri.gif)
+
+### AWS Lambda Function
+
+NOTE: The instructions for this section are based on the [Lambda getting started guide](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html). There are more advanced ways to set up Lambda functions, but the following steps and level of complexity are tailored to our simple use-case.
+
+First, [create an AWS account](https://portal.aws.amazon.com/billing/signup) (you will need a credit card, but the free tier should be sufficient). Next, you'll create a new Lambda function:
+
+1. Open the [Functions page](https://console.aws.amazon.com/lambda/home#/functions) of the Lambda console.
+2. Choose **Create function**.
+3. Select **Author from scratch**.
+4. In the **Basic information** pane, for **Function name**, enter a name (e.g., `dataLogging`).
+5. For **Runtime**, choose **Python 3.13**.
+6. Click the dropdown for **Additional Configurations**
+7. Check the **Enable function URL** box
+8. Under **Auth type**, select **None**
+9. Check the **Configure cross-origin resource sharing (CORS) box
+
+Your settings should look like the following, with everything else left as default:
+
+![AWS Lambda Function Configuration](./readme-images/aws-lambda-configuration.png)
+
+10. Finally, click Create function.
+
+Open and run the [Preparing the Python code and dependencies for AWS Lambda](https://colab.research.google.com/github/AccelerationConsortium/ac-microcourses/blob/main/docs/courses/hello-world/1.5.2-mongodb-aws-lambda-prep.ipynb) Colab notebook, which will allow you to save a `deployment.zip` file to your local machine. If you have trouble accessing or running the notebook, there is [a backup copy](https://github.com/AccelerationConsortium/ac-microcourses/blob/main/docs/courses/hello-world/deployment.zip) [[permalink](https://github.com/AccelerationConsortium/ac-microcourses/blob/7a648c7d149b955d1312013ad5c33090ea17c5a1/docs/courses/hello-world/deployment.zip)]. However, the notebook has the latest version and is instructive for how to use [Lambda functions with Python dependencies](https://docs.aws.amazon.com/lambda/latest/dg/python-package.html).
+
+Back on the Lambda function page, upload the `deployment.zip` file by clicking on the **Code** tab and selecting **Upload from** and **.zip** file, then choosing the `deployment.zip` file from your machine. Click **Save**.
+
+In the code editor within `lambda_function.py`, you will add your `ATLAS_URI` as a [Lambda environment variable](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html) (for a similar reason to why you added it as a secret to the GitHub repository). Navigate to the **Configuration** tab. Along the left sidebar, click **Environment variables**, then **Edit**.
+
+![Edit env vars](./readme-images/edit-env-vars.png)
+
+Add your Atlas URI as follows, ensuring that your database username and password are embedded correctly. Click **Save**.
+
+![AWS Lambda Function Environment Variables](./readme-images/atlas-uri-env-var.png)
+
+Note the **Function URL** from the top right of the page. This is your `LAMBDA_FUNCTION_URL`.
+
+### Adding Secrets
+
+The interfaces for adding secrets can be found via your GitHub repository's `Settings` tab under the `Security` section on the left sidebar. Click the corresponding dropdown item in the `Secrets and variables` dropdown menu. Alternatively, use the direct links to navigate to these pages, which will be of the form:
+```html
+https://github.com/ACC-HelloWorld/5-data-logging-<username>/settings/secrets/actions
+```
+and
+```html
+https://github.com/ACC-HelloWorld/5-data-logging-<username>/settings/secrets/codespaces
+```
+where `<username>` is replaced with your GitHub username. For example:
+```html
+https://github.com/ACC-HelloWorld/5-data-logging-sgbaird/settings/secrets/actions
+```
+and
+```html
+https://github.com/ACC-HelloWorld/5-data-logging-sgbaird/settings/secrets/codespaces
+```
 
 ## Insert
 
@@ -62,14 +150,33 @@ df = pd.json_normalize(data).set_index("id")
 print(list(df.columns))
 ['category1.item1', 'category1.item2', 'category1.item3', 'category2.itemA', 'category2.itemB', 'category2.itemC']
 print(df)
+#      category1.item1  category1.item2  category1.item3  category2.itemA  category2.itemB  category2.itemC
+# _id                                                                                                      
+# id1                1                2                3               10               20               30
+# id2                4                5                6               40               50               60
 ```
 
-## Testing
+NOTE: Committing and pushing `results.csv` is optional and will
+not affect the autograding results.
+
+## Setup command
+
+See `postCreateCommand` from [`devcontainer.json`](.devcontainer/devcontainer.json).
+
+## Run command
+`pytest`
+
+You can also use the "Testing" sidebar extension to easily run individual tests.
+
+## Additional Resources
+- [IP Access List docs](https://www.mongodb.com/docs/atlas/security/ip-access-list/#add-ip-access-list-entries) ("Atlas UI" instructions)
+- [Create a Database guide](https://www.mongodb.com/basics/create-database).
+
+
+Before, we mentioned that we strategically selected a minimal implementation of AWS Lambda functions. To elaborate, we chose to use a `.zip` file for the Python dependencies rather than an [AWS Lambda Layer](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html). The latter is a more advanced feature that allows you to share code across multiple functions. We chose to use the Lambda Function URL directly rather than an [API Gateway](https://docs.aws.amazon.com/lambda/latest/dg/services-apigateway.html) because it is simpler and more direct for our use-case. We also chose to allow the Lambda Function URL to be accessible by any machine ([CORS](https://docs.aws.amazon.com/lambda/latest/api/API_Cors.html)) without [authentication](https://docs.aws.amazon.com/lambda/latest/dg/urls-auth.html) (other than knowing the URL itself). Finally, we chose to add our `ATLAS_URI` as an environment variable within the Lambda function rather than using [AWS Secrets Manager secrets in AWS Lambda functions](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_lambda.html). The goal of this assignment is not to make you an expert at AWS Lambda, but rather to enable you to send data from your microcontroller directly to a database without overcomplicating things. If your usage of AWS Lambda grows, you may want to consider these more advanced features.
 
-The autograding workflow will run several tests to verify your implementation:
-1. MongoDB credentials test
-2. MongoDB connection test
-3. Data insertion test
-4. Data retrieval test
+For some additional resources related to AWS Lambda:
 
-Make sure all tests pass before submitting your assignment.
+- [AWS Lambda guide](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html)
+- [Best practices for working with AWS Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html)
+- [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html)

find.py

@@ -8,24 +8,20 @@
 
 course_id = os.environ["COURSE_ID"]
 
-# TODO: Create MongoDB client and connect to database
-# Hint: Use MongoClient(MONGODB_URI) and verify connection with ping
-...
+database_name = os.environ["DATABASE_NAME"]
+collection_name = os.environ["COLLECTION_NAME"]
 
-# TODO: Get database and collection
-# Hint: Use client['database_name'] and db['collection_name']
-...
+atlas_uri = os.environ["ATLAS_URI"]
 
-# TODO: Query all documents with your course_id
-# Hint: Use collection.find({"course_id": course_id})
+# TODO: Extract all entries from the database
 ...
 
-# TODO: Create pandas DataFrame from results
-# Hint: Use pd.DataFrame(results)
+# TODO: Create a pandas DataFrame per instructions in README
 ...
 
 # TODO: Set '_id' as index and save to CSV
 # Hint: Use df.set_index("_id") and df.to_csv()
 ...
 
-# Don't forget to close the MongoDB connection!
+# TODO: Close the client
+...