Cypress integration tests for https://github.com/ImagineLearning/ai-toolkit/

[!NOTE] These instructions are based on https://github.com/edgenuity/mpng-automation-ui/blob/main/README.MD Please direct questions on Slack to the #qa_cypress channel on the Imagine Learning Enterprise workspace.

SETUP

Install Node 20+ (current: v20.11.1) Install npm 10+ (current: 10.2.4) Clone this repo and run npm i

CREATE .env FILE

Create a local .env file in the root of the repo. The file MUST contain the following:

CYPRESS_TEAM=
CYPRESS_ENV=
CYPRESS_API_URL="https://cypress-director.imaginelearning.engineering/"

CYPRESS_ENV specifies the environment where tests will run from your local machine:

• "local"
• "qa"
• "prod"

CYPRESS_TEAM specifies which team's data to use for tests:

• "lmsadmin" uses env-mps-data (this is the default test data if CYPRESS_TEAM is not provided)
• "edgeex" uses ...
• "mpng" uses ``

PLEASE DO NOT COMMIT YOUR LOCAL .env FILE!

RUNNING TESTS

Cypress E2E UI (headed : local) Tests can be run using the Cypress E2E UI on your local machine. You can choose which test(s) to run and view it in realtime.

npm run cy-open

Sorry Cypress (headless : local) Tests can be run on Sorry Cypress from your local machine. Tests will run in the environment set in your .env file. Test tags are implemented with Gleb Bahmutov's cy-grep and can be used to run a subset of tests: @smoke

#to run tests with a tag, surround the tag in quotes
#run ALL tests with @smoke tag
npm run local '@smoke'

#to run tests with a tag OR another tag, surround each tag in quotes
#run ALL tests with @alladmin OR @teacher tags (||)
npm run local '@alladmin' '@teacher'

#to run tests with a tag AND another tag, concatenate tags with + and surround in quotes
#run ALL tests with @schooladmin AND @smoke tags (&&)
npm run local '@schooladmin+@smoke'

#to run tests with a tag AND another tag OR tests with a tag, combine the tagging methods
#run ALL tests with @teacher AND @smoke tags OR ALL tests with @student tag (&& and ||)
npm run local '@teacher+smoke' '@student'

Scripts are also included for local test runs:

__RUN ALL TESTS__
#run every test in the repo
npm run local

VIEWING TEST RESULTS

Cypress E2E UI (headed : local) Test results from Cypress E2E UI runs are stored in the repo directory:

cypress/reports/downloads
cypress/reports/screenshots
cypress/reports/videos

Sorry Cypress (headless : local or remote) Test results from local and CI/CD pipeline runs are stored online: Sorry Cypress Dashboard

TROUBLESHOOTING

CYPRESS E2E UI ISSUES

• Cypress E2E UI and Chrome can eat up a lot of memory. If you get the "Aw, Snap!" Chrome error message, you have to shut down Cypress entirely from the CLI and restart again. Restarting the Cypress E2E UI is not enough; the error will persist.
• In cypress.config.js, the value numTestsKeptInMemory is set to 0 to save memory. This can be increased to view results from local test runs if desired but please remember to set to 0 on pushes.

PACT PROVIDER TEST

• For provider verification of FeedbackEvent (pact\provider\shortwritinggradingassistant\feedbackevent-provider.spec.js), FeedbackEvent-Valid.json is used as source to verify the contract. It should always have the latest FeedbackEvent generated by the AITK QA Service.
• Whenever there are any changes in response of the AITK QA Service, this file should be updated with the latest FeedbackEvent response generated by the AITK QA Service and then the contract verification should be run.
• File path - C:\Users\aakanksha.khaire\Imagine_Learning_Workspace\ai-toolkit-qe\pact\test-data\shortwritinggradingassistant\FeedbackEvent-Valid.json C:\Users\aakanksha.khaire\Imagine_Learning_Workspace\ai-toolkit-qe\pact\test-data\shortwritinggradingassistant\FeedbackEvent-Harmful.json
• This provider verification will be triggered by EdgeEx through the github workflow - github\workflows\contract_requiring_verification_published.yaml
• Whenever consumer contract changes it will trigger this workflow to execute provider verification and check the provider is able to meet the contract
• The worklow execution in the Git repository can be monitored periodically or if any communication on consumer contract change are done to see for any failures https://github.com/ImagineLearning/ai-toolkit-qe/actions/workflows/contract_requiring_verification_published.yaml

ON DEMAND INTEGRATION TEST

Steps for executing the E2E On-Demand Integration Test -

1. Place the input test data file in S3 bucket in expected JSON format DEV - S3 bucket: aitk-dev-integration-test Folder to place input file: aitk-dev-integration-test/short-writing-grading-assistant/input/ Url: https://us-east-2.console.aws.amazon.com/s3/buckets/aitk-dev-integration-test?region=us-east-2&bucketType=general&prefix=short-writing-grading-assistant/input/&showversions=false

   QA - S3 bucket: aitk-qa-integration-test Folder to place input file: aitk-qa-integration-test/short-writing-grading-assistant/input/ Url: https://us-east-2.console.aws.amazon.com/s3/buckets/aitk-qa-integration-test?region=us-east-2&bucketType=general&prefix=short-writing-grading-assistant/input/&showversions=false

2. Submit the test-workflow from below url and mentioned parameters Workflows Url: https://workflows.data-nonprod.imaginelearning.tech/ Login above url and click on SUBMIT NEW WORKFLOW and select test-workflow Parameters: githubRepo - ImagineLearning/ai-toolkit-qe pathToDeployableFile - deploy.yaml version - latest entrypoint - /home/node/ai-toolkit-e2e/integration/shortwritinggradingassistant/entrypoint.sh environment - dev or qa (depending on which environment the test needs to be executed)

3. Once the test-workflow execution is completed successfully, output validation file will be generated at S3 location DEV - S3 bucket: aitk-dev-integration-test Folder to place input file: aitk-dev-integration-test/short-writing-grading-assistant/output/ Url: https://us-east-2.console.aws.amazon.com/s3/buckets/aitk-dev-integration-test?region=us-east-2&bucketType=general&prefix=short-writing-grading-assistant/output/&showversions=false

   QA - S3 bucket: aitk-qa-integration-test Folder to place input file: aitk-qa-integration-test/short-writing-grading-assistant/output/ Url: https://us-east-2.console.aws.amazon.com/s3/buckets/aitk-qa-integration-test?region=us-east-2&bucketType=general&prefix=short-writing-grading-assistant/output/&showversions=false