* First pass at porting to python3
* Fix import errors | Setup user argcomplete
* Update awstools CLI with user data file | Bump CI to use it
* Wait until launch is complete
* Add userdata as string | Use sudo for machine-launch-script
* Remove execute permissions on machine-launch-script
* Better match on machine-launch-script complete
* Revert python-devel removal
* Use python3 for pytests
* Update more python3 items
* Remove extra shebang
* Port docs to python3 and add to CI
* Add ISCA experiments to CI build check
* Use yum not apt-get
* Add make to doc section
* Bump multilate-loadgen for sysroot fix
* For BW test build don't use shebang
* Fix docs Makefile options
* Fix more doc warnings
* Add first set of regression tests
* Fix raw_input
* Regression bump | Run workload fix
* Add functools to topology
* Fix linux poweroff test (nic still has issues)
* Update regression scripts
* Ignore machine-launch-script.sh in regression area
* Fix map python3 issues
* Get rid of shebangs
* Fix more regressions
* Print machine-launch.log on fail | More clarification on user_data
* Transfer to CI some shorter regressions
* Add a manual approval to fpga based tests
* Fix indentation in config.yml
* Fix test symlink
* Use spot for CI manager instance | Try to use python3 for aws CI container | Version all pip packages
* Make run-ini-api-tests an executable
* Fix CI terminaterunfarm arg
* Add firesim.pem file to manager
* Bump python in CI instance
* Bump pip in CI container
* Remove pip sudo in CI container
* Fix launch script pip version equals
* Ini converted into strings
* Properly pass test_dir to opts in CI
* First pass at GH-A
* Round 2 CI fixes
* Try changes
* Remove CircleCI | Switch to fancy GH-A
* Rename self-host setup script
* Update chmod
* Use - instead of _ for env. vars
* Rename some defs | Remove extra imports
* Small comment updates
* Forgot to import in ini-api tests | Small comment on Fabric timeouts
* Add sys to linux poweroff
* Update linux timeout, fix small imports
* Update comment
* Fix-up workflow-monitor.py
* Avoid excessive logging in run-linux | Terminate spot instances after max-runtime
* Add more workflow-monitor states | Add pty=False to running workloads
* Update CI documentation | Add CI badge [ci skip]
* Don't use spot instances
* Update CI readme
* Determine runner version from remote repo and check for runner setup
* Address PR comments
* Update CI_README location of where to find IPs | Forgot ret_code
* Only run CI on prs/pushes to dev/main/master
* Fix terminate_workflow_instances in init-manager.py
* Cleanup FireSim repo cloning | Only run CI on PRs (since its runs on merge commit)
* Expose cmdline to launch/terminate instances
* Move instance liveness into the wait function
* More complete cmdline
* Have CI use API
* Forgot import
* Forgot another import
* Debug
* Use yaml parser instead of json
* Use safe yaml loading | Use default amt of instances
* Make sure to use aws-tools for cull-ci-insts
* Bump to AMI 1.11 / Vivado 2021.1
* pip2 removed on new AMIs; install it explicitly
* Knock BOOM AGFI frequency down
* Add a script to run linux boot on some default targets
* [docs] Tell users to use 1.11.0 AMI
Co-authored-by: Cloud User <centos@ip-192-168-3-37.ec2.internal>
Rather than exiting with a unhandled exception when SNS-related permissions failures happen at the end of buildafi, check whether the topic exists (or can be created) early in buildafi and warn the user that the script will be unable to send email (and log the details of the exception) but continue to finish buildafi because failure in sending the notification probably should not be a fatal error for the manager.
This PR also introduces pytest driven unit tests for the firesim manager. The tests make use of moto to mock the backend of boto and prevent tests from actually calling out to AWS API's. They also utilize unittest.mock and botocore.stub.Stubber to inject desired testing stimulus to the code under test.
Adding two tests for the new awstools.get_snsname_arn() function.
To run them:
* make sure you have the new deps in machine-launch-script.sh
* cd deploy
* pytest
Useful primers on pytest & testing AWS:
* https://tensoriot.medium.com/unit-testing-with-pytest-and-moto-e94fc2eefe7a
* https://github.com/boto/boto3/issues/2485
Useful primer on unittest.mock (fka py-mock, not to be confused with pymock):
* https://www.fugue.co/blog/2016-02-11-python-mocking-101
'mock' became part of the stdlib in python 3.3 and was backported to 2.7
as 'py-mock'. Of course, python being python, there is also a pymock
and that's totally different.
Detailed walkthrough of credential protection while using moto:
* https://blog.codecentric.de/en/2020/01/testing-aws-python-code-with-moto/
* [ci] Add manager pytests
* [ci] Provide aws credentials for pytest
* [ci] Remove AWS creds registration, andt push use conftest to provide a region
Co-authored-by: Tim Snyder <snyder.tim@gmail.com>
Co-authored-by: Tim Snyder <timothy.snyder@sifive.com>
Co-authored-by: David Biancolin <david.biancolin@gmail.com>
Enables one to run 'firesim tar2afi --launchtime <timestamp>' to run only
the tar->afi portion of buildafi for the configs listed in config_build.ini
Behavior of buildafi is unchanged
abejgonzalez