-
Notifications
You must be signed in to change notification settings - Fork 57
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'develop' into competitions-pin_dev_merge
# Conflicts: # src/main/java/beam/analysis/summary/VehicleMilesTraveledAnalysis.java # src/main/scala/beam/agentsim/agents/choice/mode/ModeChoiceMultinomialLogit.scala # src/main/scala/beam/agentsim/infrastructure/ZonalParkingManager.scala # src/main/scala/beam/router/r5/R5RoutingWorker.scala # src/main/scala/beam/sim/BeamHelper.scala # src/main/scala/beam/sim/BeamPrepareForSim.scala # src/main/scala/beam/sim/population/PopulationAdjustment.scala # src/main/scala/beam/utils/plan/sampling/AvailableModeUtils.scala # src/main/scala/beam/utils/plan/sampling/PlansSamplerApp.scala
- Loading branch information
Showing
408 changed files
with
22,500 additions
and
12,242 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,9 +16,15 @@ | |
| - sleep 10s | ||
| - opth="output" | ||
| - echo $opth | ||
| - for file in glob.iglob($opth/*); do sudo cp /var/log/cloud-init-output.log "$file" && sudo zip -r "${file%.*}_$UID.zip" "$file"; done; | ||
| - for file in glob.iglob($opth/*.zip); do s3p="$s3p, https://s3.us-east-2.amazonaws.com/beam-outputs/$(os.path.basename($file))"; done; | ||
| - sudo aws --region "$S3_REGION" s3 cp $opth/*.zip s3://beam-outputs/''' | ||
| - finalPath="" | ||
| - for file in $opth/*; do | ||
| - for path2 in $file/*; do | ||
| - finalPath="$path2"; | ||
| - done; | ||
| - done; | ||
| - sudo cp /var/log/cloud-init-output.log "$finalPath" | ||
| - sudo aws --region "$S3_REGION" s3 cp "$finalPath" s3://beam-outputs/"$finalPath" --recursive; | ||
| - s3p="$s3p, https://s3.us-east-2.amazonaws.com/beam-outputs/index.html#$finalPath"''' | ||
|
|
||
| END_SCRIPT_DEFAULT = '''echo "End script not provided."''' | ||
|
|
||
|
|
@@ -40,36 +46,72 @@ | |
|
|
||
| CONFIG_DEFAULT = 'production/application-sfbay/base.conf' | ||
|
|
||
| initscript = (('''#cloud-config | ||
| initscript = ((''' | ||
| #cloud-config | ||
| write_files: | ||
| - content: | | ||
| #!/bin/bash | ||
| data="{\\"text\\":\\"" | ||
| data+=$1 | ||
| data+="\\"}" | ||
| printf "%s" "$data" > /tmp/slack.json | ||
| curl -X POST -H 'Content-type: application/json' --data-binary @/tmp/slack.json $SLACK_HOOK_WITH_TOKEN | ||
| path: /tmp/slack.sh | ||
| - content: | | ||
| 0 * * * * curl -X POST -H "Content-type: application/json" --data '"'"'{"$(ec2metadata --instance-type) instance $(ec2metadata --instance-id) running... \\n Batch [$UID] completed and instance of type $(ec2metadata --instance-type) is still running in $REGION since last $(($(($(date +%s) - $(cat /tmp/.starttime))) / 3600)) Hour $(($(($(date +%s) - $(cat /tmp/.starttime))) / 60)) Minute."}'"'" | ||
| path: /tmp/slack_notification | ||
| runcmd: | ||
| - echo "-------------------Starting Beam Sim----------------------" | ||
| - echo $(date +%s) > /tmp/.starttime | ||
| - cd /home/ubuntu/git/beam | ||
| - rm -rf /home/ubuntu/git/beam/test/input/sf-light/r5/network.dat | ||
| - ln -sf /var/log/cloud-init-output.log ./cloud-init-output.log | ||
| - /home/ubuntu/git/glip.sh -i "http://icons.iconarchive.com/icons/uiconstock/socialmedia/32/AWS-icon.png" -a "Run Started" -b "Run Name** $TITLED** \\n Instance ID $(ec2metadata --instance-id) \\n Instance type **$(ec2metadata --instance-type)** \\n Host name **$(ec2metadata --public-hostname)** \\n Web browser **http://$(ec2metadata --public-hostname):8000** \\n Region $REGION \\n Batch $UID \\n Branch **$BRANCH** \\n Commit $COMMIT" | ||
| - curl -X POST -H 'Content-type: application/json' --data '{"text":"Run Started \\n Run Name** $TITLED** \\n Instance ID $(ec2metadata --instance-id) \\n Instance type **$(ec2metadata --instance-type)** \\n Host name **$(ec2metadata --public-hostname)** \\n Web browser **http://$(ec2metadata --public-hostname):8000** \\n Region $REGION \\n Batch $UID \\n Branch **$BRANCH** \\n Commit $COMMIT"}' https://hooks.slack.com/services/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 | ||
| - hello_msg=$(printf "Run Started \\n Run Name** $TITLED** \\n Instance ID %s \\n Instance type **%s** \\n Host name **%s** \\n Web browser ** http://%s:8000 ** \\n Region $REGION \\n Batch $UID \\n Branch **$BRANCH** \\n Commit $COMMIT" $(ec2metadata --instance-id) $(ec2metadata --instance-type) $(ec2metadata --public-hostname) $(ec2metadata --public-hostname)) | ||
| - chmod +x /tmp/slack.sh | ||
| - echo "notification sent..." | ||
| - echo '0 * * * * /home/ubuntu/git/glip.sh -i "http://icons.iconarchive.com/icons/uiconstock/socialmedia/32/AWS-icon.png" -a "$(ec2metadata --instance-type) instance $(ec2metadata --instance-id) running..." -b "Batch [$UID] completed and instance of type $(ec2metadata --instance-type) is still running in $REGION since last $(($(($(date +%s) - $(cat /tmp/.starttime))) / 3600)) Hour $(($(($(date +%s) - $(cat /tmp/.starttime))) / 60)) Minute."' > /tmp/glip_notification | ||
| - echo '0 * * * * curl -X POST -H "Content-type: application/json" --data '"'"'{"$(ec2metadata --instance-type) instance $(ec2metadata --instance-id) running... \\n Batch [$UID] completed and instance of type $(ec2metadata --instance-type) is still running in $REGION since last $(($(($(date +%s) - $(cat /tmp/.starttime))) / 3600)) Hour $(($(($(date +%s) - $(cat /tmp/.starttime))) / 60)) Minute."}'"'"'' > /tmp/slack_notification | ||
| - echo "notification saved..." | ||
| - crontab /tmp/glip_notification | ||
| - crontab /tmp/slack_notification | ||
| - crontab -l | ||
| - echo "notification scheduled..." | ||
| - git fetch | ||
| - echo "git checkout ..." | ||
| - GIT_LFS_SKIP_SMUDGE=1 sudo git checkout $BRANCH | ||
| - echo "git checkout -qf ..." | ||
| - GIT_LFS_SKIP_SMUDGE=1 sudo git checkout -qf $COMMIT | ||
| - sudo git pull | ||
| - sudo git lfs pull | ||
| - echo "git checkout -qf ..." | ||
| - GIT_LFS_SKIP_SMUDGE=1 sudo git checkout -qf $COMMIT | ||
| - echo "preparing for python analysis" | ||
| - sudo dpkg --configure -a | ||
| - sudo dpkg --remove --force-remove-reinstreq unattended-upgrades | ||
| - sudo apt-get install unattended-upgrades | ||
| - sudo dpkg --configure -a | ||
| - sudo apt update | ||
| - sudo apt install npm -y | ||
| - sudo apt install nodejs-legacy -y | ||
| - sudo apt install python-pip -y | ||
| - pip install --upgrade pip | ||
| - sudo pip install pandas | ||
| - sudo pip install plotly | ||
| - sudo pip install psutil requests | ||
| - sudo npm cache clean -f | ||
| - sudo npm install -g n | ||
| - sudo n stable | ||
| - sudo npm install -g npm | ||
| - sudo apt-get install curl | ||
| - curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash - | ||
| - sudo apt-get install nodejs -y | ||
| - sudo apt-get install libgtkextra-dev libgconf2-dev libnss3 libasound2 libxtst-dev -y | ||
| - sudo npm install -g [email protected] orca --unsafe-perm=true --alow-root -y | ||
| - sudo apt-get install xvfb -y | ||
| - echo "gradlew assemble ..." | ||
| - ./gradlew assemble | ||
| - echo "looping config ..." | ||
| - export MAXRAM=$MAX_RAM | ||
| - export SIGOPT_CLIENT_ID="$SIGOPT_CLIENT_ID" | ||
| - export SIGOPT_DEV_ID="$SIGOPT_DEV_ID" | ||
| - echo $MAXRAM | ||
| - /tmp/slack.sh "$hello_msg" | ||
| - /home/ubuntu/git/glip.sh -i "http://icons.iconarchive.com/icons/uiconstock/socialmedia/32/AWS-icon.png" -a "Run Started" -b "Run Name** $TITLED** \\n Instance ID $(ec2metadata --instance-id) \\n Instance type **$(ec2metadata --instance-type)** \\n Host name **$(ec2metadata --public-hostname)** \\n Web browser **http://$(ec2metadata --public-hostname):8000** \\n Region $REGION \\n Batch $UID \\n Branch **$BRANCH** \\n Commit $COMMIT" | ||
| - s3p="" | ||
| - for cf in $CONFIG | ||
| - do | ||
|
|
@@ -81,8 +123,10 @@ | |
| - then | ||
| - s3glip="\\n S3 output url ${s3p#","}" | ||
| - fi | ||
| - bye_msg=$(printf "Run Completed \\n Run Name** $TITLED** \\n Instance ID %s \\n Instance type **%s** \\n Host name **%s** \\n Web browser ** http://%s:8000 ** \\n Region $REGION \\n Batch $UID \\n Branch **$BRANCH** \\n Commit $COMMIT %s \\n Shutdown in $SHUTDOWN_WAIT minutes" $(ec2metadata --instance-id) $(ec2metadata --instance-type) $(ec2metadata --public-hostname) $(ec2metadata --public-hostname) "$s3glip") | ||
| - echo "$bye_msg" | ||
| - /tmp/slack.sh "$bye_msg" | ||
| - /home/ubuntu/git/glip.sh -i "http://icons.iconarchive.com/icons/uiconstock/socialmedia/32/AWS-icon.png" -a "Run Completed" -b "Run Name** $TITLED** \\n Instance ID $(ec2metadata --instance-id) \\n Instance type **$(ec2metadata --instance-type)** \\n Host name **$(ec2metadata --public-hostname)** \\n Web browser **http://$(ec2metadata --public-hostname):8000** \\n Region $REGION \\n Batch $UID \\n Branch **$BRANCH** \\n Commit $COMMIT $s3glip \\n Shutdown in $SHUTDOWN_WAIT minutes" | ||
| - curl -X POST -H 'Content-type: application/json' --data '{"text":"Run Completed \\n Run Name** $TITLED** \\n Instance ID $(ec2metadata --instance-id) \\n Instance type **$(ec2metadata --instance-type)** \\n Host name **$(ec2metadata --public-hostname)** \\n Web browser **http://$(ec2metadata --public-hostname):8000** \\n Region $REGION \\n Batch $UID \\n Branch **$BRANCH** \\n Commit $COMMIT $s3glip \\n Shutdown in $SHUTDOWN_WAIT minutes"}' https://hooks.slack.com/services/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 | ||
| - $END_SCRIPT | ||
| - sudo shutdown -h +$SHUTDOWN_WAIT | ||
| ''')) | ||
|
|
@@ -109,7 +153,6 @@ | |
| 'r5.large', 'r5.xlarge', 'r5.2xlarge', 'r5.4xlarge', 'r5.12xlarge', 'r5.24xlarge', | ||
| 'r5d.large', 'r5d.xlarge', 'r5d.2xlarge', 'r5d.4xlarge', 'r5d.12xlarge', 'r5d.24xlarge', | ||
| 'm5d.large', 'm5d.xlarge', 'm5d.2xlarge', 'm5d.4xlarge', 'm5d.12xlarge', 'm5d.24xlarge', | ||
| 'm5a.24xlarge', | ||
| 'z1d.large', 'z1d.xlarge', 'z1d.2xlarge', 'z1d.3xlarge', 'z1d.6xlarge', 'z1d.12xlarge'] | ||
|
|
||
| regions = ['us-east-1', 'us-east-2', 'us-west-2'] | ||
|
|
@@ -234,7 +277,7 @@ def deploy_handler(event): | |
| if shutdown_behaviour not in shutdown_behaviours: | ||
| shutdown_behaviour = os.environ['SHUTDOWN_BEHAVIOUR'] | ||
|
|
||
| if volume_size < 64 or volume_size > 512: | ||
| if volume_size < 64 or volume_size > 256: | ||
| volume_size = 64 | ||
|
|
||
| selected_script = CONFIG_SCRIPT | ||
|
|
@@ -272,7 +315,12 @@ def deploy_handler(event): | |
| runName = titled | ||
| if len(params) > 1: | ||
| runName += "-" + `runNum` | ||
| script = initscript.replace('$RUN_SCRIPT',selected_script).replace('$REGION',region).replace('$S3_REGION',os.environ['REGION']).replace('$BRANCH',branch).replace('$COMMIT', commit_id).replace('$CONFIG', arg).replace('$MAIN_CLASS', execute_class).replace('$UID', uid).replace('$SHUTDOWN_WAIT', shutdown_wait).replace('$TITLED', runName).replace('$MAX_RAM', max_ram).replace('$S3_PUBLISH', s3_publish).replace('$SIGOPT_CLIENT_ID', sigopt_client_id).replace('$SIGOPT_DEV_ID', sigopt_dev_id).replace('$END_SCRIPT', end_script) | ||
| script = initscript.replace('$RUN_SCRIPT',selected_script).replace('$REGION',region).replace('$S3_REGION',os.environ['REGION']) \ | ||
| .replace('$BRANCH',branch).replace('$COMMIT', commit_id).replace('$CONFIG', arg) \ | ||
| .replace('$MAIN_CLASS', execute_class).replace('$UID', uid).replace('$SHUTDOWN_WAIT', shutdown_wait) \ | ||
| .replace('$TITLED', runName).replace('$MAX_RAM', max_ram).replace('$S3_PUBLISH', s3_publish) \ | ||
| .replace('$SIGOPT_CLIENT_ID', sigopt_client_id).replace('$SIGOPT_DEV_ID', sigopt_dev_id).replace('$END_SCRIPT', end_script) \ | ||
| .replace('$SLACK_HOOK_WITH_TOKEN', os.environ['SLACK_HOOK_WITH_TOKEN']) | ||
| instance_id = deploy(script, instance_type, region.replace("-", "_")+'_', shutdown_behaviour, runName, volume_size) | ||
| host = get_dns(instance_id) | ||
| txt = txt + 'Started batch: {batch} with run name: {titled} for branch/commit {branch}/{commit} at host {dns} (InstanceID: {instance_id}). '.format(branch=branch, titled=runName, commit=commit_id, dns=host, batch=uid, instance_id=instance_id) | ||
|
|
@@ -320,4 +368,4 @@ def lambda_handler(event, context): | |
| if command_id in instance_operations: | ||
| return instance_handler(event) | ||
|
|
||
| return "Operation {command} not supported, please specify one of the supported operations (deploy | start | stop | terminate | log). ".format(command=command_id) | ||
| return "Operation {command} not supported, please specify one of the supported operations (deploy | start | stop | terminate | log). ".format(command=command_id) | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| @startuml | ||
|
|
||
| scale 1000 width | ||
| skinparam stateBorderColor #black | ||
| skinparam stateFontStyle bold | ||
| skinparam arrowColor #black | ||
|
|
||
| ' BeamAgent: States ' | ||
| state ChoosingMode #eb1921 | ||
|
|
||
| state FinishingModeChoice #eb1921 | ||
|
|
||
| 'Transition State' | ||
|
|
||
| FinishingModeChoice -[#blue,dotted]-> WaitingForDeparture : IntermediateTransition | ||
| ChoosingMode --> ChoosingMode : MobilityStatusResponse | ||
| ChoosingMode --> ChoosingMode : RoutingResponse | ||
| ChoosingMode --> ChoosingMode : RideHailResponse | ||
| ChoosingMode --> ChoosingMode : ParkingInquiryResponse | ||
| ChoosingMode --> ChoosingMode : CavTripLegsResponse | ||
| ChoosingMode --> FinishingModeChoice | ||
|
|
||
|
|
||
| @enduml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| @startuml | ||
|
|
||
| scale 1000 width | ||
| skinparam stateBorderColor #black | ||
| skinparam stateFontStyle bold | ||
| skinparam arrowColor #black | ||
|
|
||
| ' BeamAgent: States ' | ||
| state ReleasingParkingSpot #eb1921 | ||
|
|
||
| state ChoosingParkingSpot #eb1921 | ||
|
|
||
| 'Transition State' | ||
| ReleasingParkingSpot --> ReleasingParkingSpot: StartLegTrigger | ||
| ReleasingParkingSpot -[#blue,dotted]-> WaitingToDrive: IntermediateTransition | ||
| ChoosingParkingSpot --> ChoosingParkingSpot : ParkingInquiryResponse | ||
| ChoosingParkingSpot --> WaitingToDrive : RoutingResponse | ||
|
|
||
|
|
||
| @enduml |
Oops, something went wrong.