Skip to content

Commit

Permalink
Update competitions-pin_dev_merge with develop (#2232)
Browse files Browse the repository at this point in the history
* Add bike_transit

* Add bike_transit on other places

* Merge branch 'develop' into rajnikantsh/#1915-addBikeTransit-4ci

* Add bike_transit intercept

* change bike_transit mode as per drive transit

* Merge branch 'develop' into rajnikantsh/#1915-addBikeTransit-4ci

* add bike_transit in calibration file

* warmstart test

* Reused code from FileUtils

* Merge branch 'develop' into rajnikantsh/#1950-warmStart-test-4ci

* Merge branch 'develop' into rajnikantsh/#1950-warmStart-test-4ci

* add new files for warm start

* Merge branch 'develop' into rajnikantsh/#1950-warmStart-test-4ci

* scale utilities in mode choice

* Merge remote-tracking branch 'origin/develop' into zn/scale-utilites-for-mode-choice

* change bin size to 6 in relative speed

* adjust test cases

* Change graph yaxis label

* prioritize taking vehicles from people with short commutes

* fmt

* clean up

* fmt

* handle bad coords

* more logging

* more logging, improve speed hopefully

* further speedup

* mutable map of arraybuffers

* start collecting person score

* more general scoring

* simplify, add penalty for early and late trips

* revert files accidentally changed

* reduce early trip penalty

* bug

* Merge remote-tracking branch 'origin/develop' into zn/vehicle-retirement

* print vehicle type id

* Merge pull request #2180 from LBNL-UCB-STI/zn/#2179-refuelsessionevent-vehicle-id

#2179 print vehicle type id in refuelsessionevent

* make bushwhack trips normal walk speed, minkowski dist

* fmt

* scala fmt

* Merge pull request #2181 from LBNL-UCB-STI/zn/bushwhack-speed-distance

make bushwhack trips normal walk speed, minkowski dist

* Merge pull request #2176 from LBNL-UCB-STI/zn/vehicle-retirement

Consider commute in whether to retire household vehicles

* data to enable scenarios with RH EVs that charge

* update protocol so RHA drives to the charger... for now force it to happen even if distance is close... maybe will change later

* more tweaks to data to test out RH EV humans

* track vehicle location and use it to make trip to charger correct, then adjust parking stall sampler to be more appropriate

* merge

* fmt

* make the location snap to TAZ more quickly

* Merge branch 'develop' into rajnikantsh/#2143-add-bin-in-relativeNetwork-4ci

* skeleton for agent trip script

* Do not do manual GC (#2189)

* Generate trip table

* - Changed `interruptId` to be Int
- Added `InterruptIdIdGenerator` to generate monotonically increasing ids for interrupts

* Merge pull request #2191 from LBNL-UCB-STI/art/replace-interrupt-id-by-int-4ci

Use InterruptIdIdGenerator to generate interrupt ids

* fmt

* subtract fuel cost from trip cost within RHM, also fix divide by zero issue in PoolingInfo

* fmt

* Use ReservationRequestIdGenerator to generate requestId for reservation  #2191 (#2195)

* - Added `ReservationRequestIdGenerator`
- `ReservationRequest.requestId` is an integer and it uses `ReservationRequestIdGenerator`

* Sync-up test class name with actor name

* ridehail revenue fix

* merge

* Merge pull request #2194 from LBNL-UCB-STI/cs/#2192-remove-fuel-cost-from-rh-fare

Cs/#2192 remove fuel cost from rh fare

* Merge pull request #2185 from LBNL-UCB-STI/cs/rh-human-should-drive-to-charge

Cs/rh human should drive to charge

* Make sure arrays are order by hour

* Merge pull request #2166 from LBNL-UCB-STI/rajnikantsh/#2136-change-yaxis_label-4ci

Change graph yaxis label

* adding debug template

* Merge pull request #2199 from LBNL-UCB-STI/art/make-sure-arrays-are-sorted-by-hours-4ci

Make sure arrays are order by hour

* script added

* Add Repositioning Analysis

* Add CSV events files

* Scala fmt

* set requestBufferTimeoutInSeconds = 0

* heat_map/heatmap.py -> smaprt/mep/build_heatmap.py

* typo fixed

* Merge pull request #2201 from LBNL-UCB-STI/inm/script-for-draw-heatmap

inm/python script to draw a heatmap

* Changes which are in `production-sfbay-develop-debug`, but not in `production-sfbay-develop` (and not in `develop` too)

* change mode to only ride_hail

* Update RideHailWaitingAnalysis.java

* Make sure what is in AWS is what is in git (#2198)

* Bump to 2.12.9 (#2158)

* Bump to 2.12.9

* 10 is out now

* Fix warning in new scala version

* Merge

* fmt

* Refactoring - remove warnings and simplify code (#2144)

* replace type check with pattern matching

* use named arguments

* use named arguments

* removed redundant parenthesis

* removed unused import

* removed unused imports

* added name arguments

* fixed typo

* Merge branch 'develop' into rajnikantsh/#1950-warmStart-test-4ci

* Merge pull request #2161 from LBNL-UCB-STI/rajnikantsh/#2143-add-bin-in-relativeNetwork-4ci

change bin size to 6 in relative speed

* change PathTraversalEvent.apply instead of csv reading

* Merge remote-tracking branch 'remotes/origin/develop' into rw/#2187-trip-table

* fmt

* Merge remote-tracking branch 'remotes/origin/develop' into rajnikantsh/#1915-addBikeTransit-4ci

# Conflicts:
#	src/main/scala/beam/agentsim/agents/choice/mode/ModeChoiceMultinomialLogit.scala

* Merge pull request #2146 from LBNL-UCB-STI/zn/scale-utilites-for-mode-choice

scale utilities in mode choice

* change file formats

* change file format

* Merge remote-tracking branch 'remotes/origin/develop' into rajnikantsh/#1915-addBikeTransit-4ci

* set requestBufferTimeoutInSeconds to 0

* Merge pull request #2205 from LBNL-UCB-STI/art/what-is-left-to-merge-to-develop-4ci

Changes to bring back to develop because they were pushed directly to production-sfbay-develop-debug

* Add bike_transit test case

* Merge remote-tracking branch 'remotes/origin/develop' into rajnikantsh/#1915-addBikeTransit-4ci

# Conflicts:
#	src/main/resources/beam-template.conf

* resolve conflicts

* fmt

* Do not expect actor to be terminated (#2210)

* Merge pull request #2203 from LBNL-UCB-STI/rajnikantsh/repositioning-analysis-4ci

Add Repositioning Analysis

* Merge pull request #1960 from LBNL-UCB-STI/rajnikantsh/#1950-warmStart-test-4ci

beam warmstart test

* Remove unnecessary code

* Merge pull request #2207 from LBNL-UCB-STI/rw/#2187-trip-table

Rw/#2187 trip table

* Merge branch 'develop' into rajnikantsh/#1915-addBikeTransit-4ci

* fix warmstart test error

* Merge pull request #2212 from LBNL-UCB-STI/rajnikantsh/fix-warmstart-error-4ci

fix warmstart test error

* Stagger initial tick for repositioning (#2214)

* Stagger repositioning

* Initial tick for repositioning does not overlap with allocationManager init tick

* Merge pull request #1918 from LBNL-UCB-STI/rajnikantsh/#1915-addBikeTransit-4ci

Add bike_transit

* When remove `RideHailAgentLocation` from Spatial index compare only `vehicleId` as the cheapest field. This solves the problem of having multiple `RideHailAgentLocation` for the single `vehicleId` in spatial index

* Merge pull request #2219 from LBNL-UCB-STI/art/investigate-degradation-when-high-rhm-intercept-4ci

Fix the grow of idleRideHailAgentSpatialIndex

* Cover by unit test the behaviour of spatial index for `RideHailAgentLocation` (#2221)

* Merge branch 'develop' into competitions-pin_dev_merge
  • Loading branch information
Anyi Chen authored and sfwatergit committed Oct 19, 2019
1 parent 72ee3d1 commit 530dd88
Show file tree
Hide file tree
Showing 75 changed files with 2,296 additions and 365 deletions.
76 changes: 27 additions & 49 deletions aws/src/main/python/beam_lambda/lambda_function.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,36 +74,14 @@
- crontab -l
- echo "notification scheduled..."
- git fetch
- echo "git checkout ..."
- 'echo "git checkout: $(date)"'
- GIT_LFS_SKIP_SMUDGE=1 sudo git checkout $BRANCH
- sudo git pull
- sudo git lfs pull
- echo "git checkout -qf ..."
- GIT_LFS_SKIP_SMUDGE=1 sudo git checkout -qf $COMMIT
- echo "preparing for python analysis"
- sudo dpkg --configure -a
- sudo dpkg --remove --force-remove-reinstreq unattended-upgrades
- sudo apt-get install unattended-upgrades
- sudo dpkg --configure -a
- sudo apt update
- sudo apt install npm -y
- sudo apt install nodejs-legacy -y
- sudo apt install python-pip -y
- pip install --upgrade pip
- sudo pip install pandas
- sudo pip install plotly
- sudo pip install psutil requests
- sudo npm cache clean -f
- sudo npm install -g n
- sudo n stable
- sudo npm install -g npm
- sudo apt-get install curl
- curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
- sudo apt-get install nodejs -y
- sudo apt-get install libgtkextra-dev libgconf2-dev libnss3 libasound2 libxtst-dev -y
- sudo npm install -g [email protected] orca --unsafe-perm=true --alow-root -y
- sudo apt-get install xvfb -y
- echo "gradlew assemble ..."
- 'echo "gradlew assemble: $(date)"'
- ./gradlew assemble
- echo "looping config ..."
- export MAXRAM=$MAX_RAM
Expand Down Expand Up @@ -192,30 +170,30 @@ def validate(name):

def deploy(script, instance_type, region_prefix, shutdown_behaviour, instance_name, volume_size):
res = ec2.run_instances(BlockDeviceMappings=[
{
'DeviceName': '/dev/sda1',
'Ebs': {
'VolumeSize': volume_size,
'VolumeType': 'gp2'
}
}
],
ImageId=os.environ[region_prefix + 'IMAGE_ID'],
InstanceType=instance_type,
UserData=script,
KeyName=os.environ[region_prefix + 'KEY_NAME'],
MinCount=1,
MaxCount=1,
SecurityGroupIds=[os.environ[region_prefix + 'SECURITY_GROUP']],
IamInstanceProfile={'Name': os.environ['IAM_ROLE'] },
InstanceInitiatedShutdownBehavior=shutdown_behaviour,
TagSpecifications=[ {
'ResourceType': 'instance',
'Tags': [ {
'Key': 'Name',
'Value': instance_name
} ]
} ])
{
'DeviceName': '/dev/sda1',
'Ebs': {
'VolumeSize': volume_size,
'VolumeType': 'gp2'
}
}
],
ImageId=os.environ[region_prefix + 'IMAGE_ID'],
InstanceType=instance_type,
UserData=script,
KeyName=os.environ[region_prefix + 'KEY_NAME'],
MinCount=1,
MaxCount=1,
SecurityGroupIds=[os.environ[region_prefix + 'SECURITY_GROUP']],
IamInstanceProfile={'Name': os.environ['IAM_ROLE'] },
InstanceInitiatedShutdownBehavior=shutdown_behaviour,
TagSpecifications=[ {
'ResourceType': 'instance',
'Tags': [ {
'Key': 'Name',
'Value': instance_name
} ]
} ])
return res['Instances'][0]['InstanceId']

def get_dns(instance_id):
Expand Down
24 changes: 24 additions & 0 deletions aws/src/main/python/cloneAMI/lambda_function.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# coding=utf-8
import time
import json
import boto3
import logging
from botocore.errorfactory import ClientError

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
ami_id = event.get('ami_id')
image_name = event.get('ami_name')
region = event.get('region')

return copy_ami(image_name, ami_id, region)

def copy_ami(image_name, image_id, region):
ec2 = boto3.client('ec2',region_name=region)
res = ec2.copy_image(Name=image_name,
SourceImageId=image_id,
SourceRegion='us-east-2')
logger.info('ami [' + image_id + '] coppied to region ' + region + ' with id ' + res['ImageId'])
return res['ImageId']
31 changes: 31 additions & 0 deletions aws/src/main/python/createAMI/lambda_function.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# coding=utf-8
import time
import json
import boto3
import logging
from botocore.errorfactory import ClientError

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
instance_id = event.get('instance_id')
terminate_instance = event.get('terminate_instance', "false")

image_name = 'beam-automation-'+time.strftime("%Y-%m-%d-%H%M%S", time.gmtime())
ami_id = create_ami(image_name, instance_id, terminate_instance)

return ami_id + "|" + image_name


def create_ami(image_name, instance_id, terminate_instance):
ec2 = boto3.client('ec2',region_name='us-east-2')
logger.info('creating ami of instance' + instance_id)
res = ec2.create_image(InstanceId=instance_id,
Name=image_name, NoReboot=True)
logger.info('ami creation started for ' + res['ImageId'])
if terminate_instance == "true":
logger.info('shutting down instance' + instance_id)
ec2.terminate_instances(InstanceIds=[instance_id])
logger.info('shutting down complete for instance' + instance_id)
return res['ImageId']
52 changes: 32 additions & 20 deletions aws/src/main/python/updateBeamAMI/lambda_function.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,60 +2,72 @@
import time
import json
import boto3
import logging
from botocore.errorfactory import ClientError

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
instance_id = event.get('instance_id')
region_id = event.get('region_id', 'us-east-2')

image_name = 'beam-automation-'+time.strftime("%Y-%m-%d-%H%M%S", time.gmtime())
image_ids = {}



image_ids['us-east-2'] = create_ami(image_name, instance_id)
image_ids['us-east-1'] = copy_ami(image_name, image_ids['us-east-2'], 'us-east-1')
image_ids['us-west-2'] = copy_ami(image_name, image_ids['us-east-2'], 'us-west-2')
update_lambda(image_ids)

return json.dumps(image_ids)


def create_ami(image_name, instance_id):
ec2 = boto3.client('ec2',region_name='us-east-2')
logger.info('creating ami of instance' + instance_id)
res = ec2.create_image(InstanceId=instance_id,
Name=image_name)
Name=image_name)
wait4image(ec2, res['ImageId'])
ec2.terminate_instances(InstanceIds=[instance_id])
logger.info('ami created ' + res['ImageId'])
logger.info('shutting down instance' + instance_id)
ec2.terminate_instances(InstanceIds=[instance_id])
logger.info('shutting down complete for instance' + instance_id)
return res['ImageId']

def copy_ami(image_name, image_id, region):
ec2 = boto3.client('ec2',region_name=region)
res = ec2.copy_image(Name=image_name,
SourceImageId=image_id,
SourceRegion='us-east-2')
SourceImageId=image_id,
SourceRegion='us-east-2')
logger.info('ami [' + image_id + '] coppied to region ' + region + ' with id ' + res['ImageId'])
# wait4image(ec2, res['ImageId'])
return res['ImageId']

def wait4image(ec2, image_id):
waiter = ec2.get_waiter('image_available')
waiter.wait(Filters=[{'Name': 'state', 'Values': ['available']}],
ImageIds=[image_id])

def update_lambda(image_ids):
lm = boto3.client('lambda')
logger.info('updateing image ids ' + str(image_ids))
en_var = lm.get_function_configuration(FunctionName='simulateBeam')['Environment']['Variables']
en_var.update({
'us_east_2_IMAGE_ID': image_ids['us-east-2'],
'us_east_1_IMAGE_ID': image_ids['us-east-1'],
'us_west_2_IMAGE_ID': image_ids['us-west-2'],
})
'us_east_2_IMAGE_ID': image_ids['us-east-2'],
'us_east_1_IMAGE_ID': image_ids['us-east-1'],
'us_west_2_IMAGE_ID': image_ids['us-west-2'],
})
lm.update_function_configuration(
FunctionName='simulateBeam',
Environment={
'Variables': en_var
}
)
FunctionName='simulateBeam',
Environment={
'Variables': en_var
}
)
logger.info('simulateBeam image ids updated')



def check_instance_id(instance_ids):
for reservation in ec2.describe_instances()['Reservations']:
for instance in reservation['Instances']:
Expand Down
66 changes: 60 additions & 6 deletions aws/src/main/python/updateDependencies/lambda_function.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# coding=utf-8
import os
import boto3
from botocore.errorfactory import ClientError

initscript = (('''#cloud-config
runcmd:
Expand All @@ -12,20 +13,73 @@
- sudo git reset origin/HEAD
- sudo git checkout -- .
- sudo git clean -df
- git fetch
- sudo git checkout develop
- sudo git pull
- sudo git fetch
- sudo git fetch --prune
- sudo git remote prune origin
- echo "-git remote prune origin is done-"
- sudo chown -R ubuntu:ubuntu /home/ubuntu/git/beam
- for bn in $BRANCH
- do
- echo "-------------------checkout $bn----------------------"
- GIT_LFS_SKIP_SMUDGE=1 sudo git checkout $bn
- sudo GIT_LFS_SKIP_SMUDGE=1 git checkout $bn
- sudo git reset --hard origin/$bn
- sudo git pull
- sudo git lfs pull
- done
- sudo chown -R ubuntu:ubuntu /home/ubuntu/git/beam
- echo "gradlew assemble ..."
- ./gradlew assemble
- ./gradlew clean
- echo "invoke lambda ..."
- sudo aws lambda invoke --invocation-type RequestResponse --function-name updateBeamAMI --region 'us-east-2' --payload '{"instance_id":"'"$(ec2metadata --instance-id)"'","region_id":"us-east-2"}' outputfile.txt
- echo "preparing for python analysis"
- sudo dpkg --configure -a
- sudo dpkg --remove --force-remove-reinstreq unattended-upgrades
- sudo apt-get install unattended-upgrades
- sudo dpkg --configure -a
- sudo apt update
- sudo apt install npm -y
- sudo apt install nodejs-legacy -y
- sudo apt install python-pip -y
- pip install --upgrade pip
- sudo pip install pandas
- sudo pip install plotly
- sudo pip install psutil requests
- sudo npm cache clean -f
- sudo npm install -g n
- sudo n stable
- sudo npm install -g npm
- sudo apt-get install curl
- curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
- sudo apt-get install nodejs -y
- sudo apt-get install libgtkextra-dev libgconf2-dev libnss3 libasound2 libxtst-dev -y
- sudo npm install -g [email protected] orca --unsafe-perm=true --alow-root -y
- sudo apt-get install xvfb -y
- 'echo resetting git to base: "$(date)"'
- sudo git reset --hard
- 'echo fetching the latest: "$(date)"'
- sudo git fetch
- 'echo current git status: "$(date)"'
- sudo git status
- 'echo invoke create ami lambda after a 5 minute sleep to let the file system settle..."$(date)"'
- sudo sleep 5m
- sudo aws lambda invoke --invocation-type RequestResponse --function-name createAMI --region 'us-east-2' --payload '{"instance_id":"'"$(ec2metadata --instance-id)"'","region_id":"us-east-2"}' outputfile.txt
- export created_ami_id=$(sed -r 's/"(.*)\|.*/\\1/' outputfile.txt)
- export ami_filename=$(sed -r 's/.*\|(.*)"/\\1/' outputfile.txt)
- while ! aws ec2 describe-images --image-ids $created_ami_id --region us-east-2 | grep "available"; do echo "Waiting 30 seconds for AMI in us-east-2 $created_ami_id ..."; sleep 30s; done
- echo "invoke copy ami lambda to us-east-1 ..."
- sudo aws lambda invoke --invocation-type RequestResponse --function-name cloneAMI --region 'us-east-2' --payload '{"ami_id":"'"$created_ami_id"'","ami_name":"'"$ami_filename"'","region":"us-east-1"}' outputfile.txt
- export created_ami_id_us_east_1=$(sed -r 's/"(.*)"/\\1/' outputfile.txt)
- echo "invoke copy ami lambda to us-west-2 ..."
- sudo aws lambda invoke --invocation-type RequestResponse --function-name cloneAMI --region 'us-east-2' --payload '{"ami_id":"'"$created_ami_id"'","ami_name":"'"$ami_filename"'","region":"us-west-2"}' outputfile.txt
- export created_ami_id_us_west_2=$(sed -r 's/"(.*)"/\\1/' outputfile.txt)
- while ! aws ec2 describe-images --image-ids $created_ami_id_us_east_1 --region us-east-1 | grep "available"; do echo "Waiting 30 seconds for AMI in us-east-1 $created_ami_id_us_east_1 ..."; sleep 30s; done
- while ! aws ec2 describe-images --image-ids $created_ami_id_us_west_2 --region us-west-2 | grep "available"; do echo "Waiting 30 seconds for AMI in us-west-2 $created_ami_id_us_west_2 ..."; sleep 30s; done
- echo "invoke update simulateBeam lambda ..."
- sudo aws lambda invoke --invocation-type RequestResponse --function-name updateSimulateBeamEnvVars --region 'us-east-2' --payload '{"ami_id":"'"$created_ami_id"'","ami_id_us_east_1":"'"$created_ami_id_us_east_1"'","ami_id_us_west_2":"'"$created_ami_id_us_west_2"'"}' outputfile.txt
- echo "setting up auto shutdown ..."
- sudo shutdown -h +$SHUTDOWN_WAIT
- echo "shutdown in $SHUTDOWN_WAIT ..."
'''))


Expand Down Expand Up @@ -61,8 +115,8 @@ def lambda_handler(event, context):

region = 'us-east-2'
instance_type = os.environ['INSTANCE_TYPE']
shutdown_behaviour = 'stop'
shutdown_wait = "1"
shutdown_behaviour = 'terminate'
shutdown_wait = "10"
runName = 'update-beam-dependencies'
branches = os.environ['BRANCHES']

Expand Down
36 changes: 36 additions & 0 deletions aws/src/main/python/updateSimulateBeamEnvVars/lambda_function.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# coding=utf-8
import time
import json
import boto3
import logging
from botocore.errorfactory import ClientError

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
image_ids = {}
image_ids['us-east-2'] = event.get('ami_id')
image_ids['us-east-1'] = event.get('ami_id_us_east_1')
image_ids['us-west-2'] = event.get('ami_id_us_west_2')

update_lambda(image_ids)

return "Update complete"

def update_lambda(image_ids):
lm = boto3.client('lambda')
logger.info('updating image ids ' + str(image_ids))
en_var = lm.get_function_configuration(FunctionName='simulateBeam')['Environment']['Variables']
en_var.update({
'us_east_2_IMAGE_ID': image_ids['us-east-2'],
'us_east_1_IMAGE_ID': image_ids['us-east-1'],
'us_west_2_IMAGE_ID': image_ids['us-west-2']
})
lm.update_function_configuration(
FunctionName='simulateBeam',
Environment={
'Variables': en_var
}
)
logger.info('simulateBeam image ids updated')
4 changes: 2 additions & 2 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -189,11 +189,11 @@ dependencies {
/////////////////////////////////

// CORE Scala //
compile "org.scala-lang:scala-library:2.12.7"
compile "org.scala-lang:scala-library:2.12.10"
compile group: 'org.scala-lang.modules', name: "scala-xml_${scalaBinaryVersion}", version: '1.0.6'

// NEEDED FOR USING REPL //
compile "org.scala-lang:scala-compiler:2.12.7"
compile "org.scala-lang:scala-compiler:2.12.10"

// TEST Scala //
testCompile group: 'org.scalatest', name: "scalatest_${scalaBinaryVersion}", version: '3.0.1'
Expand Down
Loading

0 comments on commit 530dd88

Please sign in to comment.