gaspersic/robot-shop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Improved the python logging with the import of loggin module. Added documentation for the new behaviours of load-gen.sh and improved the previous descriptions a bit

Browse Source
This commit is contained in:
Teresa Noviello
2021-11-03 14:14:53 +00:00
parent 740d73549d
commit 18b7dd75c8
3 changed files with 58 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@@ -4,5 +4,6 @@
vendor/
Gopkg.lock
load-gen/utilities/__pycache__
load-gen/logs/php*
load-gen/logs/*.log
load-gen/logs/*.csv
agent/logs

26
load-gen/README.md
View File

@@ -14,7 +14,20 @@ $ ./load-gen.sh
Runs the load generation script against the application started with `docker-compose up` . There are various command line options to configure the load.
Alternatively, you can run the Container from Docker Hub directly on one of the nodes having access to the web service:
The script must be run in the load-gen directory. It logs all the php API calls into the file logs/php_services_calls.csv.
This command launches the load generator to run undefinitely (i.e. without time limits), simulating 5 clients calling the API reachable at the default URL http://localhost:8080:
```shell
$ ./load-gen.sh \
-h http://host:port/
-n 5 \
-v
```
The command also logs comprehensive details of all the API called in the file logs/calls.log, triggered by the option `-v` .
Alternatively, you can run the Container from Docker Hub directly on one of the nodes having access to the web service. Here there is an example of how to do it and an explanation for the variables involved:
```shell
$ docker run \
@@ -22,11 +35,14 @@ $ docker run \
--rm \
--name="loadgen" \
--network=host \
--volume ${PWD}/logs:/load/logs \
--volume ${PWD}/utilities:/load/utilities \
-e "HOST=http://host:8080/"
-e "NUM_CLIENTS=5" \
-e "RUN_TIME=1h30m" \
-e "ERROR=1" \
-e "SILENT=1" \
-e "LOAD_DEBUG=0" \
robotshop/rs-load
```
@@ -36,7 +52,11 @@ Set the following environment variables to configure the load:
* NUM_CLIENTS - How many simultaneous load scripts to run, the bigger the number the bigger the load. The default is 1
* RUN_TIME - For NUM_CLIENTS greater than 1 the duration to run. If not set, load is run for ever with NUM_CLIENTS. See below.
* ERROR - Set this to 1 to have erroroneous calls made to the payment service.
* SILENT - Set this to 1 to surpress the very verbose output from the script. This is a good idea if you're going to run load for more than a few minutes.
* SILENT - Set this to 1 to surpress the very verbose output to the stdout from the script. This is a good idea if you're going to run load for more than a few minutes.
* LOAD_DEBUG - Set this to 1 to enable the output of every API call produced from the script into the log file logs/calls.log. This is a good idea if you're going to investigate over occurred events during load generation.
The load generator logs all the php API calls into the file logs/php_services_calls.csv, despite the variables SILENT and LOAD_DEBUG being set, respectively, to 1 and 0.
The content of the directory logs is cleaned everytime the script load-gen.sh is called.
## Kubernetes
@@ -67,4 +87,4 @@ $ ./load-gen.sh \
-t 1h30m
```
The load will be run with `10` clients for `1h30m` before dropping down to `1` client for `1h30m` then looping back to `10` clients etc.
The load will be run with `10` clients for `1h30m` before dropping down to `1` client for `1h30m` then looping back to `10` clients etc.

57
load-gen/robot-shop.py
View File

@@ -1,5 +1,6 @@
import os
import random
import logging
from locust import HttpUser, task, between
from utilities.CSVWriter import CSVWriter
@@ -29,26 +30,31 @@ class UserBehavior(HttpUser):
"60.242.161.215"
]
phplog = None
php_services_api_prefix = '/api/ratings/api'
php_service_rate = '/rate'
php_service_fetch = '/fetch'
php_fieldnames = ['REQTYPE', 'SERVICE', 'INPUT', 'HEADER']
php_fieldnames = ['REQTYPE', 'SERVICE', 'INPUT', 'HEADER', 'ERRFLAG']
my_csv_writer = None
def on_start(self):
""" on_start is called when a Locust start before any task is scheduled """
print('Starting')
print('LOAD_DEBUG: ', os.environ.get("LOAD_DEBUG"))
print("ARGS ARE:\n\"")
print("\n".join(argv))
print('End of ARGS logging.infoing\n')
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)
if os.environ.get('LOAD_DEBUG') == '1':
print("ARGS ARE:\n\"")
print("\n".join(argv))
print('End of ARGS printing\n')
print('on start. php_fieldnames:{}', format(self.php_fieldnames))
logging.basicConfig(filename='logs/calls.log', format='%(asctime)s [%(levelname)s] - %(message)s', encoding='utf-8', level=logging.DEBUG)
else:
logging.basicConfig(filename='logs/calls.log', format='%(asctime)s [%(levelname)s] - %(message)s', encoding='utf-8', level=logging.WARNING)
logging.info('Starting')
logging.info('LOAD_DEBUG: %s', os.environ.get("LOAD_DEBUG"))
logging.info('on start. php_fieldnames: %s', format(self.php_fieldnames))
self.my_csv_writer = CSVWriter("logs/php_services_calls.csv", self.php_fieldnames)
self.phplog = open("logs/php.log", "a")
@task
def login(self):
@@ -64,7 +70,7 @@ class UserBehavior(HttpUser):
@task
def load(self):
self.phplog.write('new user, new load task\n')
logging.info('new user, new load task\n')
fake_ip = random.choice(self.fake_ip_addresses)
self.client.get('/', headers={'x-forwarded-for': fake_ip})
@@ -82,30 +88,33 @@ class UserBehavior(HttpUser):
if item['instock'] != 0:
break
self.phplog.write('fake_ip: {}\n'.format(fake_ip))
headers={'x-forwarded-for': fake_ip}
# vote for item
if randint(1, 10) <= 3:
ratevalue = randint(1, 5)
put_rate_api_str = '{}{}/{}/{}'.format(self.php_services_api_prefix, self.php_service_rate, item['sku'], ratevalue )
if os.environ.get('LOAD_DEBUG') == '1':
self.phplog.write('ratevalue: {}\n'.format(ratevalue))
self.phplog.write('item: {}\n'.format(item['sku']))
self.phplog.write('put_rate_api_str: {}\n'.format(put_rate_api_str))
self.phplog.flush()
self.my_csv_writer.writerow({'REQTYPE': 'PUT', 'SERVICE': '{}'.format(self.php_service_rate), 'INPUT': '{}/{}'.format(item['sku'], ratevalue ), 'HEADER': '{}'.format(headers)})
self.client.put(put_rate_api_str, headers)
logging.info('item: {} ratevalue: {} put_rate_api_str: {} by: {}\n'.format(item['sku'], ratevalue, put_rate_api_str, fake_ip))
try:
self.client.put(put_rate_api_str, headers)
self.my_csv_writer.writerow({'REQTYPE': 'PUT', 'SERVICE': '{}'.format(self.php_service_rate), 'INPUT': '{}/{}'.format(item['sku'], ratevalue ), 'HEADER': '{}'.format(headers), 'ERRFLAG': '{}'.format("")})
except BaseException as err:
logging.warnign("Last call generated an error")
logging.exception()
self.my_csv_writer.writerow({'REQTYPE': 'PUT', 'SERVICE': '{}'.format(self.php_service_rate), 'INPUT': '{}/{}'.format(item['sku'], ratevalue ), 'HEADER': '{}'.format(headers), 'ERRFLAG': '{}'.format(err)})
pass
self.client.get('/api/catalogue/product/{}'.format(item['sku']), headers={'x-forwarded-for': fake_ip})
get_rate_api_str = '{}{}/{}'.format(self.php_services_api_prefix, self.php_service_fetch, item['sku'])
if os.environ.get('LOAD_DEBUG') == '1':
self.phplog.write('item: {}\n'.format(item['sku']))
self.phplog.write('get_rate_api_str: {}\n'.format(get_rate_api_str))
self.phplog.flush()
self.my_csv_writer.writerow({'REQTYPE': 'GET', 'SERVICE': '{}'.format(self.php_service_fetch), 'INPUT': '{}'.format(item['sku']), 'HEADER': '{}'.format(headers) })
self.client.get(get_rate_api_str, headers={'x-forwarded-for': fake_ip})
logging.info('item: {} get_rate_api_str: {} by: {}\n'.format(item['sku'], get_rate_api_str, fake_ip))
try:
self.client.get(get_rate_api_str, headers={'x-forwarded-for': fake_ip})
self.my_csv_writer.writerow({'REQTYPE': 'GET', 'SERVICE': '{}'.format(self.php_service_fetch), 'INPUT': '{}'.format(item['sku']), 'HEADER': '{}'.format(headers), 'ERRFLAG': '{}'.format("") })
except BaseException as err:
logging.warnign("Last call generated an error")
logging.exception()
self.my_csv_writer.writerow({'REQTYPE': 'GET', 'SERVICE': '{}'.format(self.php_service_fetch), 'INPUT': '{}'.format(item['sku']), 'HEADER': '{}'.format(headers), 'ERRFLAG': '{}'.format(err) })
pass
self.client.get('/api/cart/add/{}/{}/1'.format(uniqueid, item['sku']), headers={'x-forwarded-for': fake_ip})