Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gsoc2021 - Phase 3 #54

Open
wants to merge 33 commits into
base: gsoc2021
Choose a base branch
from

Conversation

akshitpatel01
Copy link

No description provided.

@@ -0,0 +1,36 @@
# Overview of Dashboard features
The objective of this interactive dashboard is to allow users to visualize the logs genearted by ns-3 efficiently.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The objective of this interactive dashboard is to allow users to visualize the logs genearted by ns-3 efficiently.
The objective of this interactive dashboard is to allow users to visualize the logs generated by ns-3 efficiently.

### The layout and features of the dashboard:
Starting from top, the dashboard is divided into the following sections:
###### Dropdown filters along with a time filter:
All the filter values in the dropdown are populated (according to the log file passed) as the dashboard first initializes. The dropdown filters have an inbuilt search box as well 'Select All' and 'Deselect All' buttons. A few points to note about the filters:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
All the filter values in the dropdown are populated (according to the log file passed) as the dashboard first initializes. The dropdown filters have an inbuilt search box as well 'Select All' and 'Deselect All' buttons. A few points to note about the filters:
All the filter values in the dropdown are populated (according to the log file passed) as the dashboard first initializes. The dropdown filters have an inbuilt search box as well as 'Select All' and 'Deselect All' buttons. A few points to note about the filters:

Starting from top, the dashboard is divided into the following sections:
###### Dropdown filters along with a time filter:
All the filter values in the dropdown are populated (according to the log file passed) as the dashboard first initializes. The dropdown filters have an inbuilt search box as well 'Select All' and 'Deselect All' buttons. A few points to note about the filters:
- These filters act as global flters for both the graph and the table.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- These filters act as global flters for both the graph and the table.
- These filters act as global filters for both the graph and the table.

###### Dropdown filters along with a time filter:
All the filter values in the dropdown are populated (according to the log file passed) as the dashboard first initializes. The dropdown filters have an inbuilt search box as well 'Select All' and 'Deselect All' buttons. A few points to note about the filters:
- These filters act as global flters for both the graph and the table.
- Any unspecified filters are assmed to be 'free' and can take any value. For example, if 'context=[0,1]', 'function=['f1']' and 'Upper Time Limit:1', then the logs with (context=0 or context=1) and (function=f1) and (timestamp<1) will be displayed.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Any unspecified filters are assmed to be 'free' and can take any value. For example, if 'context=[0,1]', 'function=['f1']' and 'Upper Time Limit:1', then the logs with (context=0 or context=1) and (function=f1) and (timestamp<1) will be displayed.
- Any unspecified filters are assumed to be 'free' and can take any value. For example, if 'context=[0,1]', 'function=['f1']' and 'Upper Time Limit:1', then the logs with (context=0 or context=1) and (function=f1) and (timestamp<1) will be displayed.

- If no component is selected from the component filter, then the severity class filter acts as an independent filter.

###### The Graph:
This is a line graph which allows users to easily analyze logs with resepect to a time axis. A few points to note about the graph:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This is a line graph which allows users to easily analyze logs with resepect to a time axis. A few points to note about the graph:
This is a line graph which allows users to easily analyze logs with respect to a time axis. A few points to note about the graph:

- Hold Ctrl and drag the mouse to pan the graph.

###### The Table:
The table allows users to effectivel visualize the logs in a tabular way. A few points to note about the table:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The table allows users to effectivel visualize the logs in a tabular way. A few points to note about the table:
The table allows users to effectively visualize the logs in a tabular way. A few points to note about the table:

###### The Table:
The table allows users to effectivel visualize the logs in a tabular way. A few points to note about the table:
- The 'Show entries' dropdown menu allows the users to select the number of logs to be displayed on each page.
- The search box above the table allows the users to search in sepecific columns. the searchable columns can be selected/deselected using the 'Search columns' dropdown as per need.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- The search box above the table allows the users to search in sepecific columns. the searchable columns can be selected/deselected using the 'Search columns' dropdown as per need.
- The search box above the table allows the users to search in specific columns. the searchable columns can be selected/deselected using the 'Search columns' dropdown as needed.

- The 'Show entries' dropdown menu allows the users to select the number of logs to be displayed on each page.
- The search box above the table allows the users to search in sepecific columns. the searchable columns can be selected/deselected using the 'Search columns' dropdown as per need.
- The search box is specific to the table and has no effect on the graph.
- Clicking on a row in the table jumps to the page where that particular row is diplayed in the table. For example, if user searches for 'xyz' and logs are displayed based on this search query, then on clicking any row, the table will jump to the page (in the original table without the search query) which contains that particular row.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Clicking on a row in the table jumps to the page where that particular row is diplayed in the table. For example, if user searches for 'xyz' and logs are displayed based on this search query, then on clicking any row, the table will jump to the page (in the original table without the search query) which contains that particular row.
- Clicking on a row in the table jumps to the page where that particular row is displayed in the table. For example, if user searches for 'xyz' and logs are displayed based on this search query, then on clicking any row, the table will jump to the page (in the original table without the search query) which contains that particular row.

- The search box above the table allows the users to search in sepecific columns. the searchable columns can be selected/deselected using the 'Search columns' dropdown as per need.
- The search box is specific to the table and has no effect on the graph.
- Clicking on a row in the table jumps to the page where that particular row is diplayed in the table. For example, if user searches for 'xyz' and logs are displayed based on this search query, then on clicking any row, the table will jump to the page (in the original table without the search query) which contains that particular row.
- All the columns of the table (except extended_context) can be sorted either in ascending or descending order.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- All the columns of the table (except extended_context) can be sorted either in ascending or descending order.
- All the columns of the table (except for extended_context) can be sorted either in ascending or descending order.

Copy link
Member

@mattia-lecci mattia-lecci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job as always!

I installed it in my home pc and was missing the following dependencies:

  • flask (installed 2.0.1)
  • tinydb-smartcache (installed 2.0.0)

They could be added to the .toml file as already discussed

Furthermore, when hovering over the plot it would be nice to also get time and context information, even if it's redundant with the information from the plot's axis, and even the index.

@@ -31,18 +31,21 @@
'STA1_x': 5.0,
'stepsTime': 1,
'minPower': 0,
'steps': 2,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this necessary?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just to ensure that the example does not take a very long time to run. The default steps value in the ns-3 example is set to 200. According to me, that might take quite a long time to run. What do you guys think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry for not being clear, i meant: why did this change from 2 to [2]? It's the only element set as a list.
If it's possible to make this dictionary uniform (either all lists or all values) i'd go for it

examples/logging_example2.py Show resolved Hide resolved
}
# log_components = 'NS_LOG="PowerAdaptationDistance=all:FrameExchangeManager=info"'

# log_components = 'NS_LOG="PowerAdaptationDistance=debug:ParfWifiManager=info"'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i suggest to show the two formats in the same configuration

logging.wipe_results(db, db_path)
db, data_dir = logging.process_logs(log_path[0])
print('Filtered Logs:')
print(logging.filter_logs(db,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe too verbose?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mattia, can you please check if you have the latest version of the example?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the output that i get running the script:

[{'index': 50667, 'time': 0.800403247, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 50734, 'time': 0.800725281, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 50801, 'time': 0.801074315, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 50863, 'time': 0.801459349, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 50930, 'time': 0.801817383, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 50997, 'time': 0.802184417, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51064, 'time': 0.802623451, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51131, 'time': 0.803026485, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51198, 'time': 0.803438519, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51265, 'time': 0.803868553, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51332, 'time': 0.804217587, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51394, 'time': 0.804602621, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51461, 'time': 0.804951655, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51528, 'time': 0.805318689, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51595, 'time': 0.805676723, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51657, 'time': 0.806052757, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51724, 'time': 0.806392791, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51791, 'time': 0.806723825, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51858, 'time': 0.807162859, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51925, 'time': 0.807619893, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 51992, 'time': 0.808004927, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 52054, 'time': 0.808344961, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 52121, 'time': 0.808792995, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 52188, 'time': 0.809169029, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 52255, 'time': 0.809626063, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}, {'index': 52322, 'time': 0.809948097, 'context': '1', 'extended_context': None, 'component': 'MinstrelWifiManager', 'function': 'DoReportRxOk', 'arguments': '', 'severity_class': 'DEBUG', 'message': 'DoReportRxOk m_txrate=6'}]

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My point is: does a user care about this output when running the example or is it just interested into the dashboard?
Or should we separate this in a third example showing the logging/filtering programmatic API, instead?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My point was just to demonstrate the API output but I agree with your point. I probably wouldn't make a new example just for this. Maybe keep the print statement and just comment it?

sem/dashboard/README.md Outdated Show resolved Hide resolved
sem/dashboard/templates/dashboard.html Outdated Show resolved Hide resolved
sem/dashboard/views.py Outdated Show resolved Hide resolved
sem/dashboard/views.py Outdated Show resolved Hide resolved
@@ -92,6 +94,7 @@ def parse_logs(log_file):
# Note: '^$' - Ensures that the entire regex matches the entire log line.
regex = re.compile(r'^' + time_re + context_extended_context_re + component_function_arguments_re + severity_class_message_re + r'$')

idx = 0
with open(log_file) as f:
for log in f:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one of python's best practices is to avoid creating manual counters when using loops.

Suggested change
for log in f:
for idx, log in enumerate(f):

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I initially thought of this but we have to increment the index only if the log line is parsed correctly. If the control goes here
if groups is None: warnings.warn("Log format is not consistent with prefix_all. Skipping log '%s'" % log, RuntimeWarning, stacklevel=2) continue

we do not increment the index.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, i get it now, thank you for the explanation.
Though, I would suggest the opposite: in this way row number and log index would be consistent, and a user would easily be able to find any given line if he needs to.
On the other hand, having the index and the row number as possibly different might be annoying for a user.

What do you guys think?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not quite understand what approach did you referred to as the opposite. Can you please elaborate a little? Also, I would like to just point out that the index that we store in the database is not visible to the user at any point. It is just to make searching for the log entries easier and faster(as the index are always sorted).

sem/utils.py Outdated Show resolved Hide resolved
@akshitpatel01
Copy link
Author

I installed it in my home pc and was missing the following dependencies:
flask (installed 2.0.1)
tinydb-smartcache (installed 2.0.0)
They could be added to the .toml file as already discussed

Thanks! I verified these on my system as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants