Skip to content

Commit

Permalink
Feature/solve post 4.0 issues (#24) (#25)
Browse files Browse the repository at this point in the history
* Improvements in Mongo connection including batch script for windows and warnings, minor fixes plus parameterised JUnit allowing a mongo environment to be used for connection testing.  This is something to be expanded for use significantly in future allowing full automated testing of saving to NS too

* Fixed a couple bugs.  One in advisor related to how dates are handled.  So changed all date parsing to use the pattern matching and created date formats.  Also added Junit tests to check each format works and a performance test case comparing 1,000,000 calls using pattern matching and a fixed date format.  Second bug seemed to prevent window synchronisation from working.  The key thread was being conditionally created - and I've no idea why.  Removed the condition so now always created and synch from window works once again

* Completed changes for v4.1

* included an assert in date unit test
  • Loading branch information
gh-davidr authored Jan 30, 2022
1 parent 8d4f2dc commit 9257c19
Show file tree
Hide file tree
Showing 36 changed files with 1,226 additions and 3,547 deletions.
96 changes: 69 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,11 @@ A desktop application with the following purposes in mind:
![picture](resources/images/MainPage.jpg)

Meter & Pump Data can be regularly loaded from a range of systems:
* Roche (SQL Server Database load as well as CSV export)
* Medtronic (csv)
* Libre view (csv)
* Diasend (xls)
* Medtronic (csv)
* Tandem t:slim (csv)
* Cellnovo (csv)


A serial stream of BG, Carb and Insulin is intelligently grouped together into a Treatment. The tool will ignore previously loaded data. It will also identify contention between manually entered Care Portal entries and data from meter/pump.
Expand Down Expand Up @@ -81,53 +82,94 @@ Nightscout Loader helps get real **Insights** from reams of *Data*.
## Is it for me?

- [x] Do you use Nightscout and want to upload Treatment (and or Diasend CGM) data to Nightscout from a supported Meter and or Pump?
- Then set up for Nightscout use
- [x] Do you want an extra tool to help analyze BGs, Carbs & Insulin? (Nightscout use is optional)
- Then set up either for Nightscout use or non-Nightscout use
- [x] Do you use Nightscout and want an extra tool to visualize repeated patterns of CGM data?
- [x] Do you have Autotune configured on a separate Linux machine and want assistance with using it - particularly if you also use Diasend?
- Features were initially incorporated to help with managing a virtual Linux server like AWS to run AutoTune. Since then, [Autotune Web](https://autotuneweb.azurewebsites.net/) is available and NightscoutLoader does not yet integrate

If any of the above are true, then it could help you.

## Installation Guide

### System Requirements

The application is built for desktop or laptop computers with Java. It requires at least Java version 7 and the installation guide includes a link to check and install if necessary.
The application is built for desktop or laptop computers with Java. It requires at least Java version 8 and the installation guide includes a link to check and install if necessary.


It loads the Nightscout Treatment and CGM into memory for comparison with new data being uploaded or for analysis. The amount of memory your system needs will depend on how much data you pull in. As a guide, 8 years' worth of treatment data and 20,000 CGM entries takes up 300Mb of RAM.


### Installation Guide for Nightscout use
### Installation Guide for Nightscout use on Windows

1. Check your version of Java at this [link](https://java.com/en/download/help/version_manual.xml)
1. Download the application from the [Nightscout Loader](https://david50703.wixsite.com/nightscoutloader) page.
1. Latest stable version always available [here](https://david50703.wixsite.com/nightscoutloader).
1. Occassionally, an early beta version is made available with fixes on the stable version or access to upcoming features.
2. Copy the NightScoutLoader.jar file to your desktop
3. Launch the application by double-clicking the icon on your desktop.
1. Download the application from the [Github Releases](https://github.com/gh-davidr/NightscoutLoader/releases) page.
1. Occassionally, an early beta version is made available with fixes on the stable version or access to upcoming features.
2. Copy the **NightScoutLoader.jar** and **NightScoutLoader.bat** files to your desktop (or any other location), ensuring that both files **are in the same folder**.
1. (Note that the loadLibreview.sh and loadDiasend.sh files are template Linux files for running NightscoutLoader as a batch and so don't need to be copied)
3. Launch the application by double-clicking the **NightScoutLoader.bat** file
1. The **NightScoutLoader.bat** script checks the jar file is correctly located. If it is not correctly located, then it will alert and exit.
1. It will also attempt to create C:\Temp folder for the main application log file. If this attempt fails then it will alert and exit.
4. Click Tools => Settings to launch the Settings panel
5. Complete the Sections in Yellow
6. For example
1. if your user is "my_mongo_user"
2. password is "my_mongo_pass"
3. port is 123
4. database is "my_mongodb"
5. then, please see how the Nightscout Server, Nightscout DB and Nightscout Collection fields are set:
6. Nightscout Server would "mongodb://my_mongo_user:my_mongo_pass@ds123.mongolab.com:123"
7. Nightscout DB would be "my_mongodb"
8. Nightscout Collection is always “treatments”


5. Complete the sections in Yellow
6. For example:
1. if your user is "my_mongo_user"
2. password is "my_mongo_pass"
4. database is "my_mongodb"
5. then Nightscout Server would be "mongodb+srv://my_mongo_user:my_mongo_pass@ds123.mongodb.net/my_mongo_db?retryWrites=true&w=majority"
(Note that ds123.mongodb.net will be different for your own set up)
7. Nightscout DB would be "my_mongo_db"
8. Nightscout Collection is always �treatments�
12. Once the data is loaded, an analysis automatically runs but a more detailed report can be generated by selecting **Action** menu and then **Analyze Results**
13. Select the file type to load (it may show Cell Novo initially)
10. Navigate to the file to load by clicking the **Select** button
11. Start the synchronisation by selecting **Action** menu and then **Synchronise**
12. Once the data is loaded, an analysis automatically runs but a more detailed report can be generated by selecting **Action** menu and then **Analyze Results**
To convert a previous installation for non-Nightscout to one for Nightscout use, simply complete steps 5 and onwards above, save the settings and re-launch the application.

### Installation Guide for non-Nightscout use
### Installation Guide for non-Nightscout use on Windows

1. Check your version of Java at this [link](https://java.com/en/download/help/version_manual.xml)
1. Download the application from the [Nightscout Loader](https://david50703.wixsite.com/nightscoutloader) page.
1. Latest stable version always available [here](https://david50703.wixsite.com/nightscoutloader).
1. Occassionally, an early beta version is made available with fixes on the stable version or access to upcoming features.
2. Copy the NightScoutLoader.jar file to your desktop
3. Launch the application by double-clicking the icon on your desktop.
1. Download the application from the [Github Releases](https://github.com/gh-davidr/NightscoutLoader/releases) page.
1. Occassionally, an early beta version is made available with fixes on the stable version or access to upcoming features.
2. Copy the **NightScoutLoader.jar** and **NightScoutLoader.bat** files to your desktop (or any other location), ensuring that both files **are in the same folder**.
1. (Note that the loadLibreview.sh and loadDiasend.sh files are template Linux files for running NightscoutLoader as a batch)
3. Launch the application by double-clicking the **NightScoutLoader.bat** file
1. The **NightScoutLoader.bat** script checks the jar file is correctly located. If it is not correctly located, then it will alert and exit.
1. It will also attempt to create C:\Temp folder for the main application log file. If this attempt fails then it will alert and exit.
3. Launch the application by double-clicking the **NightScoutLoader.bat** file
9. Select the file type to load (it may show Cell Novo initially)
10. Navigate to the file to load by clicking the **Select** button
11. Load the file by selecting **Action** menu and then **Load Meter/Pump Only**
12. Once the data is loaded, an analysis automatically runs but a more detailed report can be generated by selecting **Action** menu and then **Analyze Results**

### Installation Guide for Mac

1. Sorry - I don't have regular access to a Mac that would enable the provision of instructions for use.
1. The application will definitely run on MacOS (as reported by existing MacOS users) and it should be sufficient to launch it from the **NightScoutLoader.jar** file without needing to have a run script.
1. Perhaps someone can advise.


### Obtaining the MongoDB URI from Heroku

Setting MongoDB for Nightscout Server:
1. Login to Heroku (https://id.heroku.com/login)
2. Click on your Nightscout application
3. Click on "Settings" tab
4. In "Config Vars", click on "Reveal Config Vars"
5. Scroll down to "MONGODB_URI"
6. Copy value to right - which starts with "mongodb+srv://........"
7. Paste this URI text into "Nightscout Server" of NightscoutLoader's Settings screen

### Running NightscoutLoader as a batch

NightscoutLoader can be run with arguments from the command line instructing it to synchronise data from a file (such as Diasend for example). In this way, it can be incorporated into a more streamlined set up to speed the process of getting data into Nightscout from a downloaded file.

A couple of redacted Bash shell scripts are included in the bin folder which the author uses on Linux for this purpose. They can be used to write equivalent Windows bat scripts if desired.


---

That's all there is to it!

Expand Down
119 changes: 119 additions & 0 deletions bin/NightScoutLoader.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
@ECHO OFF
REM ---------------------------------------------------------------------------
REM
REM Run script for launching the NightScoutLoader application
REM
REM 1 Checks that jar file is correctly installed in folder
REM 2 Checks that the C:\Temp folder exists for log file
REM 3 Ensures that correct parameters are passed to runtime JVM
REM
REM ---------------------------------------------------------------------------
SETLOCAL enabledelayedexpansion

REM
REM Set some variables up
REM
SET jarfile=NightScoutLoader.jar
SET tmpdir=C:\Temp
SET msgdir=%temp%

REM
REM Ensure that the script runs from the same directory the script resides in
REM The assumption is that both the BAT script and the JAR file are in the same
REM directory.
REM
cd /D "%~dp0"

REM ---------------------------------------------------------------------------
REM
REM Do some basic checks and alert if needed
REM
REM ---------------------------------------------------------------------------

REM
REM Java file needs to be present
REM
if NOT EXIST %jarfile% (
CALL :displayMessage CRITICAL "NightScoutLoader Jar file needs to be present as '%jarfile%'" "" "Did the download append '(1)' to the filename perhaps?" "" "If so, please rename the Jar file to '%jarfile%' in directory %~dp0"
SET ERRORLEVEL=1
EXIT /B %ERRORLEVEL%
)

REM
REM NightScoutLoader logs to %tmpdir% so ensure it exists too
REM
if NOT EXIST %tmpdir% (
md %tmpdir%
if NOT EXIST %tmpdir% (
CALL :displayMessage EXCLAMATION "NightScoutLoader relies on %tmpdir% by default for its log file" "" "However, you don't have permission to create this folder" "" "Use elevated permissions to create manually"
SET ERRORLEVEL=1
EXIT /B %ERRORLEVEL%
)
CALL :displayMessage INFORMATION "Successfully created %tmpdir% for NightScoutLoader log file" "" "Select OK to continue"
)

REM
REM Invoke NightScoutLoader with options to increase memory and also with the TLS option set
REM TLS is needed to successfully connect to Mongo Atlas on Windows
REM
java -Xmx1024m -Xms128m -Djdk.tls.client.protocols=TLSv1.2 -jar %jarfile%
EXIT /B %ERRORLEVEL%

REM ---------------------------------------------------------------------------
REM
REM Display Message function for notifications
REM
REM %~1 CRITICAL, QUESTION, EXCLAMATION or INFORMATION - sets msgbox symbol
REM %~2 First line message
REM %~2 Second line message
REM ...
REM %~2 Fifth line message
REM ---------------------------------------------------------------------------
:displayMessage
SET symbol=""
SET header=""

if "%~1"=="CRITICAL" (
SET symbol=, vbCritical + vbOKOnly
SET header="SETUP ERROR:"
)
if "%~1"=="QUESTION" (
SET symbol=, vbQuestion + vbOKOnly
SET header=""
)
if "%~1"=="EXCLAMATION" (
SET symbol=, vbExclamation + vbOKOnly
SET header="SETUP ERROR:"
)
if "%~1"=="INFORMATION" (
SET symbol=, vbInformation + vbOKOnly
SET header="Please Note:"
)
shift

IF NOT "%~5"=="" goto fiveLines
IF NOT "%~4"=="" goto fourLines
IF NOT "%~3"=="" goto threeLines
IF NOT "%~2"=="" goto twoLines
IF NOT "%~1"=="" goto oneLine

:oneLine
echo MSGBOX %header% ^& vbCrLf ^& vbCrLf ^& "%~1" %symbol% > %msgdir%\TEMPmessage.vbs
goto runVB
:twoLines
echo MSGBOX %header% ^& vbCrLf ^& vbCrLf ^& "%~1" ^& vbCrLf ^& "%~2" %symbol% > %msgdir%\TEMPmessage.vbs
goto runVB
:threeLines
echo MSGBOX %header% ^& vbCrLf ^& vbCrLf ^& "%~1" ^& vbCrLf ^& "%~2" ^& vbCrLf ^& "%~3" %symbol% > %msgdir%\TEMPmessage.vbs
goto runVB
:fourLines
echo MSGBOX %header% ^& vbCrLf ^& vbCrLf ^& "%~1" ^& vbCrLf ^& "%~2" ^& vbCrLf ^& "%~3" ^& vbCrLf ^& "%~4" %symbol% > %msgdir%\TEMPmessage.vbs
goto runVB
:fiveLines
echo MSGBOX %header% ^& vbCrLf ^& vbCrLf ^& "%~1" ^& vbCrLf ^& "%~2" ^& vbCrLf ^& "%~3" ^& vbCrLf ^& "%~4" ^& vbCrLf ^& "%~5" %symbol% > %msgdir%\TEMPmessage.vbs
goto runVB

:runVB
call %msgdir%\TEMPmessage.vbs
del %msgdir%\TEMPmessage.vbs /f /q
EXIT /B 0
36 changes: 36 additions & 0 deletions bin/loadDiasend.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/bin/bash

#
# Linux script to run NightScoutLoader to load all Diasend files from the $inDir
# in succession, one after the other.
# As each file gets processed, the file is moved to $arDir archive directory and
# timestamped with processing date/time
#
# Variables below need to be set carefully
#

dir="" # Directory of the NightScoutLoader.jar file and probably this script too
inDir="" # Directory where Diasend files to upload are put
arDir="" # Directory where Diasend files once processed get archived
log="${dir}/loadDiasend.log"
jarLog="${dir}/NightscoutLoader_Diasend.log"
jar="${dir}/NightScoutLoader.jar"
msr="" # Set to your MongoDB URI
mdb="" # Set to your MongoDB DB
weeks=104

processFile()
{
file=$1
ofile=`basename "${file}" .xls`.`echo $(date +%FT%H%M%S)`.xls

echo "$(date +%FT%H%M%S) - Processing Diasend file $file" | tee -a $log
java -Xmx1024m -Xms128m -jar $jar -m diasend -f "${file}" -s $msr -d $mdb -w $weeks -l "$jarLog" | tee -a $log
echo "Archiving file as $ofile in $arDir"
mv "${file}" "${arDir}/${ofile}"
}

for f in ${inDir}/*;
do
processFile "$f"
done
36 changes: 36 additions & 0 deletions bin/loadLibreview.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/bin/bash

#
# Linux script to run NightScoutLoader to load all LibreView files from the $inDir
# in succession, one after the other.
# As each file gets processed, the file is moved to $arDir archive directory and
# timestamped with processing date/time
#
# Variables below need to be set carefully
#

dir="" # Directory of the NightScoutLoader.jar file and probably this script too
inDir="" # Directory where LibreView files to upload are put
arDir="" # Directory where LibreView files once processed get archived
log="${dir}/loadLibreview.log"
jarLog="${dir}/NightscoutLoader_Libre.log"
jar="${dir}/NightScoutLoader.jar"
msr="" # Set to your MongoDB URI
mdb="" # Set to your MongoDB DB
weeks=104

processFile()
{
file=$1
ofile=`basename "${file}" .csv`.`echo $(date +%FT%H%M%S)`.csv

echo "$(date +%FT%H%M%S) - Processing Libreview file $file" | tee -a $log
java -Xmx1024m -Xms128m -jar $jar -m libreview -f "${file}" -s $msr -d $mdb -w $weeks -l "$jarLog" | tee -a $log
echo "Archiving file as $ofile in $arDir"
mv "${file}" "${arDir}/${ofile}"
}

for f in ${inDir}/*;
do
processFile "$f"
done
18 changes: 4 additions & 14 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,11 @@
<version>3.13</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.mongodb/mongodb-driver -->
<!-- https://mvnrepository.com/artifact/org.mongodb/mongo-java-driver -->
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongodb-driver</artifactId>
<version>3.11.1</version>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>3.12.10</version>
</dependency>

<!-- https://mvnrepository.com/artifact/com.jcraft/jsch -->
Expand Down Expand Up @@ -141,16 +141,6 @@
</pluginManagement>

<plugins>

<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/analysis/Analyzer.java
Original file line number Diff line number Diff line change
Expand Up @@ -1304,7 +1304,7 @@ public static Date getLastDateFromDBResults(ArrayList<DBResult> resultList)
DBResult latest = resultList.get(0);

try {
result = CommonUtils.convertDateString(latest.getM_CP_EventTime(), DBResult.getCP_EventTimeFormat());
result = CommonUtils.convertDateString(latest.getM_CP_EventTime());
} catch (ParseException e)
{
m_Logger.log(Level.SEVERE, "<getLastDateFromDBResults>" + ". Unexpected Exception converting date. " + e.getMessage());
Expand All @@ -1327,7 +1327,7 @@ public static Date getFirstDateFromDBResults(ArrayList<DBResult> resultList)
DBResult latest = resultList.get(0);

try {
result = CommonUtils.convertDateString(latest.getM_CP_EventTime(), DBResult.getCP_EventTimeFormat());
result = CommonUtils.convertDateString(latest.getM_CP_EventTime());
} catch (ParseException e)
{
m_Logger.log(Level.SEVERE, "<getFirstDateFromDBResults>" + ". Unexpected Exception converting date. " + e.getMessage());
Expand Down
6 changes: 3 additions & 3 deletions src/main/java/analysis/AnalyzerTrendResult.java
Original file line number Diff line number Diff line change
Expand Up @@ -365,9 +365,9 @@ public boolean checkForResult(AnalyzerSingleResult analyzerSingleResult2,

try
{
dt1 = CommonUtils.convertDateString(getM_AnalyzerSingleResult1().getM_DBResult().getM_CP_EventTime(), DBResult.getCP_EventTimeFormat());
dt2 = CommonUtils.convertDateString(analyzerSingleResult2.getM_DBResult().getM_CP_EventTime(), DBResult.getCP_EventTimeFormat());

dt1 = CommonUtils.convertDateString(getM_AnalyzerSingleResult1().getM_DBResult().getM_CP_EventTime());
dt2 = CommonUtils.convertDateString(analyzerSingleResult2.getM_DBResult().getM_CP_EventTime());
//m_Logger.log(Level.FINE, "<"+this.getClass().getName()+"> checkForResult " + " Comparing dt1 " + dt1 + " with dt2 " + dt2);

}
Expand Down
Loading

0 comments on commit 9257c19

Please sign in to comment.