DBe Consistency Check

[E3V3A]

This is the DBe Consistency Check detection specifications, as referenced by detection item 1 in the Detection List of #230. This detection is used to filter out bad data and corrupt BTS tower info when importing the external BTS DB from OCID and MLS etc. The table in this part of the database is called DBe_import.

The way it works is that the DBe is imported (from OCID), then all the BTS data in that table is checked for a few criteria, some of which are part of the settings. Depending on the detailed items checked for, DBe_import is updating the new table column called _rej_cause_ with an integer [0,5]. Please refer to issue #215 for the current DB structure.

IFF (rej_cause>0) we reject the BTS as bad and show it as in RED in the Map Viewer.

Here is the flowchart for this test.

I see there's a small mistake there. Should be only one Import DBe process...

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[tobykurien]

What is time_first and tf_settings (and where does one get those)? What is the value of min_gps_precision? What is RAT (and where does one get that)?

When does this flow chart execute? On first launch? Every minute? When a new BTS is connected to?

[E3V3A]

This should execute right after import of OCID data. time_first is the time stamp when the BTS was first seen from the imported data. tf_settings is the minimum of the above, that we allow, which is a new setting, that could be hard-coded for now, but later changeable. Similarly min_gps_precision, is
the minimum allowed GPS precision of the BTS, as shown. It is often included in the imported (into DBe_import) BTS data. RAT is the radio access technology used, i.e. GSM, UMTS, LTE etc.

[E3V3A]

After having attempted to lift some BTS constraints (MCC, MNC, LAC) in the boundary box used to specify which BTSs to download from OCID. (Due to 1000 BTS download limitation.) I've found that the OCID data contain a surprising quantity of "bad" measurements from strange LACs and CID, but also where both range and samples can be negative, without any explanation what that means. (I've contacted responsible OCID people, but still without response.) Thus we need to re-impose those constraints, which mean that the user will not see all BTS towers in his neighborhood, but only those that his phone would normally connect to.

These non-downloaded BTS, could complicate the detection as they'd be then be marked as red (fake BTS or IMSIC), if your phone, for some reason, need to connect to an antenna outside the downloaded LAC, which will happen if you're on a LAC boundary.

Thus making the implementation of this issue more urgent.

[SecUpwN]

These non-downloaded BTS, could complicate the detection as they'd be then be marked as red (fake BTS or IMSIC), if your phone, for some reason, need to connect to an antenna outside the downloaded LAC, which will happen if you're on a LAC boundary.

How often will that be the case, accumulated? I mean, this is counterproductive and without clean data to be used for detection, we'd be getting numerous false alarms and possibly a shit-storm of mad users...

[He3556]

No it is no problem If there are too many and maybe false BTS in the DB. Only If there are BTS missing in the OCID DB, this would be a problem. We need to use the lat/lon from the GPS Sensor to calculate the BTS in the area (for example 20km) If the user changes its position (for example 10km) we simply need to calculate again and add the BTS. Thats all

[He3556]

Why do we need to call it RAT ? Nobody will understad it + we show RAT = 2 for UMTS or is it LTE? Sometimes we should just keep things Like they are and Not change it again and again.

[E3V3A]

Well, yes, that is a valid question, but the RAT term is much more common than RAN, which is the more accurate one. In the code we have, we use RAT to indicate the numerical value, whereas we use Type as the text. The flow chart above is obviously code, which is the reason I use RAT there. Then in the DB Viewer for OCID (DBe_import) we (or at least I) try to make the info as compact as possible, which means we're showing RAT. But there is no problem adding the type to that info if we want.

As a side dev note: So yes, we should automatically re-download the ocid DB if location have changed more than X Km, unless we can find out in another way if we're on a LAC boundary. (ocid downloads are additive, so we don't loose old ones.)

[SecUpwN]

@E3V3A and @He3556 just as a reminder: If you are going to use any new or old terms either in coding or while descibing our App anywhere, please make sure we have that in our Glossary of Terms. Thanks!

[E3V3A]

I've created a separate issue #303 for better handling of OCID downloads. In addition I've gotten a positive response, from OCID maintainer, explaining the issue with negative values.

...This [will] be solved by data clean-up which is still in progress. Since 3 months we are cleaning up the data in the test environment which unfortunately takes more time than expected. Then we´ll repeat the same process for our production environment.

[E3V3A]

For the list of RAT versus Type check THIS developer page.

RAT	Type
0	unknown
1	GPRS
2	EDGE
3	UMTS
4	CDMA
5	EVDO revision 0
6	EVDO revision A
7	1xRTT
8	HSDPA
9	HSUPA
10	HSPA
11	iDen
12	EVDO revision B
13	LTE
14	eHRPD
15	HSPA+

• In the old database structure:
  • in opencellid table, the RAT column is called "Net" there is no RAT column, and
  • in cellinfo table, the RAT column is called "Net", and the Type column is called "NetworkType".
• In the new database structure:
  • in both the DBi_measure and DBi_import tables , the RAT column is called "RAT".

[E3V3A]

I've now implemented (7f7774a) the "avg_range, isGPSexact, Type" columns to the opencellid (DBe_import) table. What remains are the detection logic.

[SecUpwN]

Great thing, thanks for still rocking the hard things! Please don't forget to mention the things that are still missing until the Issue can be closed, otherwise developers will just sit there and monitor your work.

[SecUpwN]

@larsgrefer, would you please self-assign this Issue to you and see if if you can fix this in realm now?