Change all uses of history data protobuf messages so that they will not crash the calling script in the case where the grpc protocol removes the message or field being accessed. Instead, they will return None for the affected field (which most calling scripts interpret as "no data") or raise the same error as when the dish is not reachable.
Same caveats about code readability and adherence to the advertised type data as the change for protecting status usage.
This is for issue #66.
Change all uses of status data protobuf messages so that they will not crash the calling script in the case where the grpc protocol removes the message, field, or enum value being accessed. Instead, they will return None for the affected field (which most calling scripts interpret as "no data") or raise the same error as when the dish is not reachable.
This makes the code a little less readable, but it's better than breaking every time the protocol obsoletes fields.
mypy now complains about the return types for some fields that are now technically optional but are not marked as such in the type data, because the type data reflects what will currently be returned, not what may turn to None in the future if the protocol changes.
This is for issue #66.
The dish firmware stopped reporting the wedge fraction obstructed detail a few weeks ago, and now has apparently dropped it from the grpc protocol.
The core module now reports a sequence of None values for those fields, which is represented as "no data" in most of the database outputs. Without meaningful data in these fields, the obstruction_detail mode group is mostly useless, but the obstruction map was a much better representation of that data, anyway.
This should address issue #65.
I don't necessarily think all of these make the code better or easier to read, but it's easier to just go along with what the tools want, since they do generally make things better.
There should be no functional changes here.
Add CSV header and sqlite schema support for the newly added location group, along with the necessary functions in the starlink_grpc module to expose the field names and types for that group. This really should have gone in with the prior change.
Also, correct the order of the ping_latency and ping_loaded_latency groups in the CSV header, which was backwards. This would have resulted in an incorrect CSV header if both those groups were selected.
Add a new mode group, location, for physical location data. This requires access to the location data being enabled on the dish in order to return data; see README file for more details. At least for now, this functionality should be considered experimental.
This is being lumped into the "status" category for the purposes of database schema, but it's a separate grpc request from the other status data, and can fail at times when the other status request does not, so it has a separate function in the starlink_grpc module.
mypy complained about the signature of conditional def of get_args not matching the one from typing module, so I made it match, but apparently hadn't actually tested it with Python 3.7. It was just the return type hint, but it sill needs to parse at run time.
Also, fix up a few other (non-error) gripes from pylint.
Implement type hints for the data returned from status_data, history_stats, and history_bulk_data, and modify the *_field_names and *_field_types functions to make use of the same data.
This is complicated a little by the fact that the field names returned for sequences in the returned data are slightly different from those in *_field_names, for reasons that I'm sure made perfect sense to me at the time.
Should work with static type checkers on Python 3.8 and later. Python 3.7 is still supported for the run time, but possibly in a way that static type checking will not be able to understand.
This is for issue #57
Reword the is_snr_above_noise_floor description to be more consistent with the rest of the field docs.
Correct the order the status field groups in the docs for the functions that return them to reflect the actual order in which they are returned.
This adds alert_install_pending and alert_is_heating to the documentation at the top of the module. These alerts were added approximately forever ago, and get picked up automatically via reflection any time the scripts restart, so this is just to get the docs caught up.
Reintroduce validity check for prolonged obstruction duration and interval, but based on checking for 0.0 and NaN values instead of relying on the valid flag in the gRPC message. Empirical evidence suggests that the dish reports (0, NaN) for those 2 values when it has no obstruction data with which to work, such as when there actually are no obstructions or when the dish has recently rebooted.
This should fix#42.
Remove the validity check that was gating the reporting of the obstruction_duration and obstruction_interval data items.
When that got added to the grpc protocol, I assumed that the valid flag meant whether the other 2 fields were useful data. That may actually have been the intent, for all I know, but I've never seen that flag report true. However, I also never saw the banner in the app telling me I would get obstructions every X minutes, either. That is, until a few weeks ago, probably due to a dish firmware change. Now I see the app reporting this data, so it must be ignoring that valid flag, or at least not using it for that purpose.
Probably nobody cares about this data (including me), given that nobody ever complained about it always reporting empty values, but I don't want to remove it and mess up db schemas, so it might as well be populated.
Add timeouts to all gRPC remote calls and bump yagrc package requirement to a version that does same for the reflection service, as well as fixing a state issue around failed lazy import resolution.
This should address the script hang symptom on issue #36.
last_24h_obstructed_s has been removed from the grpc service protocol, along with all the other deprecated fields. This renders the seconds_obstructed item in the status mode group useless.
This is the down side of using reflection at run time to pull the protocol definitions. If something gets removed from the protocol that is still being used, it will break the run time instead of just returning default values. Then again, if it gets removed from the protocol, it's no longer useful, anyway.
This should address issue #35.
Derive connectivity state information from the "outage" field of the get_status response, which I hadn't noticed before because it only populates when the dish is not in a connected state. This restores the state data item in the status mode group, which had been rendered useless due to a grpc service change.
In addition to the previous possible state names, this adds a few more that pertain to outages while otherwise connected, which I think were just previously reported as "CONNECTED", as well as some special cases of offline.
A recent firmware change has stopped populating a number of result message fields in the grpc service that had previously been marked as deprecated. This caused the script to start crashing in some use cases.
While those fields are still in the protocol definition for now, this change removes usage of them entirely, in case they get removed. As things are now, they are useless, anyway, since they will always just return default values.
This renders useless the state, snr, scheduled, obstructed, and unscheduled items in the status, bulk_history, and ping_drop mode groups. Those items now mostly return empty values.
See issue #32 for more detail.
The prior 2 changes made the handling of the first set of polled loops inconsistent with subsequent sets with respect to maintaining data across dish reboot. This change makes them both work the same (and correctly).
This restores the ability of -o to keep polled data across a reboot. It broke due to a simple issue in concatenate_history, but I realized that counter tracking for -o accumulated data was also a bit broken, so I fixed that, too.
Fix#31
Add a new command line script, dish_obstruction_map.py, that writes a PNG image based on the obstruction map data queried from the dish.
Supports color or greyscale output and either with or without alpha channel.
Does not yet support running in an interval loop, mostly because that will require templatizing the output filename in order to be useful.
Tracked on issue #27
Change the loop polling function (-o) to aggregate the history data each polling loop instead of just keeping the last polled history so it can be logged when reboot is detected. This allows for computing statistics across a longer period than the size of the dish's history buffer, which has been reduced to 15 minutes recently.
This change also makes it so data is not logged right away when dish reboot is detected, so the logging always happens at the specified interval whether there was a reboot or not.
Finally, change the poll loop counting so data is not emitted on the first loop when polling is configured. That made sense to do when the history buffer was large enough to have the entire period's worth of data, but now it just results in a short period in the log output every time the script is restarted.
Fixes#29
Add dish direction and "prolonged" obstruction info to the status mode group.
These were added to the grpc service at some point over the last several months.
Only lightly tested, given that my dish no longer reports significant periods of obstruction.
This is related to discussion in issue #27, although it doesn't address that issue in the slightest.
statistics.quantiles was not present in Python 3.7 or earlier, which is a problem on Windows if you want to run a binary optimized version of the protobuf package, since those are not currently being posted for Python 3.8 or later.
This change switches to use the weighted median function just with equal weights. It's a bit of overkill, but it also cuts out the mess that was working around deficiencies of the statistics.quantiles implementation.
SpaceX has been using inconsistent field ordering when adding alerts, so field index cannot be used to consistently identify the specific alerts. Message number is more appropriate for that, anyway, but is not guaranteed to be a low enough number to fit into a bit field. Oh well, in the unlikely event that SpaceX switches to larger message numbers, they just won't show up in the alerts bit field (but will still show up in alert_detail).
This does make the bit ordering in alerts inconsistent with prior versions of these tools, but I've never actually seen one of these alerts report true, so hopefully this doesn't impact anyone.
The alerts are still sorted by index number in the alert_detail text output, which is a problem for CSV output, but I think ordering by message number instead would be pointlessly complex. alert_detail is not a great fit for CSV output anyway, due to its variable length, so just added a warning about that in the text script module doc.
A few things I noticed while porting this code to the JSON script. The only real change here is fixing the bulk history output to print UTC time instead of local time.
This further complicates the code, for functionality that probably only I care about, but when computing stats for relatively long time intervals, it really hurts when the dish reboots and up to an entire time period's worth of data is lost at exactly the point where it may have been having interesting behavior.
Since the alert types are determined dynamically from the protocol definition, the status schema may need to be updated even if nothing changed in the scripts, when the dish software adds a new alert type (which just happened, say hello to the "mast_not_near_vertical" alert). This allows the manual override for that case, not just schema version downgrade.
Probably not terribly useful unless someone needs to tunnel through a different network to get to their dish, but it makes testing the dish unreachable case a lot easier. This was complicated a bit by the fact that a channel (and therefor the dish IP and port) is needed to get the list of alert types via reflection due to prior changes.
This exposed some issues with the error message for dish unreachable, so fixed those.
I had only added the one that the Starlink app uses to show obstructions, because it didn't seem like the other one was all that useful, but people seem to be interested in studying the difference between the 2, so might as well have it. This is in the obstruction_detail group, along with the other one. I'm kinda regretting naming the first one as I did, though, because it's now a little confusing between my naming and the naming in the grpc message.
Since this is a new field, also had to implement schema updates for the sqlite script.
