Starlink terminal data acquisition for network engineers
I thought about posting this to only NANOG, but since a great concentration of beta testers of a technical/network engineering inclination are located in the Pacific NW, decided to also include the SIX chat list. You may have seen the Starlink android or ios consumer-friendly app, which displays network traffic, uptime/downtime, and other link stats. I believe this to be polled directly from the antenna unit itself over grpc. The beta antennas are always 192.168.100.1. If you are using your own router with the starlink beta system, in addition to its WAN interface being an ordinary DHCP client in cgnat IP space, you'll need to manually give it an address in that /24 and set up routing to reach the .1 IP as needed. reference: https://github.com/sparky8512/starlink-grpc-tools reference: https://github.com/fullstorydev/grpcui you'll need a fairly normal Linux or BSD box with: git go python3 pip use pip to install grpcio and grpcio-tools install grpcurl: https://github.com/fullstorydev/grpcurl do a git clone of the starlink-grpc-tools url above, also take a look at its readme info get the dish's protoset file and write it to new file dish.protoset , this is an index of all data that can be polled cd /home/eric/starlink-grpc-tools /home/eric/go/bin/grpcurl -plaintext \ -protoset-out dish.protoset \ 192.168.100.1:9200 \ describe SpaceX.API.Device.Device /home/eric/go/bin/grpcurl \ -plaintext \ -d {\"get_history\":{}} \ 192.168.100.1:9200 \ SpaceX.API.Device.Device/Handle | python parseJsonHistory.py output of the above looks like this: 2021-02-06T08:15:56,3600,19.92034332,14,2,0.3125,0,0,0.0,0 full CSV header for the above: datetimestamp_utc,samples,total_ping_drop,count_full_ping_drop,count_obstructed,total_obstructed_ping_drop,count_full_obstructed_ping_drop,count_unscheduled,total_unscheduled_ping_drop,count_full_unscheduled_ping_drop since we are able to acquire the above in a comma-delimited csv format, it's fairly easy to write a script storing the integers from any one of those particular columns into a mariadb db, sqlite, influxdb, or whatever. the following will output about 3.8MB of text for the full history (I believe this to be the full copy of the ring buffer stored in RAM for the terminal's statistics) , pipe it into a text file if you want to manually look at it. /home/eric/go/bin/grpcurl -plaintext -d {\"get_history\":{}} 192.168.100.1:9200 SpaceX.API.Device.Device/Handle same as the above but human readable output /home/eric/go/bin/grpcurl \ -plaintext \ -d {\"get_history\":{}} \ 192.168.100.1:9200 \ SpaceX.API.Device.Device/Handle | python parseJsonHistory.py -v current counter: 299673 All samples: 43200 Valid samples: 43200 Parsed samples: 3600 Total ping drop: 20.03700998 Count of drop == 1: 14 Obstructed: 2 Obstructed ping drop: 0.3125 Obstructed drop == 1: 0 Unscheduled: 0 Unscheduled ping drop: 0.0 Unscheduled drop == 1: 0 see the get_history_notes.txt file for more info SOME EXAMPLE QUERIES these should match with what the json query is in the grpc GUI # get status /home/eric/go/bin/grpcurl \ -plaintext \ -d {\"getStatus\":{}} \ 192.168.100.1:9200 \ SpaceX.API.Device.Device/Handle # get device info /home/eric/go/bin/grpcurl \ -plaintext \ -d {\"getDeviceInfo\":{}} \ 192.168.100.1:9200 \ SpaceX.API.Device.Device/Handle # get history, this outputs a huge amount of data /home/eric/go/bin/grpcurl \ -plaintext \ -d {\"getHistory\":{}} \ 192.168.100.1:9200 \ SpaceX.API.Device.Device/Handle The following is copied/pasted from my notes file on things we can acquire, and then use a tiny shell script with awk, sed, regex, whatever as needed to trim out just the numbers, and put them somewhere else for polling by snmp extend /home/eric/go/bin/grpcurl \ -plaintext \ -d {\"getStatus\":{}} \ 192.168.100.1:9200 \ SpaceX.API.Device.Device/Handle notes on what's what: figures we care about to parse out and turn into just the integers snr can never be higher than 9 fractionobstructed appears to be the percentage of the time that the view is obstructed, as long as the view remains unobstructed, this number appears to slowly decrement over time validS is valid seconds? i think the S is almost likely always Seconds of time "uptimeS": "304439" "snr": 9, "fractionObstructed": 0.0013524292, "validS": 61815.74, "last24hObstructedS": 53 "downlinkThroughputBps": 47915.73, "uplinkThroughputBps": 34980.496, "popPingLatencyMs": 29.266666 example of running the command above twice, the second time a few minutes after the first, to see that fractionObstructed does decrement itself over time first run: 0.0013524292 second run: 0.0013467998
Thanks Eric, this is great information. I just installed my Starlink "dishymcdishface" this weekend and it's humming along nicely, even in the -25F (-51F with windchill) temps here in northern MN! I've got a bunch of general monitoring of the link using a few Docker + grafana widgets [1], but hadn't looked into direct metrics gathering from the terminal yet. This will surely get me on that path :) --Tim [1] https://hub.docker.com/r/qlustor/speedtest_ookla-to-influxdb Tim Nelson Network Engineer / Cloud Services Phone: (218)727-4332 x4501 / Fax: (866)716-0229 5019 Airport Road, Hermantown, MN 55811 Check us out at www.sangoma.com On Sat, Feb 6, 2021 at 4:06 AM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
I thought about posting this to only NANOG, but since a great concentration of beta testers of a technical/network engineering inclination are located in the Pacific NW, decided to also include the SIX chat list.
You may have seen the Starlink android or ios consumer-friendly app, which displays network traffic, uptime/downtime, and other link stats. I believe this to be polled directly from the antenna unit itself over grpc.
The beta antennas are always 192.168.100.1. If you are using your own router with the starlink beta system, in addition to its WAN interface being an ordinary DHCP client in cgnat IP space, you'll need to manually give it an address in that /24 and set up routing to reach the .1 IP as needed.
reference: https://github.com/sparky8512/starlink-grpc-tools
reference: https://github.com/fullstorydev/grpcui
you'll need a fairly normal Linux or BSD box with:
git go python3 pip
use pip to install grpcio and grpcio-tools
install grpcurl: https://github.com/fullstorydev/grpcurl
do a git clone of the starlink-grpc-tools url above, also take a look at its readme info
get the dish's protoset file and write it to new file dish.protoset , this is an index of all data that can be polled cd /home/eric/starlink-grpc-tools /home/eric/go/bin/grpcurl -plaintext \ -protoset-out dish.protoset \ 192.168.100.1:9200 \ describe SpaceX.API.Device.Device
/home/eric/go/bin/grpcurl \ -plaintext \ -d {\"get_history\":{}} \ 192.168.100.1:9200 \ SpaceX.API.Device.Device/Handle | python parseJsonHistory.py
output of the above looks like this: 2021-02-06T08:15:56,3600,19.92034332,14,2,0.3125,0,0,0.0,0
full CSV header for the above: datetimestamp_utc,samples,total_ping_drop,count_full_ping_drop,count_obstructed,total_obstructed_ping_drop,count_full_obstructed_ping_drop,count_unscheduled,total_unscheduled_ping_drop,count_full_unscheduled_ping_drop
since we are able to acquire the above in a comma-delimited csv format, it's fairly easy to write a script storing the integers from any one of those particular columns into a mariadb db, sqlite, influxdb, or whatever.
the following will output about 3.8MB of text for the full history (I believe this to be the full copy of the ring buffer stored in RAM for the terminal's statistics) , pipe it into a text file if you want to manually look at it.
/home/eric/go/bin/grpcurl -plaintext -d {\"get_history\":{}} 192.168.100.1:9200 SpaceX.API.Device.Device/Handle
same as the above but human readable output
/home/eric/go/bin/grpcurl \ -plaintext \ -d {\"get_history\":{}} \ 192.168.100.1:9200 \ SpaceX.API.Device.Device/Handle | python parseJsonHistory.py -v
current counter: 299673 All samples: 43200 Valid samples: 43200 Parsed samples: 3600 Total ping drop: 20.03700998 Count of drop == 1: 14 Obstructed: 2 Obstructed ping drop: 0.3125 Obstructed drop == 1: 0 Unscheduled: 0 Unscheduled ping drop: 0.0 Unscheduled drop == 1: 0
see the get_history_notes.txt file for more info
SOME EXAMPLE QUERIES these should match with what the json query is in the grpc GUI # get status /home/eric/go/bin/grpcurl \ -plaintext \ -d {\"getStatus\":{}} \ 192.168.100.1:9200 \ SpaceX.API.Device.Device/Handle
# get device info /home/eric/go/bin/grpcurl \ -plaintext \ -d {\"getDeviceInfo\":{}} \ 192.168.100.1:9200 \ SpaceX.API.Device.Device/Handle
# get history, this outputs a huge amount of data /home/eric/go/bin/grpcurl \ -plaintext \ -d {\"getHistory\":{}} \ 192.168.100.1:9200 \ SpaceX.API.Device.Device/Handle
The following is copied/pasted from my notes file on things we can acquire, and then use a tiny shell script with awk, sed, regex, whatever as needed to trim out just the numbers, and put them somewhere else for polling by snmp extend
/home/eric/go/bin/grpcurl \ -plaintext \ -d {\"getStatus\":{}} \ 192.168.100.1:9200 \ SpaceX.API.Device.Device/Handle
notes on what's what: figures we care about to parse out and turn into just the integers snr can never be higher than 9 fractionobstructed appears to be the percentage of the time that the view is obstructed, as long as the view remains unobstructed, this number appears to slowly decrement over time validS is valid seconds? i think the S is almost likely always Seconds of time
"uptimeS": "304439" "snr": 9, "fractionObstructed": 0.0013524292, "validS": 61815.74, "last24hObstructedS": 53 "downlinkThroughputBps": 47915.73, "uplinkThroughputBps": 34980.496, "popPingLatencyMs": 29.266666
example of running the command above twice, the second time a few minutes after the first, to see that fractionObstructed does decrement itself over time first run: 0.0013524292 second run: 0.0013467998
Anyone else noticing major MAJOR problems with various MS services? Geoff
There is chatter on outages list outages@outages.org <mailto:outages@outages.org> and some other forums so I think you are not alone.
On Mar 15, 2021, at 1:04 PM, nanog08@mulligan.org wrote:
Anyone else noticing major MAJOR problems with various MS services?
Geoff
Notice we received indicated that they were experiencing authentication issues across all MS services. LQM -----Original Message----- From: NANOG <nanog-bounces+quincy.marshall=reged.com@nanog.org> On Behalf Of nanog08@mulligan.org Sent: Monday, March 15, 2021 4:04 PM To: nanog@nanog.org Subject: Microsoft problems... Anyone else noticing major MAJOR problems with various MS services? Geoff --------------------------------------------------------------------------------------- This email has been scanned for email related threats and delivered safely by Mimecast. For more information please visit http://www.mimecast.com ---------------------------------------------------------------------------------------
Can you be a bit more specific regarding what you're seeing or not seeing? Are you reaching MS through IP transit/peer connections, or are you having issues reaching MS cloud services over ExpressRoute circuits? Thank you jms On Mon, Mar 15, 2021 at 4:04 PM <nanog08@mulligan.org> wrote:
Anyone else noticing major MAJOR problems with various MS services?
Geoff
https://status.azure.com/en-us/status SUMMARY OF IMPACT: Starting at approximately 19:15 UTC on 15 Mar 2021, a subset of customers may experience issues authenticating into Microsoft services, including Microsoft Teams, Office and/or Dynamics, Xbox Live, and the Azure Portal. CURRENT STATUS: Engineering teams have been engaged and are currently investigating. The next update will be provided in 60 minutes or as events warrant. This message was last updated at 19:58 UTC on 15 March 2021 On Monday, March 15, 2021 01:31 PM PDT, Justin Streiner <streinerj@gmail.com> wrote: Can you be a bit more specific regarding what you're seeing or not seeing? Are you reaching MS through IP transit/peer connections, or are you having issues reaching MS cloud services over ExpressRoute circuits? Thank youjms On Mon, Mar 15, 2021 at 4:04 PM <nanog08@mulligan.org> wrote:Anyone else noticing major MAJOR problems with various MS services? Geoff
I didn't troubleshoot at all (not my job), but yes, we are having all sorts of issues accessing O365/Teams/etc --Andrey On Mon, Mar 15, 2021 at 1:33 PM Justin Streiner <streinerj@gmail.com> wrote:
Can you be a bit more specific regarding what you're seeing or not seeing?
Are you reaching MS through IP transit/peer connections, or are you having issues reaching MS cloud services over ExpressRoute circuits?
Thank you jms
On Mon, Mar 15, 2021 at 4:04 PM <nanog08@mulligan.org> wrote:
Anyone else noticing major MAJOR problems with various MS services?
Geoff
WVD seems to be affected as well... tak tsk tsk. I guess this is part of Monday blues? :P On Tue, Mar 16, 2021, 4:39 AM Andrey Khomyakov, <khomyakov.andrey@gmail.com> wrote:
I didn't troubleshoot at all (not my job), but yes, we are having all sorts of issues accessing O365/Teams/etc
--Andrey
On Mon, Mar 15, 2021 at 1:33 PM Justin Streiner <streinerj@gmail.com> wrote:
Can you be a bit more specific regarding what you're seeing or not seeing?
Are you reaching MS through IP transit/peer connections, or are you having issues reaching MS cloud services over ExpressRoute circuits?
Thank you jms
On Mon, Mar 15, 2021 at 4:04 PM <nanog08@mulligan.org> wrote:
Anyone else noticing major MAJOR problems with various MS services?
Geoff
https://status.office365.com/ On Mon, Mar 15, 2021 at 4:49 PM Nathanael Cariaga <ncariaga@gmail.com> wrote:
WVD seems to be affected as well... tak tsk tsk. I guess this is part of Monday blues? :P
On Tue, Mar 16, 2021, 4:39 AM Andrey Khomyakov, < khomyakov.andrey@gmail.com> wrote:
I didn't troubleshoot at all (not my job), but yes, we are having all sorts of issues accessing O365/Teams/etc
--Andrey
On Mon, Mar 15, 2021 at 1:33 PM Justin Streiner <streinerj@gmail.com> wrote:
Can you be a bit more specific regarding what you're seeing or not seeing?
Are you reaching MS through IP transit/peer connections, or are you having issues reaching MS cloud services over ExpressRoute circuits?
Thank you jms
On Mon, Mar 15, 2021 at 4:04 PM <nanog08@mulligan.org> wrote:
Anyone else noticing major MAJOR problems with various MS services?
Geoff
Azure is having outages atm - https://twitter.com/MSFT365Status?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ct... Ray Orsini Chief Executive Officer OIT, LLC 305.967.6756 x1009 | 305.571.6272 ray@oit.co | www.oit.co oit.co/ray How are we doing? We'd love to hear your feedback. https://go.oit.co/review From: NANOG <nanog-bounces+ray=oit.co@nanog.org> On Behalf Of Justin Streiner Sent: Monday, March 15, 2021 4:32 PM To: nanog08@mulligan.org Cc: NANOG <nanog@nanog.org> Subject: EXTERNAL: Re: Microsoft problems... CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you are unsure, please forward this email to ip@oit.co<mailto:ip@oit.co> for review. Can you be a bit more specific regarding what you're seeing or not seeing? Are you reaching MS through IP transit/peer connections, or are you having issues reaching MS cloud services over ExpressRoute circuits? Thank you jms On Mon, Mar 15, 2021 at 4:04 PM <nanog08@mulligan.org<mailto:nanog08@mulligan.org>> wrote: Anyone else noticing major MAJOR problems with various MS services? Geoff
teams is failing, powerbi is failing, azure is failing. I don't use hotmail, so I don't know... On 3/15/21 2:38 PM, Ray Orsini wrote:
Azure is having outages atm - https://twitter.com/MSFT365Status?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ct... <https://twitter.com/MSFT365Status?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor>
OIT Website <https://www.oit.co/> Ray Orsini Chief Executive Officer OIT, LLC
*305.967.6756 x1009* <tel:305.967.6756%20x1009> | *305.571.6272* <fax:305.571.6272>
*ray@oit.co* <mailto:ray@oit.co> | https://www.oit.co <https://www.oit.co/> * www.oit.co* <https://www.oit.co/>
oit.co/ray
Facebook <https://go.oit.co/facebook>
LinkedIn <https://go.oit.co/linkedin>
Twitter <https://go.oit.co/twitter>
YouTube <https://go.oit.co/youtube>
How are we doing? We'd love to hear your feedback. *https://go.oit.co/review* <https://go.oit.co/review>
*From:* NANOG <nanog-bounces+ray=oit.co@nanog.org> *On Behalf Of *Justin Streiner *Sent:* Monday, March 15, 2021 4:32 PM *To:* nanog08@mulligan.org *Cc:* NANOG <nanog@nanog.org> *Subject:* EXTERNAL: Re: Microsoft problems...
CAUTION:This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you are unsure, please forward this email to ip@oit.co <mailto:ip@oit.co> for review.
Can you be a bit more specific regarding what you're seeing or not seeing?
Are you reaching MS through IP transit/peer connections, or are you having issues reaching MS cloud services over ExpressRoute circuits?
Thank you
jms
On Mon, Mar 15, 2021 at 4:04 PM <nanog08@mulligan.org <mailto:nanog08@mulligan.org>> wrote:
Anyone else noticing major MAJOR problems with various MS services?
Geoff
Good afternoon, I work retail at a tech store in Canada as my day job, and we've been experiencing issues today only with activating Microsoft product cards. Unsure if it's related to the other Microsoft issues, but it's a good possibility as no other product card activations for other vendors are affected. Another neighboring retailer is having the same issue as well, so it's definitely a possibility. Regards, Peter P. Sent from my Bell Samsung device over Canada’s largest network. ________________________________ From: NANOG <nanog-bounces+nanog=as212934.net@nanog.org> on behalf of nanog08@mulligan.org <nanog08@mulligan.org> Sent: Monday, March 15, 2021 5:03:43 PM To: Ray Orsini <ray@oit.co>; Justin Streiner <streinerj@gmail.com> Cc: NANOG <nanog@nanog.org> Subject: Re: EXTERNAL: Re: Microsoft problems... teams is failing, powerbi is failing, azure is failing. I don't use hotmail, so I don't know... On 3/15/21 2:38 PM, Ray Orsini wrote: Azure is having outages atm - https://twitter.com/MSFT365Status?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ct... [OIT Website]<https://www.oit.co/> Ray Orsini Chief Executive Officer OIT, LLC [cid:part4.7AEA18DC.CE47A909@mulligan.com] 305.967.6756 x1009<tel:305.967.6756%20x1009> | [cid:part6.7E876534.9BBD70C8@mulligan.com] 305.571.6272 [cid:part8.7BF54727.73AB839A@mulligan.com] ray@oit.co<mailto:ray@oit.co> | [https://www.oit.co] <https://www.oit.co/> www.oit.co<https://www.oit.co/> [cid:part13.67217C5B.09C4EE25@mulligan.com] oit.co/ray [Facebook]<https://go.oit.co/facebook> [LinkedIn]<https://go.oit.co/linkedin> [Twitter]<https://go.oit.co/twitter> [YouTube]<https://go.oit.co/youtube> How are we doing? We'd love to hear your feedback. https://go.oit.co/review From: NANOG <nanog-bounces+ray=oit.co@nanog.org><mailto:nanog-bounces+ray=oit.co@nanog.org> On Behalf Of Justin Streiner Sent: Monday, March 15, 2021 4:32 PM To: nanog08@mulligan.org<mailto:nanog08@mulligan.org> Cc: NANOG <nanog@nanog.org><mailto:nanog@nanog.org> Subject: EXTERNAL: Re: Microsoft problems... CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you are unsure, please forward this email to ip@oit.co<mailto:ip@oit.co> for review. Can you be a bit more specific regarding what you're seeing or not seeing? Are you reaching MS through IP transit/peer connections, or are you having issues reaching MS cloud services over ExpressRoute circuits? Thank you jms On Mon, Mar 15, 2021 at 4:04 PM <nanog08@mulligan.org<mailto:nanog08@mulligan.org>> wrote: Anyone else noticing major MAJOR problems with various MS services? Geoff
participants (12)
-
Andrey Khomyakov
-
AS212934 - NANOG
-
Brandon Svec
-
Colin Preston
-
Craig
-
Eric Kuhnke
-
Justin Streiner
-
Marshall, Quincy
-
nanog08@mulligan.org
-
Nathanael Cariaga
-
Ray Orsini
-
Tim Nelson