PTP/Syncronized Ethernet maturity
Hello everybody, We have deployed PTP in our mobile NW since late 2019 as a part of the 4G/5G, however we are seeing a lots of instabilities and interop issues, a lot of the issues have ended up with SW bugs in the OS, I have no specific question, however I got the impression that the technology/protocol is not yet mature, anybody here got his hands dirty with PTP? Thanks, MP
Hi Macho, How are you? Glad to hear this from you. We also have PTP deployed in our transport network as one of the synchronization methods for our mobile networks. And we have encountered a lot of problems as well, such as instabilities, path asymmetry, large TIE variation, poor clock recovery performance, etc. But we have not ended up with any ascertained root causes nor effective solutions to these problems, yet. I don't know if it is true that PTP / IEEE1588v2 is not mature and reliable enough to meet the minimum syntonization / synchronization requirements of mobile networks (e.g. LTE and 5G NR). Based on my recent observations, I think there may be many we can improve in regard to the timing-over-packet network architecture design and synchronization recovery performance on the slave clock equipment. An interesting and costly issue to study and figure out as soon as possible. Look forward to hearing further input from you and others. Thanks and best regards, Taichi On Mon, Sep 7, 2020 at 3:36 AM Macho Pellegrini via NANOG <nanog@nanog.org> wrote:
Hello everybody,
We have deployed PTP in our mobile NW since late 2019 as a part of the 4G/5G, however we are seeing a lots of instabilities and interop issues, a lot of the issues have ended up with SW bugs in the OS, I have no specific question, however I got the impression that the technology/protocol is not yet mature, anybody here got his hands dirty with PTP?
Thanks, MP
We have mobile NWs in both Asia and Europe and also experience a lot of issues with PTP, - almost with every vendor. The instabilities, SW-bugs etc. related to PTP seems to indicate that very little testing of this code has been done in production networks. In some deployments we have been able to produce a clock service by installing GNSS on the cell-site. However, in other countries there are regulatory directives that the phase sync must be PTP/Network based. Currently, the optical domain is causing us huge problems when we try to engineer a T-BC/PTP solution. This is due to the path asymmetry that exist in the WDM/fiber domain. In some networks we have a lot of DCF in the fiber path and the only way we can get visibility in the asymmetry on these fiber hops is to measure in both direction:( Also, running T-BC over WDM/OTN will simply not work as the phase error introduced more or less eats up the phase error budget for 4G/5G TDD-service. best regards, Geir
On 5 Sep 2020, at 00:17, Macho Pellegrini via NANOG <nanog@nanog.org> wrote:
Hello everybody,
We have deployed PTP in our mobile NW since late 2019 as a part of the 4G/5G, however we are seeing a lots of instabilities and interop issues, a lot of the issues have ended up with SW bugs in the OS, I have no specific question, however I got the impression that the technology/protocol is not yet mature, anybody here got his hands dirty with PTP?
Thanks, MP
Hi Geir, Can we say, from your production network experiences across Asia and Europe, that getting synchronization clock signal via GNSS receiver directly on each cell site is a much more reliable, stable, and simpler way than getting it by network-based PTP? Especially when there is WDM link used in between the BC and Slave Clock? Why does WDM link cause path asymmetry? I thought the optical fibers carry forward link and reverse link are almost equal in length (distance). Aren't they? What are your solutions to overcome the PTP synchronization instability problems in your TDD 4G/5G networks? Thanks and best regards, Taichi On Tue, Sep 8, 2020 at 11:09 PM geir egeland via NANOG <nanog@nanog.org> wrote:
We have mobile NWs in both Asia and Europe and also experience a lot of issues with PTP, - almost with every vendor. The instabilities, SW-bugs etc. related to PTP seems to indicate that very little testing of this code has been done in production networks. In some deployments we have been able to produce a clock service by installing GNSS on the cell-site. However, in other countries there are regulatory directives that the phase sync must be PTP/Network based.
Currently, the optical domain is causing us huge problems when we try to engineer a T-BC/PTP solution. This is due to the path asymmetry that exist in the WDM/fiber domain. In some networks we have a lot of DCF in the fiber path and the only way we can get visibility in the asymmetry on these fiber hops is to measure in both direction:( Also, running T-BC over WDM/OTN will simply not work as the phase error introduced more or less eats up the phase error budget for 4G/5G TDD-service.
best regards, Geir
On 5 Sep 2020, at 00:17, Macho Pellegrini via NANOG <nanog@nanog.org> wrote:
Hello everybody,
We have deployed PTP in our mobile NW since late 2019 as a part of the 4G/5G, however we are seeing a lots of instabilities and interop issues, a lot of the issues have ended up with SW bugs in the OS, I have no specific question, however I got the impression that the technology/protocol is not yet mature, anybody here got his hands dirty with PTP?
Thanks, MP
Hi Taichi, It depends. GNSS at the cell site has its own operational challenges, for example making sure that the antenna has a clear enough view of the sky. A challenge in Asia is that very little of the fiber is in the ground, hence multiple fiber cuts happen on a daily basis which changes the path length when restoration mechanisms kicks in. A change in the path length is not a problem for PTP, but it require that we know the path asymmetry on all possible fiber paths between the master and the slave (we need to use protection on layer 1 in Asia due to the frequent fiber cuts). Another challenge with GNSS is that we experience that the GNSS is either jammed or even worse, that the GNSS is spoofed. When we did the PTP design, we also believed that the length of the fiber path length would be equal in both direction. However, in some of our old Metro networks, the line amplifier have embedded Dispersion Compensating Fiber (DCF) to compensate for the chromatic dispersion of different wavelengths. The length of fiber within DCF modules to compensate for the same length of fiber may vary significantly. Other parts in the optical domain can also cause asymmetry, e.g.transponders, software FEC, or FEC in general, and digital signal processors in coherent optical systems. Asymmetry increases with link speed, so we could consider running PTP over 1GE interfaces, but this is a challenge in our Core networks. We can overcome the DCF (DCF is cheap) issue by either measure the asymmetry of every fiber hop (not practical possible), change the DCP modules to Bragg filters (expensive), or deploy grand masters in the access/aggregation network in order to have less asymmetry impact from the fiber network. When it comes to the instabilities with PTP implementation, we try to work with the vendors so that they fix failing line cards, port flapping etc. However, its not always easy to get the vendor’s attention on PTP issues. OAM for PTP is another challenge, i.e. how can we make sure that the clock is healthy? We plan to solve this by deploying GNSS at certain locations in the network and use equipment that can compare the difference between GNSS-input and the received PTP clock. So key message is that PTP does not work out of the box, it requires significant engineering effort. GNSS has many issues as well, and in certain parts of Europe we cannot rely on GNSS only. So it is not an either-or, - we need both. best regards, Geir
On 8 Sep 2020, at 18:11, m.Taichi <marc101.maxmaok@gmail.com> wrote:
Hi Geir,
Can we say, from your production network experiences across Asia and Europe, that getting synchronization clock signal via GNSS receiver directly on each cell site is a much more reliable, stable, and simpler way than getting it by network-based PTP? Especially when there is WDM link used in between the BC and Slave Clock?
Why does WDM link cause path asymmetry? I thought the optical fibers carry forward link and reverse link are almost equal in length (distance). Aren't they?
What are your solutions to overcome the PTP synchronization instability problems in your TDD 4G/5G networks?
Thanks and best regards, Taichi
On Tue, Sep 8, 2020 at 11:09 PM geir egeland via NANOG <nanog@nanog.org <mailto:nanog@nanog.org>> wrote: We have mobile NWs in both Asia and Europe and also experience a lot of issues with PTP, - almost with every vendor. The instabilities, SW-bugs etc. related to PTP seems to indicate that very little testing of this code has been done in production networks. In some deployments we have been able to produce a clock service by installing GNSS on the cell-site. However, in other countries there are regulatory directives that the phase sync must be PTP/Network based.
Currently, the optical domain is causing us huge problems when we try to engineer a T-BC/PTP solution. This is due to the path asymmetry that exist in the WDM/fiber domain. In some networks we have a lot of DCF in the fiber path and the only way we can get visibility in the asymmetry on these fiber hops is to measure in both direction:( Also, running T-BC over WDM/OTN will simply not work as the phase error introduced more or less eats up the phase error budget for 4G/5G TDD-service.
best regards, Geir
On 5 Sep 2020, at 00:17, Macho Pellegrini via NANOG <nanog@nanog.org <mailto:nanog@nanog.org>> wrote:
Hello everybody,
We have deployed PTP in our mobile NW since late 2019 as a part of the 4G/5G, however we are seeing a lots of instabilities and interop issues, a lot of the issues have ended up with SW bugs in the OS, I have no specific question, however I got the impression that the technology/protocol is not yet mature, anybody here got his hands dirty with PTP?
Thanks, MP
Hi Geir, Gratefully thanks for your detailed sharing. Very informative and helpful to our network's synchronization planning and operation. We'll take your experiences into discussion and consideration. Get back to share our experiences with you soon. Thanks and best regards, Taichi On Wed, Sep 9, 2020 at 4:50 PM geir egeland <geir.egeland.ietf@gmail.com> wrote:
Hi Taichi, It depends. GNSS at the cell site has its own operational challenges, for example making sure that the antenna has a clear enough view of the sky. A challenge in Asia is that very little of the fiber is in the ground, hence multiple fiber cuts happen on a daily basis which changes the path length when restoration mechanisms kicks in. A change in the path length is not a problem for PTP, but it require that we know the path asymmetry on all possible fiber paths between the master and the slave (we need to use protection on layer 1 in Asia due to the frequent fiber cuts). Another challenge with GNSS is that we experience that the GNSS is either jammed or even worse, that the GNSS is spoofed.
When we did the PTP design, we also believed that the length of the fiber path length would be equal in both direction. However, in some of our old Metro networks, the line amplifier have embedded Dispersion Compensating Fiber (DCF) to compensate for the chromatic dispersion of different wavelengths. The length of fiber within DCF modules to compensate for the same length of fiber may vary significantly. Other parts in the optical domain can also cause asymmetry, e.g.transponders, software FEC, or FEC in general, and digital signal processors in coherent optical systems. Asymmetry increases with link speed, so we could consider running PTP over 1GE interfaces, but this is a challenge in our Core networks.
We can overcome the DCF (DCF is cheap) issue by either measure the asymmetry of every fiber hop (not practical possible), change the DCP modules to Bragg filters (expensive), or deploy grand masters in the access/aggregation network in order to have less asymmetry impact from the fiber network.
When it comes to the instabilities with PTP implementation, we try to work with the vendors so that they fix failing line cards, port flapping etc. However, its not always easy to get the vendor’s attention on PTP issues.
OAM for PTP is another challenge, i.e. how can we make sure that the clock is healthy? We plan to solve this by deploying GNSS at certain locations in the network and use equipment that can compare the difference between GNSS-input and the received PTP clock.
So key message is that PTP does not work out of the box, it requires significant engineering effort. GNSS has many issues as well, and in certain parts of Europe we cannot rely on GNSS only. So it is not an either-or, - we need both.
best regards, Geir
On 8 Sep 2020, at 18:11, m.Taichi <marc101.maxmaok@gmail.com> wrote:
Hi Geir,
Can we say, from your production network experiences across Asia and Europe, that getting synchronization clock signal via GNSS receiver directly on each cell site is a much more reliable, stable, and simpler way than getting it by network-based PTP? Especially when there is WDM link used in between the BC and Slave Clock?
Why does WDM link cause path asymmetry? I thought the optical fibers carry forward link and reverse link are almost equal in length (distance). Aren't they?
What are your solutions to overcome the PTP synchronization instability problems in your TDD 4G/5G networks?
Thanks and best regards, Taichi
On Tue, Sep 8, 2020 at 11:09 PM geir egeland via NANOG <nanog@nanog.org> wrote:
We have mobile NWs in both Asia and Europe and also experience a lot of issues with PTP, - almost with every vendor. The instabilities, SW-bugs etc. related to PTP seems to indicate that very little testing of this code has been done in production networks. In some deployments we have been able to produce a clock service by installing GNSS on the cell-site. However, in other countries there are regulatory directives that the phase sync must be PTP/Network based.
Currently, the optical domain is causing us huge problems when we try to engineer a T-BC/PTP solution. This is due to the path asymmetry that exist in the WDM/fiber domain. In some networks we have a lot of DCF in the fiber path and the only way we can get visibility in the asymmetry on these fiber hops is to measure in both direction:( Also, running T-BC over WDM/OTN will simply not work as the phase error introduced more or less eats up the phase error budget for 4G/5G TDD-service.
best regards, Geir
On 5 Sep 2020, at 00:17, Macho Pellegrini via NANOG <nanog@nanog.org> wrote:
Hello everybody,
We have deployed PTP in our mobile NW since late 2019 as a part of the 4G/5G, however we are seeing a lots of instabilities and interop issues, a lot of the issues have ended up with SW bugs in the OS, I have no specific question, however I got the impression that the technology/protocol is not yet mature, anybody here got his hands dirty with PTP?
Thanks, MP
participants (3)
-
geir egeland -
m.Taichi -
Macho Pellegrini