Active Incident

Updated 19 hours ago

Store reliability is important to us at Neto. We monitor the health of Neto both internally and through 3rd party services. If there is a disruption in service to your store, or any of the Neto operated services, our engineers will update the notes here.

Incident Status

Operational

Components

Batch Processing



June 22, 2017 2:14PM ChST
[Identified] Neto engineers have identified an issue with our Batch Processes affecting some merchants. Both scheduled and manual batch processes are currently affected. Our engineers are working to return regular operation to this service. As soon as we have a resolution or additional details become available we will post an update to this incident notification. Neto will post updates at status.neto.com.au, please check this page for further updates.

June 22, 2017 3:45PM ChST
[Identified] The cause of the issue with AWS batch processes has been identified. It is directly related to the infrastructure issues which have impacted store performance over the past 48 hours. As reported last night, a plan has been put in place to resolve these issues. Execution of the plan is progressing well and testing is due to start on the modified environment shortly. While this transition process continues over the coming hours, batch processes will continue to run slowly and some processes may fail. The batch system is quite robust in its design, so should a process not run correctly now, the next time it is scheduled to run it will pick up any left over work from this or any previous attempts to run. This means that once the platform is running smoothly again, you should find that batch catches up in most cases quite quickly. We again apologise for any inconvenience that has been caused by this incident.

June 23, 2017 7:54AM ChST
[Identified] Batch processes have been running over night however there will still be merchants with a backlog of tasks and processes may take longer than usual to run. The transition to the alternative infrastructure is progressing smoothly and as the day progresses we expect operations will return to normal.

June 24, 2017 10:05AM ChST
[Monitoring] Neto engineers have been monitoring Batch for the last 12 hours and the service is back to normal operation. Neto will continue to monitor the Batch system over the next 24 hours to make sure things continue to operate as normal.

Incident Status

Operational

Components

Control Panel, Merchant Store Fronts



June 21, 2017 2:01PM ChST
[Investigating] Neto engineers have seen a increase in the response time on some sites. The issue is being investigated, an update will be posted soon as to the progress of our team.

June 21, 2017 4:41PM ChST
[Investigating] Neto engineers are continuing to investigate the issue but site speeds have improved. Our team is continuing to work to get service back to normal operating state. Our team will send further updates when more information comes to hand.

June 21, 2017 6:04PM ChST
[Monitoring] Neto engineers will be monitoring the situation for the next 12 hours, sites are currently back to normal operating levels.

June 21, 2017 6:53PM ChST
[Monitoring] The cause of the issue has now been identified with certainty. It is related to the migration of our platform to our new hosting infrastructure. This migration has taken place over the past week. In the long term, the move to our new infrastructure will be beneficial for everyone, but it has not come without its teething issues. This is not an excuse, but rather part of our commitment to be as transparent as possible. In summary, the issue is with the file server cluster which is using a new technology. Although during our extensive testing this technology was performant, under production load it is not proving to support your needs. To resolve the issue, the decision has been made to "roll back" that part of the infrastructure to what was in place prior, which is stable and has been in use for several years now. This should resolve the performance issue you have been experiencing for the past two days. The "roll back" process is now underway and will continue through tonight and tomorrow. We sincerely apologise for the inconvenience and disruption this incident has caused.

Upstream Provider

Operational

Merchant Store Fronts

Operational

Control Panel

Operational

Point Of Sale Registers

Operational

Connect Modules

Operational

Payment Methods

Operational

Carrier Labelling Integrations

Operational

Batch Processing

Operational

NetoMail

Operational

Channel Modules

Operational

Pick 'n' Pack

Operational

Help & Support Knowledge Base

Operational

Support Call Centre

Operational

Scheduled Maintenance

Schedule

June 1, 2017 12:00AM - 12:30AM ChST

Components

Batch Processing, Carrier Labelling Integrations, Channel Modules, Connect Modules, Control Panel, Help & Support Knowledge Base, Merchant Store Fronts, NetoMail, Payment Methods, Pick 'n' Pack, Point Of Sale Registers, Upstream Provider

Description

Neto Engineering have uncovered an issue that requires urgent attention. In order to completely remedy the situation, a short maintenance window is being scheduled for midnight tonight and will last for 30 minutes. During this time, some Neto websites will be temporarily unavailable as our engineers perform the necessary work. We apologize in advance for the short notice of this maintenance.