On August 15, 2023, 3D printer maker Bambu Lab experienced a service outage caused by what it’s described as "abnormal network traffic" that caused some customers' cloud-connected printers to start creating plastic objects without user intervention.

Following the phantom printing, some customers complained on social media that their printers, which cost between $599 and $1,199 (£475 to £950), had been damaged. In some cases, printers tried to build objects on printer plates that were already occupied.

In an incident review posted last week, the manufacturer - based in China with a US division in Austin, Texas - blamed the snafu on the MQTT SDK that relays network messages between the printers and its cloud services.

"One of the two MQTT SDK clients was disconnected due to a timeout," the firm explained. "Usually, the service will automatically restore the connection, but in this case, the service returned a successful connection report when, in reality, the connection was not re-established. During this time, a number of print start messages accumulated."

There was another connection failure, after which Bambu's cloud service managed to re-establish communication - but in doing so, it dumped the backed-up print start messages to "a number of printers" and began printing if the printer was idle.

Further complicating matters, an API couldn't handle the large number of access requests that occurred during the cloud service interruption. The Bambu Studio app, which handles print tasks, is programmed to retry requests after accessing the cloud. Those requests further burdened the API service and exceeded its capacity.

To prevent further print queue dumps, Bambu Lab has already revised its service logic to increase database connection sizes - a change that should improve throughput. The outfit also plans to make changes to its printer firmware and cloud printing logic.

Firmware for the company’s X1 Series printers, which come equipped with LIDAR sensors, will gain functionality that checks to see whether models have been removed from the printing plate. This should stop the machines creating objects on printing plates that are occupied.

Bambu’s P1 Series, which lacks LIDAR, will be updated to prompt users to clean the plate before starting a print job.

These features will be enabled by default but can be disabled. That's similar to the networking setup in the Bambu Studio application - it defaults to cloud connectivity but can be switched for LAN-based operation.

Bambu’s cloud printing logic is also slated to see further refinement. When printing is initiated, the printer will check the timestamp and will toss print jobs that fall outside a specified time window. Hopefully this will avoid massive backlogs building up.

In a note published during its incident investigation, Bambu said it plans to improve LAN Mode for its printers. That change will allow the devices to be used during a future cloud outages. The note also points out that printing from SD cards is already available as an entirely offline option.

Customers had already requested the ability to print without connecting to the manufacturer's cloud service before this event.

Turning up the heat

The 3D printer biz also plans to monitor the temperature of printer components like the "hotend" - which melts the filament so it can be extruded.

"If a fault is detected, an error message will be prompted on the printer screen, Bambu Studio, and Bambu Handy, to alert the user and avoid any potential hazards," the org announced. "Of course, the heaters will be turned off to further minimize any potential risk."

In January, Bambu insisted that the "hotend" in its printers "can only reach a maximum temperature of around 400°C [752°F]," a temperature it assured is too low to start a fire.

Bambu Lab says that customers who experienced hardware failure as a result of the outage will get help with printer repair - in terms of replacement parts and spools of filament.

Additionally, two spools of randomly selected PLA filament (which retails around $25-$50) will be provided "as compensation for the trouble this has caused."

Bambu Lab promises that the firmware changes mentioned above are coming soon. Indeed, some were planned before this SNAFU. Owners should not wait up late in the hope of imminent arrival: Bambu has advised that "to ensure the quality and reliability of software updates and to also be able to bring you other new features we are working on, we will need some time to do all the changes."

The company has not enumerated "some time" and at the time of writing had not responded to a request for comment. ®

https://www.theregister.com//2023/08/24/bambu_lab_3d_printer_crazy/