Moodle LMS optimisation – the big reveal

Here at Catalyst, we often discuss the ongoing and incremental improvements we make to existing hosting and cloud services.  However, we rarely have the opportunity to apply them all at once, with ‘The Big Reveal’ that compares the before and after performance.  This post shares our experience of delivering a Moodle LMS optimisation project, looking at the before and after, and the dramatic improvements.

LMS health check –  the before 

In late 2020, a new client came to us with existing Moodle workloads running in Amazon Web Services (AWS). As is often the case, the Moodle platforms experienced LMS atrophy, resulting in performance delivery way below expectations and potential. 

Findings

Catalyst’s technical team carried out an initial assessment of the learning platforms.  Major findings included:  

•  Two Moodle sites in urgent need of security patching and upgrades, it was unclear if these sites had ever been patched;
•   Site One was a single production instance, with no test infrastructure, and no High Availability (HA);
•  Site Two was slightly better, it had a test and production site. HA was partially provided, in the test environment only;
•   Infrastructure was named incorrectly, with production sites connected to databases named with test names;
•   Infrastructure was sized poorly, with the database being provisioned far larger than required;
•  No auto-scaling;
•  No encryption, anywhere, and no SSL; 
•  A surprise WordPress CMS installation, that was unpatched;
•  S3 Buckets that were world-readable;
•  S3 Bucket credentials that were comprised, by the WordPress CMS instance;
• Security groups that were all over the place, allowing, for example, global SSH access;
• No evidence of any AWS cost optimisation, e.g. no reserved instances etc.

The list went on, essentially a representation of “What not to do when web hosting”. Whilst this may sound surprising, it is not an uncommon situation that many organisations find themselves in … embarking on an initiative, with good intentions and a “How hard can it be?” attitude. Delivering optimal cloud performance and cost management is a finely tuned skill, that must be practised to remain current.

Normally, Catalyst would set out on a project to migrate Moodle sites across to our own hosting clusters.  These clusters are already HA, high performance, are well maintained and have good cost management disciplines in place. However, in this case the client elected to maintain ownership of their own AWS account with Moodle, hosted on a dedicated platform.

The Moodle optimisation project plan

There were a number of steps involved in the project plan. 

Moodle hosting infrastructure

The first priority was to establish a new set of Moodle hosting infrastructure, following Catalyst best practices:

•     Terraform deployed for rapid and consistent stand-up of the infrastructure;
•     HA site wide for web servers, data storage, session management, and databases;
•     Staging, User Acceptance Testing (UAT) and production infrastructure.

Site upgrades to the latest version of Moodle

By updating to the latest version of Moodle, the client has the benefit of ensuring they have the latest security patches, can access new features and any improvements in the User Experience (UX). 

Setup full CI/CD deployments

CI/CD deployments enable our clients to access Catalyst’s optimised pipelines that continuously integrate and continuously deploy code changes.   

Business Analysis of the WordPress site 

We discovered  that the WordPress site was used for a single purpose, to enrol a select groups of users into Moodle. Catalyst developed a custom plugin, in Moodle, allowing the complete removal of the WordPress instance and the maintenance overhead associated with it.

Migration

The Infrastructure Team migrated the Moodle sites from the old infrastructure to the new Virtual Private Cloud (VPC) infrastructure.

Decommissioning of old infrastructure

Best practice and regulations, such as PCI DSS and HIPPA, dictate that IT assets should be decommissioned responsibly.  This includes tasks such as backup and storage of old data, plus the updating of your asset register. 

Cost optimisation

The team reviewed the scaling policies, rightsizing the infrastructure (the process of matching instance types and sizes to your workload performance and capacity requirements at the lowest possible cost) and purchasing Reserved Instances at a significant discount compared to on-demand pricing. 

Initial results – 6 months in

The project is now complete, giving us the opportunity to reflect on what has been achieved:  

•  Two Moodle sites were upgraded, with double-hop upgrades;
•  Two Moodle sites were migrated to a shiny new infrastructure;
•  A new custom Moodle theme was established;
•  Full unit tests now passing for every security release we do, and;
•  Establishment of fully automated release pipelines.

Cost management

We carried out a full cost management pass on the infrastructure to:

• Rightsize all the compute instances;
• Leverage cheaper AMD or ARM instances, where appropriate;
• Use AWS Aurora MySQL database and configure Moodle to split read and write traffic across the Primary Writer instance and the DR read instances;
• Enable horizontal scaling of both the web servers and the read replica databases;
• Intelligently route traffic to the read replica database and Redis cache instance in the same Availability Zone, to keep costs down;
• Leverage AWS spot instances, wherever we can;
• Configure life cycle policies on the S3 buckets, to leverage cheaper storage for less frequently accessed objects;
• Enable Brotli compression to reduce outbound traffic, as much as possible;
• Support the purchase of reserved instances and AWS Savings plans, to cover the consistent on-demand portion of the workload. 

The work has resulted in approximately 50% reduction in average daily costs, including the amortised upfront reservation payments for the AWS workload.   These costs now include HA capacity and environments, that didn’t exist prior to the project. 

Performance optimisation

Moodle can perform well when installed on a single machine with the database and site-data file system, both being local to PHP code. When moving to an HA setup, where the database is on a remote machine and the site-data is a network file system shared between multiple machines, quite a lot of work needs to be done to bring performance back to those single machine levels.  

For this particular project, Catalyst applied a number of actions to support performance optimisation of the LMS platforms:

• Optimised Moodle Unified Caching setup, utilising a combination APCu, Redis, local and shared file system caches;
• Stacked MUC cache setups, so in the event of scaling, cache items can be retrieved from a network cache instead of rebuilding them;
• Varnish Reverse Proxy caching allows us to offload approximately 40% of Moodle web-server requests, resulting in much faster page loads and a lot less load on PHP and the database;
•  Over-provisioning of web-servers, by leveraging spot instances for additional capacity;
• Routing cache and database traffic to the the cache or database instance in the same Availability Zone (data centre) – saves a few milliseconds of each round trip;
• Brotli Compression in our web-servers maximise compression of the resulting web traffic, meaning web browsers take less time to download the resulting pages;
• Upgrade from PHP 5.6 to 7.4. PHP is getting faster with each release, in this case page generation time dropped by about 25%.

Initially, we set out of with a goal of protecting the UX, ensuring that the page load speed was maintained, despite the introduction of the HA components. In reality, we reduced the average page render time by 40% overnight, recorded using Matomo Analytics:

Information Security

Information Security is baked into our standard infrastructure builds, as an integral part of our ISO 27001 certification. Most of this is entirely transparent to our Moodle users and probably worth it’s own series of blog posts. In summary, the infrastructure we have built now includes:

•   SSL only implementation for Moodle;
•   Encryption at rest, for all data including: Web server EBS, cache, file servers, databases and offsite backups;
•   Egress proxy servers, with filtered allow lists for traffic;
•   Automated patching of infrastructure;
•   Automated patching of Moodle code, with robust CI/CD pipelines;
•   Extensive logging at the application and infrastructure levels;
•   24/7 monitoring;
•   Reduced set of Identity and Access Management (IAM) permissions, following the Principles of Least Privilege;
•   Monitoring on all key rotations dates, and; 
•   Web application firewall integration

Apologies to our CISO for the things I may have left out of this list!

LMS health check – the after

Reflecting on the impact of this Moodle re-invigoration project, some significant optimisations  were achieved: 

•  Fully patched modern Moodle environment 
•  Cheaper hosting, a 50% cost saving
•  Faster page loads, time reduced by 40%
•  Full HA infrastructure, with no single point of failure
•  Excellent security posture, to withstand the modern ‘Wild West’ that is the Internet

There is no doubt that this project delivered amazing results for our client.  It has also been a fantastic project to showcase what can be done when the small incremental changes we make day-in-day-out are applied all at once.

Services to transform your Moodle LMS
 

If you’d like to explore how the Catalyst Team of LMS experts can help you achieve your LMS goals, we’d love to hear from you.