Blog | Coherent - Part 3

Make your documents searchable without giving up control

Ever year, I get a letter from my accountant. Our year end just finished and he wants a pile of paperwork from me to prepare our annual accounts.

I print it all off (sorry, tree), put it in a box, walk it over and dump it on the receptionist’s desk.

Someone their end then has to do some menial tasks like looking through our bank statements and make sure there’s an invoice for every payment in and out. I don’t envy that person!

They might scan it all in or make another paper copy of it for their records.

This sort of workflow is really common amongst accountants, solicitors, conveyancers, IFAs, estate agents etc.

It’s a huge waste of time because:

  1. A lot of the time, you are literally looking for a needle in a haystack
  2. You are following the same process for each customer

You might have heard of OCR. It’s where you scan the paperwork in and the text becomes searchable like on a web page or in a Word document. Maybe you’ve even been pitched to by a company offering a “document management” company. They want you to send them all of your files (??!) to do the recognition and for “safe”-keeping.

But what if there were a way to get the same benefits without ever giving up control of the documents? What if you didn’t have to pay a subscription or tie yourself in?

What if it were dead easy to find precisely what you’re looking for? (an invoice from the council on the 2nd of May for £120).

Enter, bespoke software. Neato!

Bespoke software is a great fit for document management because it gets around the biggest security concerns, eliminates vendor tie in and cuts your ongoing costs substantially.

We’re experienced software developers and we’ve done this before and we can do it again. Give us a call if you’d like to start taking control of your documents.

Read more →


Preparing your store for Black Friday and Christmas shopping

First and foremost, Christmas is not here yet. We have two more months before we have to sit down with our most annoying relatives. Phew.

But, if you’re in retail, you’ll know how important the next two months are. The next two months pay for the rest of the year. If they go well, you’ll be in great stead for 2018. If they go badly, you might get your P45 in your Christmas card.

If you have physical stores, you might be advertising temporary vacancies to help cope with the rush. Good thinking, Batman.

Your online store might not have the capacity it needs.

There are four factors that influence the capacity of your online store:

  1. The number of people who visit your website
  2. How they interact with your website
  3. Your online store’s coding
  4. The servers that host your online store

If you use Google Analytics, (1) and (2) are really easy to find so you should definitely do this now.

Log in and select your store and go to Audience > Overview.




Drop down under Overview. You’ll probably see Sessions selected. Choose Pageviews.

On the right hand side, choose Hourly.

Then, hover over the graph and you’ll see the number of pageviews per hour.

Select your busiest period. This might be your Black Friday or Christmas sale from last year. Find the busiest hour or the busiest day.

To make the maths easy, let’s say there were 36,000 pageviews in that hour. There are 3600 second in an hour, so divide that number by 3600 to get your pageviews per second. In our example, we arrived at 100.

We then need to adjust this number for your projections this year. If you’re spending more on advertising this year, you might want to add 20%. If you’re spending less, you might want to subtract 20%. It’s better to overshoot than to undershoot, though, and it’s worth adding a buffer in case your estimate is a bit out.

The number you arrived at is the number of pageviews per second that your store and hosting need to be able to accommodate so your Christmas card contains an awful joke and not a P45 (I’m not selling this well, am I?)

Armed with your pageviews per second, your developer and hosting company (or Coherent… just sayin’) will be able to run tests to determine if you’re already in good stead or if you need to make changes so your online store doesn’t fall flat when you most need it up and running.

Read more →


Give your customers the self-service portal they want

The photocopier is broken again. Ugh. At least their phone number’s on my recent calls list from yesterday…

.. Ring ring .. For sales, press 1 ..

.. This is really adding insult to injury ..

.. For accounts, press 2 ..

.. I need to remember the menu options for next time ..

.. For support, press 3 ..

.. At last! ..

.. We record all our calls ..

.. That’s great. So do we. …

.. All of our agents are busy .. doo doo .. doo be doo .. Your time is valuable to us ..

Sound familiar? Of course it does. But that doesn’t mean there isn’t a better way.

Self-service portals and apps are huge win-win opportunities for you and your clients. Your customers can get the information they need when they need it (even if they’re photocopying their face at 4am after a classic work night out). You don’t have to employ people, to help to one person at a time, who probably knows what they need and doesn’t want to have to phone in.

That person might even click a button that the customer could have clicked themselves. Efficient, huh?

This sort of workflow belongs in the ’90s. You just have to let go!

That photocopier problem could have been solved with an online troubleshooter. It could also have been solved with an in-app chat that would allow one staff member to help several customers at once (some companies take this to the extreme which is why they seem to have just woken up). In-app chat would also let the customer take photos of the problem. How much quicker would that be?

Whatever systems you have in place now might be great for what they do but if customers can’t help themselves then there is a tremendous opportunity to improve your customer experience and lower costs at the same time. It’s what we do as software developers. We create software that could become part of your website, or could be a mobile app, to plug into whatever systems you have now and deliver extra functionality.

Even if your current systems are decades old black boxes, even if you’ve never worked with software developers before, we know our stuff and we can make it easy for you to deliver the service your customers expect. Call us!

Read more →


How we use automation in our workflow

Bureaucracy isn’t just the domain of large corporates. Even as a young, hip software development company, we have it. In this post, I want to describe how we use automation and bespoke software in our own workflow.

Systems we use

We use several off the shelf systems:

  • Xero for accounting,
  • GitHub is a key part of our software development workflow,
  • Harvest to fill some of the gaps in Xero,
  • Dial9 for our phone system,
  • Matomo for website analytics,
  • Kayako (legacy) as a support desk / shared inbox.

Harvest integrates nicely with Xero (which is why we use it) but apart from that, there’s little integration between our systems.

Sometimes it doesn’t matter much. Integrating our phone system with our accounting system wouldn’t help much. We had a few challenges, though.


  1. “He said, she said”
    Most of our customers are an absolute pleasure to work with (thanks!) but every so often, we had to trawl through emails and phone call logs to find out if we’d made a mistake or if a customer was trying to pull a fast one. It was sometimes so time consuming that we would just assume that we made a mistake even if it meant the company made a loss on a project.By integrating with our phone system, we can quickly find out who spoke to whom, when, and what was said (i.e. the recording) on a timeline simply by clicking on a customer’s name. The timeline also has emails and meeting notes, and everything is searchable. A very detailed audit trail isn’t for everyone but for us, it occasionally saves a lot of time in “he said, she said” scenarios.
  2. Keeping data in sync
    Xero has a list of our customers, invoices and payments. Harvest has a similar list. Dial9 and Kayako have address books that, if they’re accurate, make it easy to know who’s who. By keeping our Dial9 address book up to date, we can see the caller’s name on all of our phones. Because the right person can pick up the phone, we rarely have to transfer calls.By integrating Harvest and Dial9, our phone system’s address book is always up to date. Every time we take on a new customer, their name pops up on our phones when they call without us having to fiddle with the phones or Dial9’s address book.
  3. A single pane of glass
    We’re a small company and mitigating the risks that small companies face (such as cash flow) and making sure that we’re doing the right things to grow our company (such as making sure our marketing budget is well spent) are hugely important. We track loads of KPIs from search engine ranks, to Adwords clicks, to the number of customers with whom we work in a given month, to our website’s traffic, to aged debtors, to projects that are likely overrun and so on.All of our KPIs are always up to date and visible on one page. Really. It wasn’t hard, either. All of our systems had the data we needed inside them but, because every company’s approach to reporting is unique, the data we needed in the format we wanted it in took some time to get to. Now it doesn’t.
  4. That feature we needed
    One of the big drawbacks of off the shelf software is that if there’s something you want it to do, that it doesn’t do (say, because most of their customers don’t need it), it can feel like you’re paying for something that doesn’t fit or that you’re tied in to it.For us, the feature was retainers. A lot of our work is on retainer – if customers commit to a certain volume of work, it gives us the stability we need to grow our company and in return, we can get things done more quickly and be more flexible. Harvest’s concept of retainers didn’t fit out own so we decided to layer bespoke software on top of Harvest to use Harvest’s invoicing, time tracking and project functionality but keep a separate log of retainers.
    5. Correlating data
    One of the greatest advantages of connecting systems is that you can see data from different systems on the same table or the same graph. We correlate our marketing spend with our website traffic and our monthly revenue which allows us to quickly see how well our marketing is performing.

Read more →


Why understanding the GDPR can’t wait

The General Data Protection Regulation (GDPR) is all over the news. Whilst the deadline of the 25th of May 2018 for UK businesses to be compliant might seem like a long way away, many businesses will have to make substantial changes before then to avoid hefty penalties. It is hard to overstate the differences between the already complex and important Data Protection Act (DPA) and the GDPR and the impact it will have on businesses and consumers.

Who does it apply to?

The GDPR will have an impact on every business and consumer with any EU presence. It “upgrades” the protections enshrined in law by the DPA for today’s digital world, thereby offering more protection to consumers and placing more responsibilities on businesses.

What about Brexit?

The Government has, with unusual (!) clarity, stated that the changes in the GDPR will continue to apply after Brexit.

Read more →


How bespoke software can improve your workflow

What is it?

Bespoke software is a very broad descriptor that encompasses a large range of products. Essentially, it is any custom built piece of programming featuring sets of instructions, which enables specific tasks to be carried out automatically.

For example, consider a used-car dealership. To keep up with today’s market, at the very least the dealership will need to have a well designed website listing all available vehicles and details. Now imagine a customer visiting this website. What will make them more likely to get in touch? Some applications we might include in our software are:

  • The ability to filter available vehicles according to the customers needs
  • An online booking system where customers choose a viewing time that fits your diary online.
  • A live chat option, where customers can speak directly to your team while browsing the site.

It is a fairly common and easy job for us to develop websites with this functionality. But there are many more advanced operations we can incorporate.

Market research and advertising

  • We can develop personalised systems that track things like how the customer reached your site, what pages they spent longest on etc. and produce statistics, graphs and diagrams analysing this information, so you know where to focus your advertising.
  • We can create a system that modifies your listed prices according to the length of time an item has been available and how frequently it is viewed.
  • When a customer books a viewing we can store their name, contact details, budget, vehicle preferences and payment method for future use.
  • We can automate advertising, so that emails advertising specific types of vehicles/services are sent to the relevant customers at pre-programmed times.

Billing and accounting

  • Whether you want to integrate a current system or develop one from scratch, we can make it so that the data you have collected when arranging viewings is automatically entered into your billing software.
  • You may also want to consider incorporating an accounting system, so that billing transactions are automatically added to your accounts when a purchase is made through the billing system.
  • We can customise accounting systems to synchronise regularly with business bank account statements.
  • When a sale is made we can develop software that automatically updates the inventory on your website, removing the sold item from your listings.

Server management and security

  • We are experts at managing servers, ensuring they are capable of running web applications reliably.
  • We can organise automated backups so that your website and data can be recovered in case of hardware failures or infection.

Essentially, what develop automated systems and integrate all these processes so you don’t have waste time re-entering data, or performing long-winded menial tasks like manually inputting every transaction into your accounts. You have all the information you need at the click of a button.

We believe that almost all businesses can save substantial labour time by automating processes like those mentioned. This saves you from spending money on additional administrative staff, and frees you up to focus on your areas of expertise and passion.

Read more →


Servers: buying hardware vs renting hardware vs cloud

As your online presence grows, you may start to consider the pros and cons of buying/renting hardware and using cloud services for your hosting. All of these options have pros and cons and this article will provide our view on the realities of this decision. We offer cloud servers and dedicated servers – find out more about hosting with us.

Buying Hardware

If you buy hardware, you can get exactly what you want. Choose the chassis, choose the motherboard and then start adding more components. If you need a storage server, buy a large chassis and add lots of disks. If you need a compute-intensive server, buy a small chassis and put a 4 socket motherboard in it. The choices are endless.

This is an excellent choice for those whose infrastructure spans a large number of servers. Why? Because buying 100 servers has economies of scale over buying 1 (and buying 1000 has economies of scale over buying 100). Intel openly publish the “tray price” of their CPUs (i.e. the cost to buy them in bulk) and you probably won’t find those prices at any retailers. It’s the same for other components too.

It’s not just the immediate economies of scale though. There are also economies of scale co=locating your servers. A rack is cheaper than 46 co=location packages and also gives you physical access (so you don’t have to pay for remote hands). You can co=locate tools and spares, for which many co=location providers charge extra. You can connect your servers over a private network and reduce your bandwidth cost, especially if you co=locate a backup server and do backups over the private network.

Renting Hardware

Renting hardware can offer almost the same level of control as buying hardware. Sure, you don’t get physical access, but many large dedicated server providers now have very advanced control panels where you can add extras, take remote control of your server, reprovision your server and so on so there isn’t always a need for physical access.

Not owing the hardware might seem like a bad financial decision (if you draw a parallel to renting a home) but in reality it often isn’t. Consider the following hypothetical yet realistic example where the server provider can take advantage of their economies of scale. Of course these economies of scale don’t apply if you need an usual server configuration.

Owning: £5000 upfront (needs upgrading after 3 years), £50/month to co-locate, 5 remote hands requests at £75 each and 3 disk failures at £100 each. Total: £7475.

Renting: £200 per month all inclusive. Total: £7200.

It’s also a lot more reassuring than worrying about who will be around to do a drive replacement in a faraway city in the middle of the night.


There are several myths about cloud, its performance, its reliability and its scalability. Put simply, cloud is great for small deployments but as you start to approach the need for a dedicated server, you won’t be able to do the same with the cloud at a similar cost. There is a very simple reason for this:

There's no cloud. It's just someone else's computer.

To provide a large cloud server with a certain specification, a quality provider will usually allocate resources on a physical server with slightly more than your virtual server specifications (so you’re paying for more than you can use). This is because the software that creates the virtual server uses resources too. That software is often the cause of a lot of problems, too. Virtualisation software can be highly complex, needing experienced systems administrators 24×7 to deal with mishaps, further adding to the cost. There are software solutions out there to take away some of the complexity but most of these are also very complex themselves and require a lot of investment and room for price variation due to vendor tie-in. It is for this reason that I would advise anyone in the market for an important cloud service not to rely on a company whose whole virtual infrastructure is dependent on their business relationship with a particular vendor. We have known a particular vendor to increase the price of their software tenfold where affected clients were told via a mailing list.

Again, cloud is great for small setups but with the added complexity, you won’t get the performance/cost ratio of a dedicated server.

Claims are often made about the cloud’s scalability and reliability. Scalability may be useful where you need to grow significantly very quickly, then back again, often. This is a rare occurrence and most often, if you compare the cost of a dedicated server to match your largest demand with a cloud server that offers the scalability you need, you’ll find the former is more cost effective. Discussions with one hosting company revealed that one client’s obsession with scalability pays for their whole cloud infrastructure.

Reliability is a hot topic too. Many cloud technologies claim to have automatic failover resulting in essentially zero downtime. CloudHarmony has impartial, accurate uptime statistics for all of the major clouds and all of the major clouds have more downtime than a many dedicated servers. This again, is due to the complexity. More software, means more complexity, means points of failure. Interestingly, many of the cloud providers that people associate with reliability (because of their marketing and higher prices), have uptime similar to, or worse than, the cheaper providers.

Compare this with a dedicated server. We tend to recommend Kimsufi for a cheap place to store backups. Our 10 euro-per-month Kimsufi has been up for over a whole year without any downtime at all. If you need more reassurance than that, look at the same company’s enterprise servers, which are still reasonably priced and come with financially-backed SLAs, RAID and redundant power/network supplies.

Read more →


Benefits of bespoke Xero development

Your accounting system is at the heart of your business. Whether you use Sage, Xero, Quickbooks or something else, there is a wealth of data about your suppliers, customers, financials and KPIs in it. One of the key advantages of Xero specifically, is that it’s cloud based. Being cloud based makes it easier to link it with other systems and to get data out of it in your preferred format.

However, as your accounting package, Xero or otherwise, can’t encapsulate everything you do, it’s common to have other systems with overlapping data. Maybe you use Office365 or Google Apps (Gsuite) for the address book but when a customer moves to new premises, you have to update both systems.

Advantage 1: Keeping Xero in sync

Bespoke software development opens the door to synchronising data in Xero with any other system. This includes Xero’s official integrations – but also proprietary systems, legacy systems and systems that don’t officially connect with Xero. Bespoke software can help you to keep using your other systems – or to use a system that doesn’t officially integrate with Xero – and keep them in sync.

Advantage 2: Reporting

The wealth of data in Xero, Sage and other accounting systems can be hard to extract in the format you want. Let’s say that your KPIs include:

  • The number of customers with whom you work in a given month,
  • The amount of value generated by each individual,
  • Your monthly profit and turnover growth,
  • The ROI on your marketing spend.

Your accounting package holds the keys to this data – but you might have to download a spreadsheet from it, copy and paste in other data from other sources and do some manual analysis. Whilst there is something to be said for manual analysis, there is also a lot to be said for keeping up to date. Bespoke software can provide you with a single pane of glass for your important metrics, presented how you want them to be presented.

Advantage 3: If-this-then-that

Aspects of what you do will undoubtedly be complex. However, it’s likely that some of your processes could be encapsulated by a simple order of events. For example, “when the Xero invoice has been paid, send the equipment to the customer”. Keeping abreast of when each invoice has been paid and ensuring that the correct information is conveyed to shipping could be time consuming. However, this sort of repetitive work can often be simplified or automated entirely by software.

Read more →


Hosting Magento on AWS: why, how and should you?

We can customize applications to run on Amazon Web Services + We manage and develop ecommerce websites built using the Magento platform.
A question that seems to appear strangely often is, “can you host Magento on Amazon Web Services (AWS)?”. The short answer is, “yes”. You absolutely can – but that doesn’t mean that you necessarily should.
AWS provides a range of services for hosting. The best known is EC2, its VPS business. However, there are some 30+ other services that provide IT infrastructure through different channels. For example, Amazon’s Route53 service provides domain registration and DNS services, and CloudFront provides a CDN service (so that visitors further from your server can browse your website more quickly).
The main benefit of AWS, in my opinion, isn’t EC2. EC2 is similar to other good quality VPSs on the market. The main advantage is that you can combine EC2 with the other services and build a very scalable, very reliable and very secure environment for Magento.
Example of a resilient AWS architecture

This is an example of a resilient architecture at AWS. Everything is replicated so that the store is online even if an entire data centre goes offline.

The reason that’s often cited for a move to AWS is scalability. It’s attractive because of the seasonal nature of many e-commerce stores, which is understandable. However, the scalability of EC2 needs to be weighed carefully against the cost of simply purchasing a larger server. If scalability is your main concern, do the maths. How much would AWS cost you over a year, including scaling up and down? Could you use the same budget to get a powerful dedicated server that would serve you all year round?
Depending on the number of peaks during the year and the size of the peaks, AWS may be cost effective for scalability alone. If your traffic is such that one server simply won’t do then you absolutely should consider AWS as one possible alternative.
Another key benefit of AWS is its security profile. It’s common for Magento stores to take credit card payments directly. If you accept credit card payments directly on your store (without redirecting your clients to a payment page hosted by another company such as PayPal) then you probably need to be PCI compliant.
PCI compliance is necessary if your e-commerce store accepts credit card payments
Depending on the type of PCI compliance that you need, there is a good chance that AWS will be one of a very small number of virtual hosting providers that can offer the compliance that you need. This is because, typically, hosting is shared between customers and there are technical reasons why it cannot be guaranteed that your customers’ cardholder data wouldn’t be intercepted by the hosting company or another customer with whom you share hosting resources.
Actually, to the best of my knowledge, Amazon is the only commercial hosting provider that has achieved PCI compliance at Level 1 for its virtual hosting services (EC2, RDS and many of its other services). AWS is also certified to ISO27001, HIPAA and other standards that may be relevant in your industry. If you need to be compliant but don’t need a dedicated server, AWS could be one of very few choices.
Amazon’s EC2 has an excellent track record for resilience. CloudHarmony, that scores leading cloud hosting providers for resilience and performance, haven’t identified one instance of downtime on Amazon’s EC2 for around five years.
However, by harnessing AWS’s other services, it’s possible to combine EC2 and keep them in sync so that, if one fails, even if the data centre if hit by a meteor, you can continue to trade. Such architectures are relatively complex to maintain so you may need external maintenance and support to facilitate the move to AWS and to keep everything running smoothly.
You absolutely can move to AWS – and we can help! Hosting Magento on AWS would best suit stores with highly volatile traffic, stores that require PCI compliance and stores that cannot afford downtime. Other stores might see their hosting bills increase with little benefit. Scalability is all the rage but scalability for scalability’s sake is a poor choice.

Read more →


Lesser-known facts about CPU on virtual servers


Cloud is increasingly popular due to its low upfront and perceived reliability. I wrote an article a few weeks ago about why it can be a sensible choice, but needs to be considered carefully.

One of the main drawbacks of cloud (VPS / VDS / VM / cloud server / IaaS / whatever you want to call it – it’s generally the same thing) – is the CPU power. The physical servers that host virtual servers are very easy to add disks and memory to, and since the incorporation of solid state disks into cloud offers, disk IO is no longer a primary concern either. However, the ratio of CPU resources to memory and disk resources can be rather limiting. Consider the following hypothetical but realistic scenario, supported by anecdotal evidence and server suppliers’ hardware specifications:

2x Xeon E5-2620V3 (12 cores, 24 theads)
256GB Memory
12 x 512GB SSD in RAID10 = 3TB usable space

Let’s package that into VMs by dividing it:

0.375 access to one CPU thread!
4GB Memory
48GB Disk

As you can see, the disk and memory seem plausible but the CPU seems dismal. Of course, this will vary from provider to provider – and you won’t see “0.375 of 1 CPU thread” but more likely something like “shared CPU”.

It can get worse, however, as many providers will oversubscribe the resources on the physical servers – i.e., they will sell more resources than they have, knowing that generally, the physical server will cope ok.

What does this mean for the customer? It means that you need to think carefully about CPU, because it’s the most expensive aspect of cloud, the most necessary for many common cloud applications and the hardest to scale and often, to get solid information on. It reinforces the argument in my earlier post that cloud is great for small deployments, but needs to be weighed very carefully against dedicated options for larger deployments.

The technically interesting bit

A grand challenge in cloud is not just gathering generalised, historical performance data but knowing how my application will perform when I deploy it. There does not exist a definitive solution to this challenge yet, however, there are ways of getting some idea of how it is performing now. Utilities like top, which systems administrators are familiar with, provide some input and have changed in recent years to accommodate the prevalence of shared resources.

VM resource usage


Systems administrators will probably know the first 5 columns of the bottom row very well – they are the most commonly used and, generally, are very useful indicators of how a system is performing. They rely on what is called, tick-based CPU accounting, where, every CPU tick (the atomic unit of CPU resources) is categorised based on how the kernel decides it should be used, and a quick calculation is made to arrive at percentages. The kernel uses a scheduler to manage the CPU resources efficiently when there are potentially far more applications running than there are CPU threads available, by switching them in and out.

You may be interested to know that the unit of time between CPU scheduler decisions is, in technical parlance, called a jiffy and that the algorithm used to schedule CPU resources in most modern Linux systems is based heavily on the Brain Fuck Scheduler, written by an anaesthesiologist turned computer scientist, who liked to argue on forums.

Anyway, the very last column, “st”, stands for “steal” time. This is supposed to account for CPU time that is “stolen” by the virtualisation software on the physical server and is therefore completely unavailable to the kernel on the virtual server. The virtual server isn’t really aware of why it isn’t available but in a virtual system, it’s safe to assume that it’s probably the result of noisy neighbours and high contention.

Steal time is not easy to measure, as virtual servers are dependent on the virtualisation software on the hypervisor for the resources that would otherwise be physical – that includes the obvious ones like storage devices, network devices and so on, but also the simple act of keeping time, which is an integral part of the system. Thus, the number that you see under the “st” column depends not only on contention but also the virtualisation technology. Literature suggests that IBM’s virtualisation technology allows for CPU time to be accounted for extremely accurately, followed by Xen, followed by the others that are less effective. In our Proxmox setup, no steal time was reported even under very high load and oversubscription

Read more →