Sometimes you bring an idea to the table, and then roll with it. Other times, you are given a job to do, that has already been sold in at director level. This began life as the latter. With a timeline of a month or so.

After sitting down with our reporting analyst, expectations had to be reset. Sometimes, data comes in from a single source, nicely structured, easily accessed and life is simple. More often, information is buried deep in semi structured documents, such as Excel, or in systems that are quite rightly secure and protected. In our case, we had a real mix of these. And one data source in particular, was buried within around 500 spreadsheets which represented six months of data. Every week, more were created. Crazy it may seem and a clear candidate for some process changes, but that was outside of the remit.

A responsive web app was sold as a concept, quicker and easier to build than a mobile app, the reach would be to a wider audience, and a web API would have been required for a mobile app anyway.

All the information sources were collated by a custom built windows C# app, the Data Miner, which could run on an overnight batch, pull from a wide variety of sources, deal with overlaps of timeframes for existing data versus new or changed data, and repackage the entire set into JSON, automatically pushing the data to the web app via sFTP.

For the web app, I chose ASP.NET as the platform, with the application around on Microsoft Azure. Responsive web design centered around the use of BootStrap, with some styling to fit with our corporate branding. Authentication was tricky. Some off the shelf stuff was available, but needed work and then it had to pass our formal IT security audit, including penetration testing and a full code review. Ho hum! It was a long two weeks waiting for the outcome. Fortunately, the issues identified weren’t insurmountable.

The app still runs today, despite some of the underlying data sources changing, only the Data Miner has needed updating.