Building generic Shiny applications with a data module

We’re rebuilding our patient experience dashboard at the moment, partly to incorporate some of the work that we’re doing on text mining and partly to connect it up to other types of information we have like staff experience and clinical outcomes. It has to be reusable because we’re using it as the basis of the text mining dashboard that we’re building for any provider trust to use with their friends and family test data. We’re trying to make everything reusable anyway partly because the different bits of the NHS should all cooperate and produce stuff together that everyone can use and partly because we’re realising that when you make code reusable the first person who can reuse it is you, 6 months later on a similar project.
So we have a patient experience dashboard that we want to hook up to our other data in our own context, and we have other trusts who want to take what we produce and hook their stuff in to it. An obvious approach is to use modules in Shiny and have a data module at the heart of the operation, which can take account of the different datasets available in each context. The next level would be a “summary type” module which would bring together all of the inputs and outputs for a particular type of data- one for patient experience, one for staff experience, etc. And the bottom level would be submodules which would carry out the individual tasks- one module to produce graphical summaries, one to produce downloadable reports, etc. We would use the {golem} package for all the reasons that we normally use {golem} (help with dependencies, packaging, testing, and deployment)
The basic structure would look like this:

• Data module. This module would be responsible for loading ALL data, patient experience data and any other datasets that the person deploying the application might want- clinical outcomes, staff experience, whatever it might be. This module would contain upload and download features as appropriate (users may wish to upload their own data or download processed data) but it would in the main merely draw from databases, process the data into a useful form, and prepare them to be imported into the other modules
• Function module. Each of these modules is responsible for a particular type of data (staff, patient, clinical outcome). In some trusts the dashboard wouldn’t have access to data of all of the necessary types and these modules would not show in these contexts.
• Sub modules. Each function module would have access to submodules which do different tasks- draw graphs, make reports, etc. Because some types of data are quite similar (e.g. staff and patient experience data) there is potential to reuse these modules in some areas.

Although this is a nice approach there are lots of different ways of achieving this and I’m hoping for a bit of guidance as to which might work best.

1. “Automatic dictator” data module. This would be a data module that works with no instruction from the person deploying the application and which instructs all the other modules. This data module would simply attempt to load all of the data, catch any errors that result in doing so, report them to the user (in an unobtrusive format- “No staff data found”) and then instruct which of the function modules should appear. Further, it would export not only dataframes but also a set of instructions to the function modules as to how to carry out their processing. For example, there may be just one or several free text questions in a patient experience function module. The data module would determine that and would export with the dataframe a description of the dataset for the use of each function module
2. “Automatic” data module. As above, except it would not export instructions, but just data. Function modules would be responsible for understanding the shape of the data and would instruct their submodules accordingly.
3. “Programmable” data module. This data module would accept a list of options within the run_app() function and would therefore be set up individually within each trust. For example, the run_app() function might be passed list(patient_experience = c(TRUE, 2, 5), staff_experience = c(FALSE)), indicating to draw the patient experience function module with 2 free text questions and 5 Likert-type questions, and to omit staff experience. In practice the options list would need to be longer, but this is just to illustrate the principle. The data module would load and process the data according to the arguments in run_app(). It could then either instruct the function modules as in the dictator data module or leave it up to them as in the automatic dictator module.
4. “Dumb” data module. This data module would be dumb. It would just load the data, process, and export it to the other modules. The function modules would be expected to react intelligently to the presence or absence of data and to make sense of the structure of the data. I don’t think this example would work well because there would be nothing to turn off the function modules, which mean there would be blank tabs in the application, but I’m including it for the sake of completeness in case it gives me or anyone else an idea at a later stage.

Most of the options, except 4, are pretty similar, but they have a lot of scope to change the overall way that the application is programmed. Leaving stuff up to the function modules is probably more flexible but it may be that all the function modules need tweaks in different trusts, which is not really what we want. Having an automatic dictator model is very attractive in that you should just be able to seal the whole thing and deploy it anywhere but the reduced flexibility inherent in that approach may make it more difficult to actually get it working in different contexts. In practice some of it is probably a bit of a grey area anyway- I don’t think you could have a completely dumb data module, and nor could you have a completely dumb function module. Even the submodules will have various degrees of “dumb”. It’s more about getting an idea of where you build the flexibility and how much control you give each module over its behaviour.
I’ve written this blog post to show it to one person whom I will now ask politely on Twitter for their opinion but if anybody else has any thoughts I would be very glad to hear them. It’s better to tweet @chrisbeeley because I don’t seem to get email notifications from blog comments for some reason.