Laravel Data-Driven Strategies #2: setup user session recording in your application
“No matter how cool your interface is, it would be better if there were less of it.” ~ Alan Cooper
First time I read this quote I thought it was kinda fun because every time you think with your UX Designer about the connection between interface and features, it needs a lot because of that button, or that text colour, or that shadow… in short, if you ever worked with this, you understand what is like!
The real problem is that while projecting a UX you have to focus on user behaviours and… be realist, you’re not your users and even your UX Designer isn’t! You’re compromised, and that’s a fact, you’ve to accept that.
Obviously, I’m not saying that your UX Designer is not good at his work, the UX Design Process is something really difficult and needs a lot of creativity, research and focus, but times to times it may need help from the outside.
And the outside I’m talking about is made of your users! How can you include your users into the UX design? It’s kinda easy: recording them.
It may sound strange if you never thought about it. Now I suppose you’re thinking about what happens when you get into a website, move your mouse and someone on the other side is looking at you in pyjama scrolling down your feed for the latest news, that’s awkward! When I talk about recording users, I’m referring to the user sessions: the entire time-lapse, mouse movements and events that happens from when the user opens the first page until it closes it. It’s interesting to see how a user really behave on your website because it can lead you to reconsider lots of stuff. For example, It can show that users don’t use some features because they don’t find them easily accessible or that a feature isn’t accessed as you thought it would be.
Recording user sessions can really change your way of projecting a UX, so let’s dig into it and see how it works!
Setup
Many online services allow you to record user sessions, some are free like Yandex Metrika, some are paid like Hotjar, it all depends on what you want to achieve. Here we will try to build our custom solution for screen recording based on Cimice, an experimental user session recorder that you can find on Github.
Everything you need to start session recording is the cimice build available in dist/cimice.min.js.
To help to record user sessions on every page just create a custom blade file to embed where needed. The views/cimice_recording.blade.php will contain:
<script src="{!! asset('public/path/to/cimice.min.js') !!}"></script>This is just the import for the cimice script, but it must also be configured.
The Github second example is really simple but for testing purpose or very simple cases, it just works.
If you put a console.log(movieJson) to check what’s inside, you will see a JSON object containing:
- frames: an array containing the recorded frames.
- left: left page offset.
- top: top page offset.
- scrollX: horizontal scroll offset.
- scrollY: vertical scroll offset.
- scene: the Base64 encoded HTML currently on screen.
Note that if the HTML vary during the same page session, you should tweak the above script to pass the entire movie instead of sending just frames.
If you want to store the entire session in your database, you have to use an appropriate model via the relative migration php artisan make:migration create_session_recordings_table --create=session_recordings. The table should keep:
- session_id: the recorded session.
- user_id: the user you’re recording (if you wanna track it).
- payload: the JSON sent from the client.
- previous_session_recordings_id: this will be used to chain every recording sent from the client.
$table->id();
$table->string('session_id');
$table->foreignId('user_id')
->nullable()
->constrained('users')
->onUpdate('cascade')
->onDelete('cascade');
$table->json('payload');
$table->timestamps();We want to use this data via a Model so just php artisan make:model SessionRecording. And we also want to be sure that every property is fillable.
Now we have to prepare an API to store our recordings, start with a controller php artisan make:controller SessionRecording\RecordingController and a request php artisan make:request SessionRecording\StoreRecordingRequest.
To keep request simple we just want to authorize it when a JSON request come and put our data in rules.
public function authorize()
{
return $this->wantsJson();
}public function rules()
{
return [
'frames' => 'required_if:scene,=,null|array',
'left' => 'sometimes|nullable|numeric',
'top' => 'sometimes|nullable|numeric',
'scrollX' => 'sometimes|nullable|numeric',
'scrollY' => 'sometimes|nullable|numeric',
'scene' => 'sometimes|string',
];
}
The controller logic will be simple too, it just needs to store data.
public function store(StoreRecordingRequest $request)
{
$inputs = $request->only(['frames','left','top','scrollX','scrollY','scene']); $user = $request->user(); SessionRecording::create([
'session_id' => $request->session()->getId(),
'user_id' => $user ? $user->id : null,
'payload' => $inputs
]); return response()->json([
'success' => true,
]);
}
So the sendRecordings can be finally tweaked:
function sendRecordings(recordings) {
let movieJson = JSON.stringify(recordings);
fetch(
'{!! route('recordings.store') !!}',
{
method: 'POST',
body: movieJson,
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json',
'X-CSRF-Token': '{!! csrf_token() !!}'
},
}
).then(console.log);
}Please note this is just example code, you can tweak for your usage.
The recording
Recording user sessions now is super easy, you just have to include the cimice blade file, and that’s the trick!
@include('cimice_recording')That’s it, nothing more to do!
Dev through the looking glass
Once everything is set up and you started recording sessions, you probably want to even see what’s recorded.
Please note that here we won’t build a dashboard for looking sessions and protecting data from being watched from unauthorized users, that’s up to you. If you want to quickly build a dashboard, you can check my other article about Infyom Generator🤓!
Here we want to gather data from our database. The database structure I outlined before it’s really simple, you can group data by session ID or by user ID, or you can add other data to the database structure in order helping you gathering more specialized data: for example, you can add user-agent, request headers, or aggregate this data with other analytics systems, gathering it’s up to you!
To fetch every session recorded this way you can just:
SessionRecording::distinct('session_id')->get('session_id');Or if you want every ID related to a session using MySQL/MariaDB raw syntax you can get them this way:
SessionRecording::groupBy('session_id')->selectRaw('session_id, GROUP_CONCAT(id) AS ids')->get();If recording sessions are too chunked you can even merge them using reduce like this:
$recordings = SessionRecording::where('session_id', '=', 'a_user_session_id')
->orderBy('created_at')
->get()
->reduce(
function ($rec, $v) {
if (empty($rec)) {
$rec = json_decode($v->payload, true);
} else {
$rec['frames'] = array_merge($rec['frames'], json_decode($v->payload, true)['frames']);
}
return $rec;
},
[]
);
usort($recordings['frames'], function ($v1, $v2) { return $v1['ts'] - $v2['ts']; });How does this work?
- You get every recording about a specific session via the query
- The reducer will start using an empty array
- The first time the reducer will just put the payload inside the reduced array
- Every time from there, the new recorded frames will be put together with the previous frames.
- After that, just to keep it safe, the
usortwill just sort the frames basing on thetsproperty (timestamp).
Now that you have all your recordings, you’re ready to show them! You can put the code to show the recorded session into a separate file like cimice_play_script.blade.php.
Note that you can tweak how the player will show every single event. The registered events can be mousemove, click, scroll, resize, contextmenu (you can also listen to any other event but you have to tweak the initial configuration), so using player.on('event', ...) you can edit how your mouse will appear.
Once you put this file into your views folder, you should simply do something like this:
<div id="player"></div>@include('cimice_play_script')
<script>
let movieJSON = {!! json_encode($recordings) !!};
playMovie(movieJSON, document.getElementById('player'));
</script>
This will create an iframe into the div passed as an argument that will show the user session!
Remember that user session is not necessarily linear: every user can tab to another window, wait for seconds without doing nothing, can open the developer tab, you cannot exactly predict how the user will use your website, but it’s a start!
But… privacy?
The purpose of your Data-Driven Strategies should always be to give users a better service!
Your users are not things you need to exploit for your purposes. If you’re building something like this is because you want to improve your service, to make it more usable, to give your users a greater experience. Your KPI and your metrics exist because of your users and if you want to build something like this should be because you want to give them a service they can use and they appreciate.
Your users should always be conscious of how you gather and use their data: remember to explicit this kind of system on your privacy policy, anonymize data where you don’t need them to be related and keep and respect a data retention policy where it’s explicit when data will be deleted.
Now it’s up to you to build something great! 🥂
Stay tuned for other Data-Driven Strategies and if you want, take a moment ️️to leave a comment about how you take data-driven decisions supported by tools! ☕️
