Observing the decentralised social web
(one telescope at a time)

Web Observatories concern the use of peoples' digital footprints as the subject of academic inquiry [9, 26]. Such data encompasses all manner of online and offline activities, and it may be collected passively by systems with which individuals interact, or actively logged, or some combination of the two. In order to address negative privacy implications of collecting and analysing this data, [11] introduces the idea of a Personal Web Observatory. Personal data stores are presented as an architecture for a decentralised Web Observatory, to allow individuals to maintain control over their data whilst still participating in scientific studies or otherwise releasing their data for use by third parties.

A Personal Web Observatory relies on individuals opting in to self-tracking activities; connecting their personal data store to sensors or user interfaces.

There have been a small number of high profile instances of individuals collecting a large volume of data about themselves, then offering it up for auction [5, 20]. Even more common are those who track data about themselves in order to: orchestrate behavioural changes; monitor bodily functions; learn new things about themselves; discern cause-and-effect relations; aid memory; manage information and life events; make better decisions; or just for the fun of it (for surveys of self-tracking motivations and tools, see [12, 2, 7, 23]). This personal self-tracking is disruptive to traditional notions of big data and data science [23].

The Quantified Self (QS) community is large, global, and growing. In-person conferences and meetups are held regularly [14, 15]. Whilst they use a wide range and combination of DIY and off-the-shelf devices for self-tracking [15, 21], what is particularly interesting are their reflective practices. QS practitioners engage in creative ways to collect, visualise, and understand data about themselves. At meetups, talks are focussed around deep personal insights, optimisation and improvement, and active self-awareness, rather than technology or tools [2, 21, 15].

However, the psychological impact of quantifying day-to-day activities is more complex than detecting trends and setting goals. Self-tracking may work against our best interests as interpretation of data is both subjective, and prone to re-interpretation at each viewing. There is also evidence that attempting to enforce a particular behaviour can have the opposite effect. Peoples' memories and impressions are easily influenced by external and internal factors [3]. A participant in [12] expressed concern about becoming "compulsive" about data collection, and [12] also describes how many off-the-shelf self-tracking services do not provide adequate means to aid user reflection.

Another negative effect of QS tracking is poor security of sensor software and third-party storage which can compromise individuals' data [1]. Plus, using third-party software typically comes with terms of service which are problematic from a privacy perspective.

Quantified self tracking is gamification of non-play activities, and [28] describes gamification as having surveillance at its root. [28] emphasises that quantification is a tool for governance and control and [15] concurs that statistics are historically used to manage populations, and this form of control is internalised by individuals for management of self. Prevalence of QS devices and habits can serve to normalise surveillance. Further, QS tracking in the workplace is being introduced in ways that are becoming increasingly difficult to opt-out of, and raises unrealistic expectations of workers "fostered by a quantified, machine-like image of human productivity" [13].

Nonetheless, [15] describes QS participants who pushing back against the expectations and categories of the companies whose devices they use with their own interpretations of their data, "calling into question who gets to do the aggregation and how".

Though most bloggers and social media users do not explicitly set out to log small events in their lives, other kinds of social publishing can also be considered a form of self-tracking. Indeed, it is these types of data which Web Observatories today are most commonly consuming and analysing. Blogs have been around since the early days of the Web, initially for collecting links and then for documenting ideas and experiences. More recently, such expressions are commonly published to centralised social networking site, and many systems facilitate deeper data logging than just prose content. For example, the Facebook 'status update' input presents preset options for mood, location, films, books, friends, and other activities. These are easily attached to a text post through a seamless user interface, where the intent feels quite different to services and devices specifically marketed for self-tracking, but the end result in terms of data collected is largely the same.

While there are various efforts to address decentralisation of the Social Web [8], of particular note are ongoing formal standardisation efforts at the W3C. The W3C Social Web and Web Annotations Working Groups are developing standards for not only publishing content, but interacting with content in the same manner as one would through centralised social networking sites. Unlike centralised systems, instead of coordination between common technology stacks or shared access to databases, they use mutually implemented communication protocols. The ActivityStreams 2.0 (AS2) [19] specification provides a syntax and vocabulary for types of data commonly found on the Social Web, as well as an extension mechanism for adding new types of activities and objects. ActivityPub [24], Micropub [16] and the Web Annotations Protocol [18] define protocols for publishing which facilitate decentralision by decoupling clients from the servers which store the data. Linked Data Notifications (LDN) [4], Webmention [17], and WebSub [6] provide mechanisms for propagating social interactions and other data between otherwise unrelated servers.

The system (which is named sloph) constitutes a central database which is an RDF quadstore (layered on top of a MySQL database by the ARC2 PHP library). Using a graph data model facilitates the addition of new data without the overhead of updating schema or models in the core code.

For incoming data, it uses two endpoints: publication and notification, which implement the server portions of ActivityPub and LDN respectively. To compensate for overlapping standards, the publication endpoint additionally includes bridging code which converts Micropub requests in ActivityPub requests before proceeding, and the notification endpoint converts Webmentions into LDN. Data is processed to examine its validity, and stored as-is in the quad store. Publication data all uses the AS2 syntax and vocabulary (or extensions thereof), and notification data is stored using whatever vocabulary is sent by the notifying party. Data from each endpoint are stored in different graphs in the quad store to manage provenance.

The notifications endpoint can receive unsolicted messages from any LDN Sender, which may be somebody else's personal data store, or a clientside tool. In addition, some third-party services have been configured to send notifications to the notification endpoint. Webmention.io and GitHub have webhook settings, which are set to the notifications endpoint. They forward JSON data, which is easily convertible into JSON-LD, used by the LDN standard. Brid.gy is a service which runs in the background and monitors my social media profiles for replies to my posts, then sends these as Webmention notifications. Upon certain new incoming notifications, the endpoint sends a request to the PushOver API, which sends a push notification to my Android phone.

Data may be retrieved as individual items (AS2 Objects or Activities), or in sets (AS2 Collections); all are identified by URLs. Content negotiation is employed so that requesting clients may access the data in any desired RDF syntax, or HTML. For HTML display, simple templates are created for each 'type' (or shared between a set of similar types) of data item. It is expected that most requests come from Web browsers, so the HTML content is delivered most often. However, other applications or services may consume the data, including readers (which may mix together multiple streams of data for the user), aggregators (which read the data and perform some manipulation or calculation over it to display the results) or publishing clients (which offer editing or combining of existing data). All data is public; I did not implement access control.

The publication endpoint performs additional functions for data enhancement and distribution. It automatically adds missing metadata to posts if necessary (for example, published date and author), as well as storing new relations between posts and relevant collections such as tags, as well as the specific collections required by ActivityPub. The publication endpoint forwards text posts to Twitter, if necessary truncating them and adding a link back to the original, which helps with reach of content (since I don't yet have a subscription mechanism implemented). The publication endpoint also scans the content and certain attributes of incoming data for URLs, and behaves as an LDN and Webmention Sender to deliver notifications to others, if possible.

[11] suggests that the first feature of a PWO should be to allow individuals to consolidate data collected by third parties into one repository under their control. Rather than attempting to aggregate - or even find - all data about myself spread across the Web, I chose a handful of services which have particular value to me, that I have been using to actively log particular things. I exported data from Lastfm (over ten years of music listening history), Twitter (7 years of short notes), Runkeeper (1 year of runs, walks and hikes with GPS traces), and Github and Bitbucket (5 years of code commit history), and Firefox bookmarks (2 years). I also exported data from 750words.com (almost 7 years of intermittent use) but did not import this into my store due to private content and no reliable access control built in.

On top of these data dumps I created the following templates:

Type of data	Attributes displayed *
* All posts contain tags and a published date, and may contain a last modified date.
Articles (blog posts)	name, content
Short notes (like Tweets)	content
Meals logged	description, restaurant if applicable
Travel plans	start and end location (map), date and time, means of transport, cost
Checkins to specific places	location (map)
Checkins to categories of place, aka 'vague' checkins (eg. 'home', 'office')	location, duration of time there, associated colour
Likes	URL of thing liked
Bookmarks	URL of bookmark, name of bookmark, optional comment or quote
Reposts (aka shares aka retweets aka reblogs)	URL of post, optional comment
Acquisitions (purchases and free stuff)	description, source, cost, photo
Additions to photo albums	photos and URL of album
Events and RSVPs	location, date and time, name, description, event website
Subscriptons / follows	URL of profile followed
Sleep times	start and end date and time, optional comment

Templates were created not all at once, but as I decided to start tracking something new and wanted to visualise it. Templates were continually modified and improved over the course of the year.

At the time of writing, I display posts in three different formats on my homepage (figure 2): a feed of the most recent eight article and note posts displayed in full; a list of the most recent of each type of post, displayed as a sentence (eg. "the last thing I ate was toast with peanut butter, 25 minutes ago"); and the last 1600 posts of all kinds, visualised as a string of small coloured boxes with icons. The colours represent where I was at the time of making the post (according to the most recent prior 'vague' checkin) and the icons indicate the type of post. Clicking on any of these boxes takes you to the post itself. In addition, the background colour of the homepage changes according to where I am at the present time. I also show my top 128 tags, and the number of posts for each.

Another type of output is a /summary page, which aggregates data between any two dates, defaulting to the past seven days. This is useful for producing a year- and week-in-review, and includes total amount of money spent, top foods eaten, number of words written, and various averages.

All data is actively recorded; that is, I enter all of the data and no posts are triggered by some other process or sensor. Unfortunately, for the sources of data exported from other services mentioned previously, I did not (yet) implement a connection to their various APIs to post subsequent data directly to my site.

As I decided to log a new kind of data, I either created or sought out a suitable client. As an intermediary measure (eg. while a client was in development), I could insert data into the quadstore directly using my SPARQL endpoint, which turns out to be a pretty useful bootstrapping measure. Clients I developed myself are simple web forms, which post AS2 data as JSON-LD to my publication endpoint.

I typically logged data at the time of its occurrence if possible. If I didn't have an internet connection (a frequent occurrence whilst traveling) I used a local 'timestamp' Android app to record the time and note crucial details, then back-dated posts at the earliest opportunity.

[11] proposes that "small data analytics, while sparse, could be made statistically viable when gathered longitudinally over time." Whilst I haven't done any complex statistical analysis on my data, certainly aggregated results, counts, and some calculations of means have been personally insightful. To take some trivial examples: that I drank on average 0.8 cups of coffee per day in 2016 was lower than I expected; that I spent an average of $28.71 per day was higher than I expected; and that I spent 1 month, 15 days, 6 hours, 41 minutes, and 15 seconds travelling between places is just kind of interesting (and something I particularly wanted to find out when I started recording 'vague' checkins). I don't expect these statistics to be remotely captivating for anyone else; such is the value of personal "small data."

Such aggregations were able to be put to more focused uses. Logging all of my purchases did not raise my internal awareness of how much I was spending on a weekly or monthly basis, but when I realised I wasn't saving money after about six months it was trival to write a small web app which consumed my data stream, aggregated the total cost by certain categories, and displayed the amount I spent in any given month or week. Adding a setting for a monthly budget goal let the app send me notifications to stop spending when it noticed I was on track to exceed this in the current week. The app is not coupled to my personal site or data, so it can be used by anyone who publishes a stream of their purchases using AS2.

An initial motivation of recording 'vague' checkins was so people could check my site to see whether I was on my way if we were planning to meet, or if I shouldn't be disturbed (eg. if I was in a meeting). Industrious computer science friends created an IRC bot that consumed my /where endpoint and responded when asked $whereis rhiaro. Other users of the same IRC channel added their own location reporting endpoints for the bot to read, some down to GPS accuracy with a map, shortly thereafter. Other unanticipated uses of data I was recording include launching a travel blog which filtered travel-related posts and photos from my main feed, and a food blog which includes my food logs, food-related photos, and any posts or bookmarks about recipes or restaurant reviews.

Having visible output for each log on my website motivated me to keep logging. I am missing lots of checkins to specific cities because I had trouble with a maps API and didn't complete the template to display them. Similarly I never got around to creating a compelling view of sleep records, so logs for those are few and far between.

I felt an internal pressure to ensure my data stream was complete. If I was offline or out of battery for a length of time, I would keep logs on paper to back-date later. This was due both to wanting to ensure aggregate data was accurate, and fear of 'losing' associated memories. Relatedly, I looked (and in fact still do) through historical logs with surprising frequency, and found memories of events, people and places were triggered by descriptions of meals or photos of purchases that I might otherwise have forgotten. I worry that this is unhealthy, boarding on obsessive behaviour though. At times, particularly before I had a streamlined offline-logging plan in place, or if it was socially inappropriate to be writing or typing logs, I felt anxious that I would neglect to log something.

When my webhost experienced outages, leaving me unable to record data, I both noticed far more quickly and felt a far stronger personal impact than I previously would have when my site was mostly static and for infrequent blogging. I complained to my hosting provider more frequently, and projected a sense of urgency that was probably disproportionate. I was left with a feeling of if my website is down, I do not exist, and I found myself wondering if sysadmins in a data center the other side of the world could possibly know or care about the anxiety they were causing me.

Other, predicted, psychological impact was that publicly logging photos of all of my purchases made me more conscious about what I bought. Even though I didn't actually think anyone looked at my feed, I found being aware that someone could see it helped me to, for example, resist buying junk food at the supermarket.

Good user interfaces for visualising data from logs and perhaps making recommendations for improvements to a particular aspect of someone's life can be a strong way to encourage people to engage in self-tracking. However given the potential for compulsive behaviour around self-tracking, we also need to enable people to be 'off the grid' without affecting their overall aggregates or statistics. The motivational write-every-day site 750words.com has a 'vacation' setting, by which you can tell the system when you'll be away so that it doesn't penalise you for 0-word days during this time. If self-tracking is going to become the norm (as implicit self-tracking via social media already is in certain parts of the world, and as would be beneficial for Web Observatories) then taking breaks from tracking should be part of that norm. The benefits of continuous self-tracking can be acknowledged, but disconnecting can also be accepted as a healthy practice. Building this in from the outset can perhaps help to mitigate against internalised normalisation of surveillence, too.

Relatedly, the effects of missing data or inaccurate logs for any reason may have a damaging psychological effect on people. I posit that an important part (and in fact an ethical responsibility) of Web Science research would be to examine these effects with at least as much priority as working on the best gamification techniques for encouraging people to self-track.

It is critical that interfaces for visualising data logs are effective in helping the subject of the data to interpret its meaning. On the one hand, simply displaying correlations may incline people towards inferring spurious cause-effect relationships; on the other hand, as members of the QS community have shown, individuals may be far more effective at interpreting their own data than a third party service - which is perhaps missing other relevant information - could be. A challenge for PWOs would be to offer data visualisation interfaces which not only take into account all available data, but account for what is (or might be) missing as well.

Furthermore, there is no one-size-fits-all for causing positive behavioural change based on self-tracking. How we can use personal data logs to improve the lives of individuals might vary by personality as well as the social and cultural context in which the tracking takes place. Ways of tailoring PWOs may be extremely important when it comes to actually benefitting their users.

From a traditional social media Web Observatories perspective, subjects are not aware necessarily that their data is being observed and analysed by academics. People use social media for different reasons, and I suspect that enabling social scientists to understand the world is pretty far down the list. Especially given the assumption that users of PWOs are actively opting in to being observed, and perhaps have fine grained control over what can and cannot be used for the purposes of research, is going to impact how they engage in self-tracking. As people shape and are shaped by social media use, people will shape and be shaped by their use of PWOs. What are the tradeoffs when it comes to benefits to individuals, and benefits to academic research? How can we take this into account when drawing conclusions from collected data?

People are likely to be discouraged from using a PWO if it comes with a high burden of maintenance or cost of running [11]. I was able to minimise the impact of this on myself by integrating it into a system I was already maintaining (my personal website). Doing this had a significant impact on the technologies I was able to use, which was in some ways restricting, but beneficial in the long term. Shared Web hosting, PHP and MySQL are widespread and well-supported; this demonstrates that a PWO need not be architected around specialist or niche technologies, and need not be difficult or burdensome to set up and use. Certainly lowering the barrier to entry to people who already run their own websites could help to springboard adoption.

Centralised services are frequently bought out, shut down, or change focus or terms of service. Whilst the technical burden of maintaining ones own personal data store may be higher than delegating this to a third-party comes with different, but not insubstantial, risks.

Serious review of common practices and formal Web standards can smooth the path to interoperability between different instances of WOs and PWOs. Though the standards discussed in this article were designed from a decentralised Social Web perspective rather than a Web Observatories one, the overlap is clear. Implicit self-tracking that makes up a part of ordinary social media use can be supported in the move towards decentralisation, and privacy-preserving PWOs may start to exist as a side-effect. In my implementation, I did not address the use case from [11] of aggregation of data from a crowd to produce net benefit. This is also something that shared use of open standards, in particular for data representation, subscription and notification, could facilitate.

Anticipation of future needs is raised as a challenge by [11]. With regard to my changing motivations and day-to-day requirements, I did not set out with a specific list of things I wanted to record and design the system around that. Instead, I used a flexible schemaless architecture which meant that for any new type of data I decided to log I had a minimum of new engineering to do: build a client (or potentially hook up a sensor) to generate the data; and (optionally) create a template to view it.

Working with open standards for creating, updating and deleting data helped here too, as I had the option to look for pre-existing standards compliant clients to post to my data store. Conversely, others whose data stores use the server portion of the publishing protocols can use my clients with their own storage.

Discussion of Personal Data Store architectures tends to revolve around reducing fragmentation and prioritises gathering together all kinds of data in one place [25]. This results in generalised tools and interfaces, which try to make it possible to do a variety of things in one place. I argue that more effective and appealing applications are specialised: particularly good at doing one thing. Whilst the data store itself is generic, standards for decentralisation permit the decoupling of clients - both for creating and displaying data - which is perhaps the best of both worlds.

Web Observatories which collect data from centralised social networking sites may be vulnerable to ingesting misinformation, ranging from subconscious selective disclosure to deliberate acts of protest against privacy infringement [10]. It is difficult to say whether proliferation of personal WOs would mitigate this, but it becomes increasingly important to find ways to capture contextual information when data is recorded.

If Web Observatories begin to collect data from both decentralised and centralised services, it is natural to want to align the data so it can be combined into the same data set. However the source of the data cannot simply be discarded. The same type of data from different types of sources is not necessarily equivalent or directly comparable. Just as different centralised social media sites (and the communities and subcommunities within) have their own cultural norms and expectations, as well as technical constraints or affordances, individual personal data stores come with their own unique contextual information as well. In the decentralised case, the context for data logs may be more difficult to discern, as well as capture.

To take a concrete example: for researchers to find more meaning in the logs in my PWO they need to look at in the context of all of the systems I interact with. My system forwards text notes and longer articles to Twitter, which in term drives traffic back to my site where all my other kinds of logs can be seen. Researchers studying my data would need to consider how my awareness of my Twitter audience (directly through known followers and extended through their audiences in the case of retweets) impacts all of the content I post. This might be different for someone who shares their PWO content with a different social network, or not at all.

The interface used to log the data can also have an impact. Whilst I use a Web form based user interface to check in to a place, someone else might generate exactly the same data automatically by the GPS on their phone, making less of a conscious effort to record their movements. Researchers studying how people use, for example, Swarm, have the advantage of knowing that the interface used by everyone is consistent as well as being able to directly explore it themselves. I anticipate that PWOs will be far more diverse, personalised to fit into their users' day-to-day lives, and possibly inaccessible to researchers.

Using data from QS activities for scientific research comes under fire for a number of reasons: self-reported data may be unreliable or biased; context is often lost when quantifying qualitative data; and data collection is limited to individuals who are inclined to record details about their lives [2]. Though I am not using the contents of my data logs in this report, the same issues apply to my recall of the effects of logging.

Perhaps most importantly, though the length of the study is significant, my sample size is 1. As such, I make no claims about generalisability or conclusivity of the results. I am documenting these experiences as a starting point, to begin to explore Personal Web Observatories in depth, and to highlight areas for focus in the future; this is similar to [25], in which the authors conducted their research on MyLifeBits with large amount of a single person's data as well.

I invested a considerable amount of time in building new features, fixing bugs, and making small improvements and adjustments to my PWO. As the only user and only developer, I was first to notice if something wasn't working properly, and unhindered by conflicting interests or opinions when it came to making changes. This has advantages for rapid prototyping of a somewhat novel system, as well as commitment to the ideology of dogfooding: if I don't want to use a system I've built, why should anybody else? It similarly meant that features I lost interest in fell by the wayside, whilst features I used regularly were well-maintained, so I didn't invest time in developing unnecessary functionality as I might have done had I been designing this for others. Whilst use cases and UI improvements were constrained strictly to my personal needs, and discussion of features was limited to a handful of like-minded developers, this approach was nonetheless appropriate for the purposes of this analysis.