trenchant.org

by adam mathes

Are There Viable Open Source Slack Alternatives?

Some are close, but I believe there will need to be serious investment in native mobile applications for the existing open source solutions to get there.

What Is Slack

Slack is incredibly popular, high growth, proprietary chat software.

You should probably try it if you haven’t already, it’s nice.

What IS Slack

It’s a little more than that. Key elements that make Slack Slack

  1. real time chat
  2. asset hosting and sharing (images, links, etc)
  3. search of text and assets
  4. understandable authentication and permissions
  5. client software for web
  6. platform for automated users (bots)
  7. native mobile clients for iOS and Android with push notifications
  8. a bunch of social norms and behaviors around talking to coworkers (with emoji and gifs)

1-5 are nearly a commodity at this point in various forms, though generally not packaged very well together as a system.

The value of 6 can mostly be replicated by competing platforms as de facto standards emerge, but hasn’t.

7 is actually a competitive differentiator for Slack right now, and is the focus of the conclusion of this piece.

It may seem like I’m mocking 8 but I’m not. Setting the cultural context and norms for communication software matters a lot more than people think, and how software makes you feel is part of why people will or will not use it. (This is something that open source software, and software generally, ignores.)

All communication and social software is about people, and how they feel. When you breathe culture and personality into software, it has an impact on that. 1

Why Use Slack

Slack is a great way to get the above with very little hassle in a hosted, integrated offering with momentum.

I like the software! It’s nice. There’s an attention to detail, polish, and craftsmanship in what they’ve made that is very admirable.

I have friends that work there, and they’re all nice too.

Why Not Slack

Cost may be an issue, though for most business usage I doubt that will be the deciding factor.

For personal usage, it seems cost prohibitive if you want to keep access to your archives and get the “full” experience.

Bigger issues that may be important to you:

  • data security and ownership
  • federation
  • long term viability of Slack Technologies Inc.
  • cost structure in long term
  • long term viability of proprietary protocols, closed source code vs. open protocols
  • lack of control over your own communications archives and destiny
  • uptime and stability
  • government surveilance, end to end encryption, privacy

You can argue that exports and backups and documentation can mitigate a lot of the above but in 2016 it feels like real time chat communications platform should be something anybody can spin up and use without hassle, and without having to deal with a giant VC funded US corporation, no matter how benevolant you think those entities may or may not be.

Much of the above I think are just fundamentally not going to be addresseable with centralized, hosted solutions (privacy, surveillance, encryption, ownership). All the convenience of magical self-hosted centralized software comes at a cost, some people may deem the tradeoffs not worth it.

For me, while I work on proprietary software and have for many years, I choose to spend some of my free time pretending to be an open source curmudgeon, sometimes, and want to understand what the alternatives are.

The older I get, the more it seems like RMS was right more often than I ever thought.

Money

Even if you don’t care about any of that, there’s probably viable businesses in getting open source slack alternatives up to parity and selling services or other complements around it.

Slack has hundreds of thousands of paying users and is growing — there’s hundreds of millions of dollars in annual revenue over the next few years in this space, it will be surprising to me if the open source community doesn’t capture some of it.

Three Contenders

Looking at the current landscape of Slack alternatives, I found three that I think can get you most of the elements listed above: IRC, Mattermost, and Rocket Chat.

After limited usage my take is IRC is a glorious mess, Mattermost and Rocket Chat are both incredibly impressive efforts at this stage, though my personal take is Mattermost feels a bit disappointing, and Rocket Chat shows more promise and momentum.

IRC

I’ve been playing around with IRC lately and it’s, well, IRC. It’s a beautiful mess.

Pros:

  • tons of servers
  • tons of clients
  • tons of bot and automation tools
  • tons of logging tools
  • lots and lots and lots of options for everything!

Challenges with IRC to get it to be a reasonable Slack alternative:

  • things may or may not work together
  • everything feels incomprehensible
  • very hard to integrate context / archiving / logs / search into client experiences
  • authentication is difficult
  • UX has a steep learning curve by default
  • native mobile clients — protocol works poorly on mobile, slow connection/starts
  • native push — extremely difficult to set up, non-standard
  • non-text assets — hard to handle by default, need to go beyond protocol
  • search — requires logging and external tools

If I were the CIO of a decent sized tech company that could recruit engineers, I might rather match the tens of thousands I’d spend on Slack with an internal open source effort to build out the capabilities of IRC to future proof and have full ownership of my data and processes.

(Etsy may be that company, which is awesome.)

Most of the above challenges are addressable, but it’s not trivial, and requires a lot of different kinds of work to come together (server, client, authentication integration, user experience) and by the time you’re done you end up making something that doesn’t work with most IRC software anymore.

For small teams, personal projects, and other things it feels like IRC is a lost cause at this point. Unless your audience is UNIX loving open source nerds, you’re in trouble.

Membership has been declining so it’s unlikely people are already using it - you’re probably trying to get people to use it from scratch.

(I tried and failed, but I didn’t try very hard, and I have no friends.)

Mattermost

As of this writing this piece, its github has 341 watchers, 8462 stars, 987 forks. It’s an active, impressive project.

Mattermost feels like the open source project most directly positioned to challenge Slack credibly, but my short time trying it was kind of disappointing.

For example, here’s me trying to upload an image and look at it and figure out why it doesn’t work:

I failed.

I had a lot of trouble running it (mostly because I had a bunch of legacy 32-bit system libraries, but that’s a tale for another day.)

Pros

  • written in Go — I love GoLang!
  • core feature: real time communication, archiving, search basics seem to work
  • nice platform for integrations
  • single-sign-on authentication with GitLab
  • projects a more enterprise, professional face and sells services

Cons

  • limited deployment options documented
  • felt buggy to me, almost immediately
  • overall UX feels very janky, even on web
  • authentication with GitLab but not others easily
  • doesn’t feel like a fun, vibrant community project from their online presence
  • weak native apps — wrappers around webviews lead to high latency and low usability

It’s an impressive piece of tech, but more fuzzily, Mattermost doesn’t feel like it has momentum and delight yet as a product, it feels more like enterprise software that is in development.

Rocket Chat

As of writing this piece, its github has 499 watchers 8099 stars, 1686 forks, so a similar level of activity as Mattermost.

But Rocket Chat feels like the most viable Slack open source alternative right now to me. It gets the fundamentals right of real time communication, asset sharing, and search. The community has made efforts to make it very easy to deploy in a whole lot of ways, and discussions around it seem like it has emerged organically and has a lot of passionate developers working on it.

Pros

  • mostly seems to work with little tweaking
  • lots of deployment options and one click deploys that work
  • authentication, including social authentication on Twitter
  • lots and lots of features popping in
  • feels like a lively, active community

Cons

  • some of those features seems half-finished, buggy or not well integrated
  • buggy — a key push notifications feature didn’t work
  • some docs are non-existent or hard to follow
  • platform / integration tools are in flux, not easy to drop in arbitrary bots besides Hubot
  • weak native apps — wrappers around webviews lead to high latency and low usability

Native Mobile As Leverage Point

The success and ascendance of chat applications has mirrored smartphones.

Chat is, in many ways, the ideal mobile use case and interface, and certainly one of its killer apps.

Webviews wrapped in a mobile app to handle push notifications are just not as responsive, fluid, or usable as native applications. They’re subpar experiences.

It’s going to be very hard to compete against Slack without high quality, fast, real mobile experiences.

A viable Slack open source competitor needs high quality native mobile applications that don’t exist today.

Free Advice

If I were advising one of these or other Slack alternatives, I’d encourage them to make native mobile top priority and invest in it. I think it’s the highest leverage point of any of them right now. 2

The rest (while not trivial) seems to be coming along OK. (Though there’s probably something in security and encryption that could be interesting too.)

The open source community is weaker in mobile, compared to its strength in server and web experiences (and desktop.) Given that both major mobile platforms are managed computing environments owned by giant corporations and come with lots of restrictions and strings attached, this is not surprising. But it is going to hamper adoption of new communications tools like this and let others take the market.

The next priority would be to emulate Slack’s APIs for third party integrations where possible — the webhooks and bot integrations Slack offers are very powerful and being able to re-use that ecosystem is very powerful.

· · ·

My guess is Slack understands how important their mobile clients are to their current usage and growth. If I were advising them, I’d suggest finding ways to keep the mobile experiences extremely performant (my experience is it feels like it takes a long time to get back to a chat when relaunching the app) and figure out how to stay dramatically ahead of whatever open source alternatives eventually come up. (As usual, advice is easier to give than to execute on.)

Footnotes

[ 1 ] A more interesting and perhaps more controversial piece would delve into this topic, in the context of soul-less enterprise software, but I’m trying to be positive here and if I hear one more person complain about brand x proprietary enterprise soulless software vs. brand y I may cry.

[ 2 ] I estimate a first version is doable with a 3-5 person team in 3-6 months, if that team has the right mobile expertise and scopes the project appropriately. Given Rocket Chat is built on Meteor, it may be (slightly) easier to get there by using existing software that bridges Meteor changes to CoreData. You still have to build the real UI, which isn’t trivial.)

Are There Viable Open Source Slack Alternatives?

Some are close, but I believe there will need to be serious investment in native mobile applications for the existing open source solutions to get there.

What Is Slack

Slack is incredibly popular, high growth, proprietary chat software.

You should probably try it if you haven’t already, it’s nice.

What IS Slack

It’s a little more than that. Key elements that make Slack Slack

  1. real time chat
  2. asset hosting and sharing (images, links, etc)
  3. search of text and assets
  4. understandable authentication and permissions
  5. client software for web
  6. platform for automated users (bots)
  7. native mobile clients for iOS and Android with push notifications
  8. a bunch of social norms and behaviors around talking to coworkers (with emoji and gifs)

1-5 are nearly a commodity at this point in various forms, though generally not packaged very well together as a system.

The value of 6 can mostly be replicated by competing platforms as de facto standards emerge, but hasn’t.

7 is actually a competitive differentiator for Slack right now, and is the focus of the conclusion of this piece.

It may seem like I’m mocking 8 but I’m not. Setting the cultural context and norms for communication software matters a lot more than people think, and how software makes you feel is part of why people will or will not use it. (This is something that open source software, and software generally, ignores.)

All communication and social software is about people, and how they feel. When you breathe culture and personality into software, it has an impact on that. 1

Why Use Slack

Slack is a great way to get the above with very little hassle in a hosted, integrated offering with momentum.

I like the software! It’s nice. There’s an attention to detail, polish, and craftsmanship in what they’ve made that is very admirable.

I have friends that work there, and they’re all nice too.

Why Not Slack

Cost may be an issue, though for most business usage I doubt that will be the deciding factor.

For personal usage, it seems cost prohibitive if you want to keep access to your archives and get the “full” experience.

Bigger issues that may be important to you:

  • data security and ownership
  • federation
  • long term viability of Slack Technologies Inc.
  • cost structure in long term
  • long term viability of proprietary protocols, closed source code vs. open protocols
  • lack of control over your own communications archives and destiny
  • uptime and stability
  • government surveilance, end to end encryption, privacy

You can argue that exports and backups and documentation can mitigate a lot of the above but in 2016 it feels like real time chat communications platform should be something anybody can spin up and use without hassle, and without having to deal with a giant VC funded US corporation, no matter how benevolant you think those entities may or may not be.

Much of the above I think are just fundamentally not going to be addresseable with centralized, hosted solutions (privacy, surveillance, encryption, ownership). All the convenience of magical self-hosted centralized software comes at a cost, some people may deem the tradeoffs not worth it.

For me, while I work on proprietary software and have for many years, I choose to spend some of my free time pretending to be an open source curmudgeon, sometimes, and want to understand what the alternatives are.

The older I get, the more it seems like RMS was right more often than I ever thought.

Money

Even if you don’t care about any of that, there’s probably viable businesses in getting open source slack alternatives up to parity and selling services or other complements around it.

Slack has hundreds of thousands of paying users and is growing — there’s hundreds of millions of dollars in annual revenue over the next few years in this space, it will be surprising to me if the open source community doesn’t capture some of it.

Three Contenders

Looking at the current landscape of Slack alternatives, I found three that I think can get you most of the elements listed above: IRC, Mattermost, and Rocket Chat.

After limited usage my take is IRC is a glorious mess, Mattermost and Rocket Chat are both incredibly impressive efforts at this stage, though my personal take is Mattermost feels a bit disappointing, and Rocket Chat shows more promise and momentum.

IRC

I’ve been playing around with IRC lately and it’s, well, IRC. It’s a beautiful mess.

Pros:

  • tons of servers
  • tons of clients
  • tons of bot and automation tools
  • tons of logging tools
  • lots and lots and lots of options for everything!

Challenges with IRC to get it to be a reasonable Slack alternative:

  • things may or may not work together
  • everything feels incomprehensible
  • very hard to integrate context / archiving / logs / search into client experiences
  • authentication is difficult
  • UX has a steep learning curve by default
  • native mobile clients — protocol works poorly on mobile, slow connection/starts
  • native push — extremely difficult to set up, non-standard
  • non-text assets — hard to handle by default, need to go beyond protocol
  • search — requires logging and external tools

If I were the CIO of a decent sized tech company that could recruit engineers, I might rather match the tens of thousands I’d spend on Slack with an internal open source effort to build out the capabilities of IRC to future proof and have full ownership of my data and processes.

(Etsy may be that company, which is awesome.)

Most of the above challenges are addressable, but it’s not trivial, and requires a lot of different kinds of work to come together (server, client, authentication integration, user experience) and by the time you’re done you end up making something that doesn’t work with most IRC software anymore.

For small teams, personal projects, and other things it feels like IRC is a lost cause at this point. Unless your audience is UNIX loving open source nerds, you’re in trouble.

Membership has been declining so it’s unlikely people are already using it - you’re probably trying to get people to use it from scratch.

(I tried and failed, but I didn’t try very hard, and I have no friends.)

Mattermost

As of this writing this piece, its github has 341 watchers, 8462 stars, 987 forks. It’s an active, impressive project.

Mattermost feels like the open source project most directly positioned to challenge Slack credibly, but my short time trying it was kind of disappointing.

For example, here’s me trying to upload an image and look at it and figure out why it doesn’t work:

I failed.

I had a lot of trouble running it (mostly because I had a bunch of legacy 32-bit system libraries, but that’s a tale for another day.)

Pros

  • written in Go — I love GoLang!
  • core feature: real time communication, archiving, search basics seem to work
  • nice platform for integrations
  • single-sign-on authentication with GitLab
  • projects a more enterprise, professional face and sells services

Cons

  • limited deployment options documented
  • felt buggy to me, almost immediately
  • overall UX feels very janky, even on web
  • authentication with GitLab but not others easily
  • doesn’t feel like a fun, vibrant community project from their online presence
  • weak native apps — wrappers around webviews lead to high latency and low usability

It’s an impressive piece of tech, but more fuzzily, Mattermost doesn’t feel like it has momentum and delight yet as a product, it feels more like enterprise software that is in development.

Rocket Chat

As of writing this piece, its github has 499 watchers 8099 stars, 1686 forks, so a similar level of activity as Mattermost.

But Rocket Chat feels like the most viable Slack open source alternative right now to me. It gets the fundamentals right of real time communication, asset sharing, and search. The community has made efforts to make it very easy to deploy in a whole lot of ways, and discussions around it seem like it has emerged organically and has a lot of passionate developers working on it.

Pros

  • mostly seems to work with little tweaking
  • lots of deployment options and one click deploys that work
  • authentication, including social authentication on Twitter
  • lots and lots of features popping in
  • feels like a lively, active community

Cons

  • some of those features seems half-finished, buggy or not well integrated
  • buggy — a key push notifications feature didn’t work
  • some docs are non-existent or hard to follow
  • platform / integration tools are in flux, not easy to drop in arbitrary bots besides Hubot
  • weak native apps — wrappers around webviews lead to high latency and low usability

Native Mobile As Leverage Point

The success and ascendance of chat applications has mirrored smartphones.

Chat is, in many ways, the ideal mobile use case and interface, and certainly one of its killer apps.

Webviews wrapped in a mobile app to handle push notifications are just not as responsive, fluid, or usable as native applications. They’re subpar experiences.

It’s going to be very hard to compete against Slack without high quality, fast, real mobile experiences.

A viable Slack open source competitor needs high quality native mobile applications that don’t exist today.

Free Advice

If I were advising one of these or other Slack alternatives, I’d encourage them to make native mobile top priority and invest in it. I think it’s the highest leverage point of any of them right now. 2

The rest (while not trivial) seems to be coming along OK. (Though there’s probably something in security and encryption that could be interesting too.)

The open source community is weaker in mobile, compared to its strength in server and web experiences (and desktop.) Given that both major mobile platforms are managed computing environments owned by giant corporations and come with lots of restrictions and strings attached, this is not surprising. But it is going to hamper adoption of new communications tools like this and let others take the market.

The next priority would be to emulate Slack’s APIs for third party integrations where possible — the webhooks and bot integrations Slack offers are very powerful and being able to re-use that ecosystem is very powerful.

· · ·

My guess is Slack understands how important their mobile clients are to their current usage and growth. If I were advising them, I’d suggest finding ways to keep the mobile experiences extremely performant (my experience is it feels like it takes a long time to get back to a chat when relaunching the app) and figure out how to stay dramatically ahead of whatever open source alternatives eventually come up. (As usual, advice is easier to give than to execute on.)

Footnotes

[ 1 ] A more interesting and perhaps more controversial piece would delve into this topic, in the context of soul-less enterprise software, but I’m trying to be positive here and if I hear one more person complain about brand x proprietary enterprise soulless software vs. brand y I may cry.

[ 2 ] I estimate a first version is doable with a 3-5 person team in 3-6 months, if that team has the right mobile expertise and scopes the project appropriately. Given Rocket Chat is built on Meteor, it may be (slightly) easier to get there by using existing software that bridges Meteor changes to CoreData. You still have to build the real UI, which isn’t trivial.)

Living In the Era of Infinite Computing

Yesterday I wrote about how technology at scale is itself a commodity.

So if scaled computing infrastructure and tools to use them are now commodities, what’s the complement that is becoming more valuable?

  1. collection of unique and gigantic data sets (big companiess, governments benefit)
  2. ability to do something compelling with data (data backed product innovation, data science, you should have paid more attention in statistics!)
  3. products and services that don’t rely on data (give up, go home)
  4. products and services that get worse with more data (left as an exercise to the reader)

More cynically, tracking, surveillance, or other means to continually gather and improve data sets, which is kind of terrifying.

Distribution was a critical capability — but distribution today is often just the free byproduct to a data gathering effort that makes that distribution profitable. Television and newspapers could sell access to an audience on distribution alone, they didn’t need the audience to ping back everything they did to the TV station to be stored for eternity.

As sensors and computation become cheaper, more ubiquitous and more embedded in more places, the owner and aggregation point for the data produced becomes more and more powerful.

I feel like I see a lot of people talking about consumer social media, user activity at scale and the matching ad tech behind it, but less so some of these others that seem a hell of a lot more important than targeting ads —

  • Insurance companies — health care usage and outcomes via claims, financial and claims data
  • Financial institutions — transactions, credit, asset changes at scale
  • Telcos — cellular data — as internet enabled devices and transportation adds networking they get tracking of vehicles, people, objects 24/7

…and of course

  • State-level actors — government intelligence agencies, surveillance at scale

Increasingly option (3) seems to a path worth looking at from my perspective.

Living In the Era of Infinite Computing

Yesterday I wrote about how technology at scale is itself a commodity.

So if scaled computing infrastructure and tools to use them are now commodities, what’s the complement that is becoming more valuable?

  1. collection of unique and gigantic data sets (big companiess, governments benefit)
  2. ability to do something compelling with data (data backed product innovation, data science, you should have paid more attention in statistics!)
  3. products and services that don’t rely on data (give up, go home)
  4. products and services that get worse with more data (left as an exercise to the reader)

More cynically, tracking, surveillance, or other means to continually gather and improve data sets, which is kind of terrifying.

Distribution was a critical capability — but distribution today is often just the free byproduct to a data gathering effort that makes that distribution profitable. Television and newspapers could sell access to an audience on distribution alone, they didn’t need the audience to ping back everything they did to the TV station to be stored for eternity.

As sensors and computation become cheaper, more ubiquitous and more embedded in more places, the owner and aggregation point for the data produced becomes more and more powerful.

I feel like I see a lot of people talking about consumer social media, user activity at scale and the matching ad tech behind it, but less so some of these others that seem a hell of a lot more important than targeting ads —

  • Insurance companies — health care usage and outcomes via claims, financial and claims data
  • Financial institutions — transactions, credit, asset changes at scale
  • Telcos — cellular data — as internet enabled devices and transportation adds networking they get tracking of vehicles, people, objects 24/7

…and of course

  • State-level actors — government intelligence agencies, surveillance at scale

Increasingly option (3) seems to a path worth looking at from my perspective.

Technology At Scale is A Commodity

One explanation for Google’s early success is that it was derived from the application of computer science at scale. This scale was not just a difficult technical achievement fueled by incredibly smart technologists, but so difficult that competitors didn’t pursue similar efforts due to difficulty, uncertainty, and cost until they were proven out.

Google crawled the web faster, then did complex graph analysis on it nobody else did or even could do, creating a better search experience.

Gmail provided effectively infinite storage for email — a gigabyte — at a time when competitors were limiting storage to an amount that required user attention and deleting — single or double digit megabytes.

Moore’s law may slow as we reach physical limits, but the unit economics of things like computation, memory, and storage fall, and fall faster at scale. So as you grow Gmail, unit costs decrease as usage increases and technological advances carry you ahead.

This isn’t a long term defensible play in and of itself: competitors can swoop in and realize these same cost advantages, as the big webmail providers did quickly.

But now, you don’t have to be a giant company to make that play.

Scale at Scale

Amazon Web Services (and the similar efforts from IBM and others) have made computing power at scale itself a commodity. Tiny startups have access to it with minimal capital, and they can realize much of the advantages of huge scale before hitting it themselves.

Companies like Google could develop innovations like Gmail by asking questions - what if storage was so cheap users never had to delete an email? And then build out the solution to deliver it.

Crazy Technology Ideas Are Sensible Now

Probably the coolest feature I ever product managed at Google started with a conversation that went something like

“Hey, this is computationally infeasible but could we brute force the linking between books via quotes? Hash every sentence or n-gram and compare?”

“Well, that sounds computationally impossible. So no, not that way. That’s well, that’s not going to happen. But, yeah, maybe.

Then brilliant research scientists went off and wrote ground breaking software , comfortable in the fact that they had access to nearly infinite computing resources to run giant mapreduces on to solve it.

I don’t want to discount the high cost of complexity and technical achievement in something like popular passages or similar work — just that getting great engineers to solve problems is not the scarce resource. If you want to do something crazy today, the gating factor is not owning a data center, or having access to tools to handle large amounts of data, or getting access to smart engineers.

Open Source

While AWS has made scaled computing a commodity, open source software increasingly makes complex tasks like machine learning more accessible to more people and cuts their costs too.

It makes sense for Google, Amazon IBM and others to invest and distribute these tools — they drive additional usage of the scaled computing platforms.

While hiring is difficult we have plenty of brilliant people who will jump to work on interesting giant problems (with enough compensation.)

The cost of scaled computing continues to drop and become more accessible, and the tools to leverage it become cheaper and cheaper.

Conclusion:

  1. companies won’t win on scaling infrastructure
  2. much of groundbreaking computer science 10 years ago is now a commodity
  3. a lot of what we think of as groundbreaking CS today will be a commodity in 10 years

Technology At Scale is A Commodity

One explanation for Google’s early success is that it was derived from the application of computer science at scale. This scale was not just a difficult technical achievement fueled by incredibly smart technologists, but so difficult that competitors didn’t pursue similar efforts due to difficulty, uncertainty, and cost until they were proven out.

Google crawled the web faster, then did complex graph analysis on it nobody else did or even could do, creating a better search experience.

Gmail provided effectively infinite storage for email — a gigabyte — at a time when competitors were limiting storage to an amount that required user attention and deleting — single or double digit megabytes.

Moore’s law may slow as we reach physical limits, but the unit economics of things like computation, memory, and storage fall, and fall faster at scale. So as you grow Gmail, unit costs decrease as usage increases and technological advances carry you ahead.

This isn’t a long term defensible play in and of itself: competitors can swoop in and realize these same cost advantages, as the big webmail providers did quickly.

But now, you don’t have to be a giant company to make that play.

Scale at Scale

Amazon Web Services (and the similar efforts from IBM and others) have made computing power at scale itself a commodity. Tiny startups have access to it with minimal capital, and they can realize much of the advantages of huge scale before hitting it themselves.

Companies like Google could develop innovations like Gmail by asking questions - what if storage was so cheap users never had to delete an email? And then build out the solution to deliver it.

Crazy Technology Ideas Are Sensible Now

Probably the coolest feature I ever product managed at Google started with a conversation that went something like

“Hey, this is computationally infeasible but could we brute force the linking between books via quotes? Hash every sentence or n-gram and compare?”

“Well, that sounds computationally impossible. So no, not that way. That’s well, that’s not going to happen. But, yeah, maybe.

Then brilliant research scientists went off and wrote ground breaking software , comfortable in the fact that they had access to nearly infinite computing resources to run giant mapreduces on to solve it.

I don’t want to discount the high cost of complexity and technical achievement in something like popular passages or similar work — just that getting great engineers to solve problems is not the scarce resource. If you want to do something crazy today, the gating factor is not owning a data center, or having access to tools to handle large amounts of data, or getting access to smart engineers.

Open Source

While AWS has made scaled computing a commodity, open source software increasingly makes complex tasks like machine learning more accessible to more people and cuts their costs too.

It makes sense for Google, Amazon IBM and others to invest and distribute these tools — they drive additional usage of the scaled computing platforms.

While hiring is difficult we have plenty of brilliant people who will jump to work on interesting giant problems (with enough compensation.)

The cost of scaled computing continues to drop and become more accessible, and the tools to leverage it become cheaper and cheaper.

Conclusion:

  1. companies won’t win on scaling infrastructure
  2. much of groundbreaking computer science 10 years ago is now a commodity
  3. a lot of what we think of as groundbreaking CS today will be a commodity in 10 years

Try Shorter Forms

“What IRC thing?”

“Well, clearly someone doesn’t read my web site.

“Oh was that the most recent thing? That was… too long.”

Try Shorter Forms

“What IRC thing?”

“Well, clearly someone doesn’t read my web site.

“Oh was that the most recent thing? That was… too long.”

Post-modern IRC

The last time I regularly used IRC was sometime in college to keep up with the underground classic arcade and console game piracy/preservation community.

I didn’t pay much attention to IRC itself — it was just an interesting way to communicate with a weird subculture I was on the periphery of.

Now that I’m looking at IRC with fresh eyes as a technologist and conscientious objector of modern social media, all the wondrous beauty, potential, incomprehensibility and flaws seem all the more apparent.

Protocol

Unlike proprietary modern tightly coupled solutions, IRC is a protocol with an RFC from 25 years ago. So it runs everywhere, there’s clients for every platform, servers galore, and whole ecosystem of related software.

There’s a certain durability and comfort to these kinds of standards running on commodity hardware with an open source operating system. You can be pretty sure if you set some unix server to run IRC, you’ll be able to keep using IRC in some form years from now.

I love boring software now in a way I never thought I would.

If you choose some hot shit modern chat app thing, who knows where it will be in a few years. (On that note, does anybody still use XMPP? Wasn’t it funny that in the late 90’s when using XML for everything seemed like a modern and awesome future-proofing idea?! We’ll probably feel that way about Javascript in 20 years.)

Usability Nightmare

Unlike proprietary modern tightly coupled solutions, IRC is an old school protocol where everything is an incomprehensible nightmare of disparate configuration files, arcane commands, usability nightmares, security flaws, and duct tape and chewing gum to keep it all together.

It’s great though. Really. Really? Really. Maybe.

Missing Pieces

Ignoring the general incomprehensibility of IRC and the vocabulary that is required for a user, setting up a private IRC server actually isn’t that bad as far as services go. At its core, typing in a box to send a message is more or less something that makes intuitive sense once you get people there.

(That was one of the only things that people could figure out how to do on Orkut and social networks.)

Some of the challenges with IRC —

  • authentication and identity — IRC has always had a weird hodgepodge of “services” built on top to try and handle identity to fill the gaps but it feels particularly awkward now in an age of single-sign on services
  • persistence — the ephemeral nature of IRC is kind of wonderful, but makes it hard to keep up with things, and the weird world of bouncers and loggers to fill in are not super easy to comprehend or deal with it
  • mobile — native mobile clients for IRC exist but the protocol (and the lack of push notifications easily available) make using IRC as a modern messaging platform on the dominant platforms challenging to normal human beings who aren’t in front of a keyboard all day

Don’t Let That Stop You

Never one to let usability, security, or sanity stop me in technical endeavors for antisocial software, I now have a setup that includes:

  • IRC daemonngIRCd set up to only allow connections from my server
  • preconfigured web client that connects to it for easy access — Lounge
  • loggerLogBot
  • bouncer to enable persistent connection for myself and BFFs — znc
  • Mac clientLimeChat
  • iOS clientMutter with push notifications
  • no friends actually using it

I declare that a huge success, personally.

Anyway, give it a shot. Relive the glory days of the trenchant.org webarrific discussioney board but in realtime over IRC at irc.trenchant.org

Or don’t. I’ll probably just ignore whatever happens on there anyway, and regret that I just made a thing that allows anyone to anonymously send me push notifications.

TODO

Ppersistence/login working seamlessly on a web version without a complicated bouncer setup would be nice.

Figuring out a way to integrate images/image hosting.

IRC bouncers are great but the multiple level of authentication / logins / etc needed to get all this to work is pretty daunting — something that handled all that seamlessly would be interesting.

Bots. There should probably an adammathebooks bot on the channel at all times, and really a host menagerie of bot pals since the internet for me has always fundamentally been about me talking to myself in various ways anyway.

At what point do I just give up and decide we all should just use the same UNIX server to talk to each other.

I feel like I’m inching ever and ever closer to just throwing everything out the window and trying to find a way to live in hacked up Raspberry Pi’s.

Post-modern IRC

The last time I regularly used IRC was sometime in college to keep up with the underground classic arcade and console game piracy/preservation community.

I didn’t pay much attention to IRC itself — it was just an interesting way to communicate with a weird subculture I was on the periphery of.

Now that I’m looking at IRC with fresh eyes as a technologist and conscientious objector of modern social media, all the wondrous beauty, potential, incomprehensibility and flaws seem all the more apparent.

Protocol

Unlike proprietary modern tightly coupled solutions, IRC is a protocol with an RFC from 25 years ago. So it runs everywhere, there’s clients for every platform, servers galore, and whole ecosystem of related software.

There’s a certain durability and comfort to these kinds of standards running on commodity hardware with an open source operating system. You can be pretty sure if you set some unix server to run IRC, you’ll be able to keep using IRC in some form years from now.

I love boring software now in a way I never thought I would.

If you choose some hot shit modern chat app thing, who knows where it will be in a few years. (On that note, does anybody still use XMPP? Wasn’t it funny that in the late 90’s when using XML for everything seemed like a modern and awesome future-proofing idea?! We’ll probably feel that way about Javascript in 20 years.)

Usability Nightmare

Unlike proprietary modern tightly coupled solutions, IRC is an old school protocol where everything is an incomprehensible nightmare of disparate configuration files, arcane commands, usability nightmares, security flaws, and duct tape and chewing gum to keep it all together.

It’s great though. Really. Really? Really. Maybe.

Missing Pieces

Ignoring the general incomprehensibility of IRC and the vocabulary that is required for a user, setting up a private IRC server actually isn’t that bad as far as services go. At its core, typing in a box to send a message is more or less something that makes intuitive sense once you get people there.

(That was one of the only things that people could figure out how to do on Orkut and social networks.)

Some of the challenges with IRC —

  • authentication and identity — IRC has always had a weird hodgepodge of “services” built on top to try and handle identity to fill the gaps but it feels particularly awkward now in an age of single-sign on services
  • persistence — the ephemeral nature of IRC is kind of wonderful, but makes it hard to keep up with things, and the weird world of bouncers and loggers to fill in are not super easy to comprehend or deal with it
  • mobile — native mobile clients for IRC exist but the protocol (and the lack of push notifications easily available) make using IRC as a modern messaging platform on the dominant platforms challenging to normal human beings who aren’t in front of a keyboard all day

Don’t Let That Stop You

Never one to let usability, security, or sanity stop me in technical endeavors for antisocial software, I now have a setup that includes:

  • IRC daemonngIRCd set up to only allow connections from my server
  • preconfigured web client that connects to it for easy access — Lounge
  • loggerLogBot
  • bouncer to enable persistent connection for myself and BFFs — znc
  • Mac clientLimeChat
  • iOS clientMutter with push notifications
  • no friends actually using it

I declare that a huge success, personally.

Anyway, give it a shot. Relive the glory days of the trenchant.org webarrific discussioney board but in realtime over IRC at irc.trenchant.org

Or don’t. I’ll probably just ignore whatever happens on there anyway, and regret that I just made a thing that allows anyone to anonymously send me push notifications.

TODO

Ppersistence/login working seamlessly on a web version without a complicated bouncer setup would be nice.

Figuring out a way to integrate images/image hosting.

IRC bouncers are great but the multiple level of authentication / logins / etc needed to get all this to work is pretty daunting — something that handled all that seamlessly would be interesting.

Bots. There should probably an adammathebooks bot on the channel at all times, and really a host menagerie of bot pals since the internet for me has always fundamentally been about me talking to myself in various ways anyway.

At what point do I just give up and decide we all should just use the same UNIX server to talk to each other.

I feel like I’m inching ever and ever closer to just throwing everything out the window and trying to find a way to live in hacked up Raspberry Pi’s.

Operators Are Standing By

In the absence of twitter — the last social media site I used with any regularity — I’ve been thinking a lot more about centralization of communication, publishing, and its impacts.

Orange XML Buttons Were Ugly And Unusuable But Now All We Have Are Embedded Tweets

The death of RSS as a platform for decentralized notifications and re-engagement of audience has had a crippling effect on the ability of publishers (small and large) to keep audiences coming back. Pretty soon I will need to sell a Winer was right t-shirt next to the RMS one.

I apparently realized this six years ago when one of my first post-Google projects was mailmedaily — a simple RSS to email subscription toolset. But I didn’t realize how drastic the changes would be over the next few years.

Email is the last open protocol with notifications that people actually rely on, so it makes sense to retreat and retrench there.

Like most of my projects, this one failed to get traction and I shut it down due to other things that required my time, even though it did have a tiny active user base. (Also, running email services and avoiding being marked as spam is really, really hard.)

I probably should have thought of publishers as the customers rather than readers, and figured out how to make that work, but I think that market is pretty crowded and I wasn’t interested in solving that problem at the time — only scratching my own itch of subscriptions for myself and readers. But there’s something important to this one — re-engagement is actually the longer term problem to solve rather than mass audience. Mass audiences may live on social media, but trying to be a personal publisher within that cesspool seems like a Faustian bargain.

Well, that seems unfair — at least the devil felt obligated to do cool shit for Faust for 24 years in exchange for his soul — nobody will give you that sort of value for your soul in an algorithmically optimized feed!

Anyway, I should be thinking about how to deal with this sort of thing if I’m regularly writing again and want an audience, but luckily I do not care about the size of my audience anymore. (I mean, I care in the sense that I’m probably better off if it’s smaller.)

The Other Forgotten Text Protocol

In the near term I’ll just let my weird custom CMS tweet out my writing and have it do battle against the auto-delete bot because that seems like the most entertaining approach. (I was going to do that on Medium too but it seems too complicated to get my Medium posts auto-deleted.)

I’m also setting up a private IRC server. Which seems ridiculous that I’d even consider when I’m too lazy/weird/whatever to even respond to any of the personal inquiries I get and am avoiding all social media but it makes sense in my head.

Web version is available to play with if I haven’t messed it up yet. I may be unqualified to manage UNIX services but in the modern era everybody is dev-ops whether they like it or not.

Operators Are Standing By

In the absence of twitter — the last social media site I used with any regularity — I’ve been thinking a lot more about centralization of communication, publishing, and its impacts.

Orange XML Buttons Were Ugly And Unusuable But Now All We Have Are Embedded Tweets

The death of RSS as a platform for decentralized notifications and re-engagement of audience has had a crippling effect on the ability of publishers (small and large) to keep audiences coming back. Pretty soon I will need to sell a Winer was right t-shirt next to the RMS one.

I apparently realized this six years ago when one of my first post-Google projects was mailmedaily — a simple RSS to email subscription toolset. But I didn’t realize how drastic the changes would be over the next few years.

Email is the last open protocol with notifications that people actually rely on, so it makes sense to retreat and retrench there.

Like most of my projects, this one failed to get traction and I shut it down due to other things that required my time, even though it did have a tiny active user base. (Also, running email services and avoiding being marked as spam is really, really hard.)

I probably should have thought of publishers as the customers rather than readers, and figured out how to make that work, but I think that market is pretty crowded and I wasn’t interested in solving that problem at the time — only scratching my own itch of subscriptions for myself and readers. But there’s something important to this one — re-engagement is actually the longer term problem to solve rather than mass audience. Mass audiences may live on social media, but trying to be a personal publisher within that cesspool seems like a Faustian bargain.

Well, that seems unfair — at least the devil felt obligated to do cool shit for Faust for 24 years in exchange for his soul — nobody will give you that sort of value for your soul in an algorithmically optimized feed!

Anyway, I should be thinking about how to deal with this sort of thing if I’m regularly writing again and want an audience, but luckily I do not care about the size of my audience anymore. (I mean, I care in the sense that I’m probably better off if it’s smaller.)

The Other Forgotten Text Protocol

In the near term I’ll just let my weird custom CMS tweet out my writing and have it do battle against the auto-delete bot because that seems like the most entertaining approach. (I was going to do that on Medium too but it seems too complicated to get my Medium posts auto-deleted.)

I’m also setting up a private IRC server. Which seems ridiculous that I’d even consider when I’m too lazy/weird/whatever to even respond to any of the personal inquiries I get and am avoiding all social media but it makes sense in my head.

Web version is available to play with if I haven’t messed it up yet. I may be unqualified to manage UNIX services but in the modern era everybody is dev-ops whether they like it or not.

Globalization and The Economics of Fear

More importantly still — and directly contrary to what establishment liberals love to claim in order to demonize all who reject their authority — economic suffering and xenophobia/racism are not mutually exclusive. The opposite is true: The former fuels the latter, as sustained economic misery makes people more receptive to tribalistic scapegoating. That’s precisely why plutocratic policies that deprive huge portions of the population of basic opportunity and hope are so dangerous. Claiming that supporters of Brexit or Trump or Corbyn or Sanders or anti-establishment European parties on the left and right are motivated only by hatred but not genuine economic suffering and political oppression is a transparent tactic for exonerating status quo institutions and evading responsibility for doing anything about their core corruption.

Glenn Greenwald, Brexit Is Only the Latest Proof of the Insularity and Failure of Western Establishment Institutions

Globalization, the frictionless flow of capital, labor, and ideas has been great for a lot of people — including myself, I work in tech in silicon valley! — but it’s had devastating impacts elsewhere, without key political institutions taking them seriously or addressing them.

This combined with the very large increases in wealth inequality over the past four decades are having a cumulative effect. Blaming “the other” is unfortunately a historically effective tactic for channeling that anger — and in the absence of better ideas to address the root causes and effective policy it’s likely to become even more effective and damaging to western culture.

I am perhaps less surprised by all this since I spent my teenage years reading Thomas Frank articles in The Baffler that mostly seemed shocked this sort of thing wasn’t happening sooner, and how weird it was that populist tactics had been co-opted by plutocratic establishment figures so easily.

Meanwhile In Neo-Liberal Utopia, Nobody Can Afford To Live

While those like myself in the cosmopolitan California peninsula may claim to be beyond the reach of such base tactics, ask your liberal friends in CA (the ones that are renting and complaining about housing costs and threatening to leave for Portland) the following thought experiment —

Would you support a candidate that promised to decrease housing costs 25% by banning foreign investors from residential real estate purchases?

Blaming foreign money for the Bay Area housing debacle — when it’s statistically pretty much the fault of increasing wages, geography preventing sprawl, and mostly a decades long refusal of local governments to increase housing supply, density, and infrastructure to meet increased demand — is the kind of thing I have heard regularly over the past couple years. And while there may be some shred of truth to some tiny bit of it, it’s more indicative of the potential for xenophobic policies to ignite even where people least expect when there’s just the slightest economic pain and lack of security (even amongst the well off) to fuel the fire.

Expect The Unexpected

Bernie Sanders — a 74-year-old socailist who until recently wasn’t even in the Democratic party — led a campaign that nearly beat the establishment candidate.

Donald Trump — a reality television actor and living lifestyle brand of questionable products and services — did beat the establishment candidates on the Republican side, somehow.

Clinton and Trump have historically high negative favorability ratings so we are in uncharted territory in US politics for a lot of reasons.

Regardless of what happens in this election cycle — seems hard to fathom Clinton losing given polling, demographics, and Trump’s inability to stop saying crazy things — the economic stagnation, voter sentiment, and other indications mean we should expect to see more outcomes that are at odds with conventional wisdom and establishment predictions soon.

Globalization and The Economics of Fear

More importantly still — and directly contrary to what establishment liberals love to claim in order to demonize all who reject their authority — economic suffering and xenophobia/racism are not mutually exclusive. The opposite is true: The former fuels the latter, as sustained economic misery makes people more receptive to tribalistic scapegoating. That’s precisely why plutocratic policies that deprive huge portions of the population of basic opportunity and hope are so dangerous. Claiming that supporters of Brexit or Trump or Corbyn or Sanders or anti-establishment European parties on the left and right are motivated only by hatred but not genuine economic suffering and political oppression is a transparent tactic for exonerating status quo institutions and evading responsibility for doing anything about their core corruption.

Glenn Greenwald, Brexit Is Only the Latest Proof of the Insularity and Failure of Western Establishment Institutions

Globalization, the frictionless flow of capital, labor, and ideas has been great for a lot of people — including myself, I work in tech in silicon valley! — but it’s had devastating impacts elsewhere, without key political institutions taking them seriously or addressing them.

This combined with the very large increases in wealth inequality over the past four decades are having a cumulative effect. Blaming “the other” is unfortunately a historically effective tactic for channeling that anger — and in the absence of better ideas to address the root causes and effective policy it’s likely to become even more effective and damaging to western culture.

I am perhaps less surprised by all this since I spent my teenage years reading Thomas Frank articles in The Baffler that mostly seemed shocked this sort of thing wasn’t happening sooner, and how weird it was that populist tactics had been co-opted by plutocratic establishment figures so easily.

Meanwhile In Neo-Liberal Utopia, Nobody Can Afford To Live

While those like myself in the cosmopolitan California peninsula may claim to be beyond the reach of such base tactics, ask your liberal friends in CA (the ones that are renting and complaining about housing costs and threatening to leave for Portland) the following thought experiment —

Would you support a candidate that promised to decrease housing costs 25% by banning foreign investors from residential real estate purchases?

Blaming foreign money for the Bay Area housing debacle — when it’s statistically pretty much the fault of increasing wages, geography preventing sprawl, and mostly a decades long refusal of local governments to increase housing supply, density, and infrastructure to meet increased demand — is the kind of thing I have heard regularly over the past couple years. And while there may be some shred of truth to some tiny bit of it, it’s more indicative of the potential for xenophobic policies to ignite even where people least expect when there’s just the slightest economic pain and lack of security (even amongst the well off) to fuel the fire.

Expect The Unexpected

Bernie Sanders — a 74-year-old socailist who until recently wasn’t even in the Democratic party — led a campaign that nearly beat the establishment candidate.

Donald Trump — a reality television actor and living lifestyle brand of questionable products and services — did beat the establishment candidates on the Republican side, somehow.

Clinton and Trump have historically high negative favorability ratings so we are in uncharted territory in US politics for a lot of reasons.

Regardless of what happens in this election cycle — seems hard to fathom Clinton losing given polling, demographics, and Trump’s inability to stop saying crazy things — the economic stagnation, voter sentiment, and other indications mean we should expect to see more outcomes that are at odds with conventional wisdom and establishment predictions soon.

Unfollow Everything

Quitting twitter in 2016 is less an act of courage than one of desperation.

You Might Like: Never Coming Back To This Desolate Hellscape

The tweeting of “delete your account” — the equivalent of screaming “kill yourself” at a schoolyard bully — by a woman who rose to national attention by being married to a president who cheated on her to such an extent it caused a constitutional crisis and is now running for that same office of president based on a campaign predicated on having better judgment than an orange reality television actor and peddler of comical get rich quick real estate schemes — may be the defining awful moment of the toxic intersection of addiction-focused media, ultra-polarizing politics, and my inability to even know what to think.

The optimist in me wants to believe that will be the defining moment, but the pragmatist in me knows it’s just going to get worse.

So I give up.

Also, you can’t write incomprehensible run-on sentence-paragraphs like that on Twitter without compromising them. So why even bother writing at all.

If You Quit Twitter And Don’t Tweet About It Are You Counted as A Monthly Active User

After decades of technology and media eradicating quiet moments and filling them with content that constantly assault us for attention we finally succeeded in the war against boredom.

Boredom is dead.

I’ve spent nearly a decade curating my Twitter feed but I’m sure there’s plenty of completely valid data science that indicates people are less effective at choosing what to see in terms of optimizing their own engagement.

Monetizing all this freely supplied content with advertising that tries even harder to get our attention means that social media systems like Twitter and Facebook are fundamentally architected and designed to increase “engagement” and “retention” — which is mostly just a euphemism for “addiction.”

Which means I have to see things like a viral Hillary Clinton tweet because statistically speaking I’m more likely to engage even though for me personally my fight or flight instinct has finally kicked in and you can’t fight social media. You can only run away.

We fixed the boredom problem. Now we have a misery problem.

I feel worse after using Twitter. So I just have to stop.

The internet — and specifically social media — in 2016 feels like an overflowing toxic sludge of misery built on social envy, isolation, and the taxoplasma of rage.

Day 1

I deleted Twitter from my iPhone.

I deleted Tweetbot from my iPhone.

Day 2

I deleted Twitter from my iPad.

I deleted Tweetbot from my iPad.

Day 3

I logged out of Twitter on my personal computers.

Day 4

I’m writing this. It’s probably too over the top. Tone it down man, geez.

And why didn’t I bother to carefully craft a pseudonymous following in the past 20 years for this sort of thing? Total lack of forethought in my teenage years.

Design For Inattention

What we consume in our media diet matters. What we create matters. Where we put it matters. How we interact maters.

How do we break out of this cycle of social media hearts and stars rage-fueled engagement optimized algorithmic garbage?

I don’t know, but I guess step one is admitting we have a problem.