Process events before they're ingested (via processEvent). Just before we dig into the architecture, it's important to get a briefing on key plugin concepts as well as plugin capabilities in PostHog. runEveryMinute), and "async handlers" (webhooks, onEvent). The three key areas it covers are plugin jobs, scheduled tasks (e.g. Lastly, the "async" server handles everything the "ingestion" server doesn't. However, any additional processing triggered through running that function is not processed by the "ingestion" server. The one bit of processing it does for plugins is handling the processEvent function - the function allowing events to be modified pre-ingestion. The "ingestion" mode handles ingestion-related processing almost exclusively. It might eventually disappear, but for now it greatly simplifies dev setups, as well as the setup for small/medium self-hosted instances. The "default" mode follows the old architecture - one server doing everything. The plugin server can now be run in three modes - "default", "ingestion", and "async". It was time to split the service into two. We could keep on adding checks - but ultimately the best solution here was a fundamental refactor. The plugin server has various mechanisms in place to prevent things like plugins running for too long, but ultimately the number of plugins that can run is unbounded, and they can trigger a great deal of tasks and jobs that can end up hogging resources meant for ingestion. This is a poor design from an architectural perspective - plugins, meant to "extend" the platform, can cause problems for event ingestion, a core aspect of the PostHog product. Yet, because of that first use case (processing events in flight), ingestion and all this extra processing are tightly coupled together. They can access storage, trigger jobs, run scheduled tasks, process events in flight, access events after ingestion, and a lot more. Over a year passes, and now PostHog plugins are more powerful than ever. The first implementation of plugins was built directly into our core Django server, but we found the JavaScript ecosystem to be more mature when it came to running arbitrary code in a sandboxed environment.Īs a result, a new service was born: the plugin server.įast forward a bit, and we decided that it seemed overly complex and inefficient to do a round trip while ingesting an event (Django -> Plugin Server -> Django -> Database), so we kept the endpoint for sending events in Django, and moved all the other logic pertaining to ingestion over to the plugin server. Now, while this may sound like irrelevant context - it set the stage for many architectural decisions to come. Much like CDPs offer "transformations", the goal was to be able to provide a way for PostHog users to enrich events with additional data from other sources.įrom this hackathon, our most widely used plugin to date was born - GeoIP.Īn initial step towards the ultimate goal of displaying a "world map" in PostHog (solved 2 offsites later in Iceland) - the GeoIP plugin took the IP from an event and used that to enrich the event properties with location data (country, city, etc). The plugin server was born out of a hackathon in Italy during the first ever PostHog offsite.īack then, the project had one goal: transforming events in flight. Note that these are the responsibilities covered by the plugin server codebase, but not necessarily by the same service. Handling all updates to the various models events can trigger changes to (e.g.Validating, parsing, and ingesting events.What does the plugin server do?īefore getting into the hows and the whys, it's important to understand what the plugin server is responsible for. So buckle up, the journey is about to start. This doc should give you a sense of what the plugin server is, how it operates, why it works this way, as well as a better sense of how to operate it in production. The plugin server is the core of event ingestion and the plugins platform at PostHog.ĭespite being called the "plugin server", it is a core service that PostHog cannot function without.
0 Comments
Leave a Reply. |