Rendered at 00:03:03 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
brianjlogan 5 hours ago [-]
> A syslog server is a foundational tool for centralized log management in modern IT environments
I'd very much recommend a more modern log stack than a traditional syslog server.
There are many articles covering the limitations of Syslog. Better to replace that component by utilities like OpenTelemetry and JSON structured logging.
Or use something like the Otel collector to send your logs to a remote host.
I have done my fair share of rsyslog and syslog-ng.
I would not say a "Syslog" server belongs in a modern stack.
skullone 5 hours ago [-]
I disagree. I work in a large environment, and rsyslog is where 90% of data goes to first. It can keep up with millions of messages per second, route them to higher order services for indexing (bigquery, splunk, elastic etc etc). Has rules engines, encryption, supports multiple protocols and obviously has TLS too. You can surely augment with otel and such where you can, but syslog is uhhhh, deployed in so many places that it would make an average app developer's head spin when all they're used to is application logging in a controlled structured place in their silo.
otterley 2 hours ago [-]
On the other hand, traditional syslog is UDP based, so as soon as the receiver experiences CPU or I/O starvation and its receive buffer overflows, it will begin dropping messages. That's not great for observability, and may well be impermissible at many sites that need end-to-end log integrity (e.g. audit logs).
xorcist 1 hours ago [-]
Thirty years ago and more that might have been a valid objection, but at that time the alternatives weren't great either. No one has suggested running syslog over unreliable transport after that.
In fact, the queue management and at least the possibility of some rudimentary end-to-end cryptographic integrity checks are some of the stronger points of rsyslog. Splunk Cloud and Elastic, as far as I know, lacks the latter completely which rules them out as a single log sink for environments with that type of requirements.
otterley 1 hours ago [-]
Which major Linux distro ships rsyslog with TCP as the default remote protocol and durable local-buffer configuration out of the box for remote delivery? Genuinely curious.
"Beginner tutorial", I'm sorry for not taking your point seriously, but I can't take it seriously. TCP for syslog (and RELP) have been around a long time (late 90s for syslog over TCP, 2006 for RELP).
rsyslog and syslog-ng support it all, and operators have had choices given the import of their log data and what they can tolerate.
otterley 43 minutes ago [-]
> "Beginner tutorial", I'm sorry for not taking your point seriously, but I can't take it seriously.
Well, maybe go observe how a broad array of sites implement it in practice, then you might take it more seriously. Maybe you don't implement it that way, but a lot of people will just follow the tutorials or shortcut their way to something that works (but is brittle).
At any rate, I was responding directly to the claim that "No one has suggested running syslog over unreliable transport" which is obviously untrue.
skullone 28 minutes ago [-]
Yes, go observe a broad array of sites - for someone who says theyre a (non)practicing attorney, you know in environments where logs and audit are considered evidence, to such a degree that they must be reliably transported and immunutable, someone doesn't just turn on UDP syslog to a box and let it sit there. Architecture and implementation happen, where it matters. So what if anyone uses otel or syslog, people can configure em both to be lossy or lossless, I struggle to understand the "gotchas" you point to.
otterley 26 minutes ago [-]
Then I don't know what to tell you. *shrug* I feel like you're arguing just for argument's sake, and I'm not really interested in having a conversation with someone who's not demonstrating open-mindedness or a willingness to learn from others' experiences in the field.
lanstin 2 hours ago [-]
cheaply dropping log msgs you cannot handle is absolutely essential for an observability system - otherwise excess load can take down the logging infra which can (if msgs aren't dropped) take down the prod network/app trying to send reliable log msgs.
Audit logs are a distinct feature.
otterley 2 hours ago [-]
That's fine if you plan for it and can clearly delineate which logs can be dropped and which can't. The challenge is that most applications log to a single stream that consists of both high and low-priority logs muxed together and is sent to a single destination, making it impossible to distinguish the two, and there's only one receive buffer.
Excess load can't take down a well-engineered log collection infrastructure. There can be overload, but the backpressure should propagate downstream and senders and intermediaries should buffer locally if needed. Once the collectors are able to catch up again, the spooled messages will be dispatched, and the backlog should recover.
A well-engineered logging system for sites that care about integrity and durability should look a lot like a distributed message queue.
> Audit logs are a distinct feature.
In my experience, this is not always as distinct as one might hope. On multiple occasions in my career, a customer demanded we perform research using our logs to answer, and the information they sought were not in the class of logs that were considered "audit logs" in advance. Everyone chooses differently what qualifies as "audit logs"; it doesn't have an objective definition.
skullone 57 minutes ago [-]
If you are doing massive log streams with mixed priority/durability, use named queues if you have different priority and retry and durability needs. You keep bringing up edge cases, but they've already been considered and solutions already engineered and available in every major syslog implementation.
otterley 40 minutes ago [-]
I didn't say they those choices aren't viable; I'm strictly talking about architectural decisions. You can solve the problem with different solutions, be they rsyslog or otherwise. That said, I probably wouldn't go with rsyslog as my default choice anymore since the world is moving on to OpenTelemetry.
skullone 2 hours ago [-]
You can use TCP and other mechanisms for guaranteed delivery. Like, almost all LIDR logging across the planet uses TCP syslog, which is durable and attestable to in courts.
otterley 19 minutes ago [-]
> TCP syslog...which is durable and attestable to in courts.
TCP alone won't get you there. It's certainly not durable in and of itself. All TCP can do is ensure that streamed data is received in the correct order, and confirm that a segment's data was successfully placed into the right buffer on the receiver side. You also need immutable storage, stronger integrity checks than what TCP itself provides, and many other requirements I haven't researched in a while.
shmoe 4 hours ago [-]
Even SC4S, splunk's docker appliance for turnkey syslog uses rsyslogd.
Edit: being pedantic -- it's syslog-ng actually.
skullone 3 hours ago [-]
And the number of k8s envs that log stdout through them into.... more rsyslog, it's truly everywhere. Plus all the sidecar containers deployed that shuffle app logs, lots of syslog there, its so lightweight and simple and reliable. I watch all the gyrations people go through to achieve the same result, and it's always changing, hurts my brain thinking how much time they waste
1970-01-01 4 hours ago [-]
The problem with modern stuff is it doesn't do the very basics. Sometimes I really do want UDP dumping out into a file on another part of the network. The modern setups forget how to do this.
edoceo 4 hours ago [-]
I saw one place that had the logs going into a database. On the same connection as the app-data. So, when the transaction failed, the logs also didn't get written. LMAO. I made them do syslog in their code, which for some of the devs was a mind-blowing. They were amazed at that we could just barf text quick&lightweight over UDP.
throw0101a 2 hours ago [-]
> I'd very much recommend a more modern log stack than a traditional syslog server.
Once everything is sent to the syslog server it can be bounced to whatever 'non-traditional' stack you want.
My firewalls, PDUs, rear-door heat exchangers, etc, do not talk (nor can they run) "Loki". Just about everything in existence can (a) talk syslog and/or (b) send SNMP traps.
xorcist 1 hours ago [-]
That recommendation is hard to understand. Unless you have some completely trivial environment you will have syslog in it. It is not something you choose.
What you can do is have syslog ingestion for your Loki/Elastic/Splunk/whatever you use. But it will not magically structure your logs for you beyond the standard syslog date, host, severity, system, line of text format.
Both syslog products you mention are solid and mature, and absolutely has an important place in any modern environment. They manage queues, throttle, guarantee delivery and support any backend you can wish for.
Their respective rule engines are fast. They do log ingestion, and they slice data into fields and can enrich it (for example by resolving dns). This means rules are set in advance, and traps on data can be triggered immediately (not subject to races).
aftbit 3 hours ago [-]
Disagree. JSON logging is fine, but make it line based and just log to a syslog server. Then I can route it wherever I want (or to multiple places) including to a simple file on a disk that I can actually inspect, rather than having to use an API.
Exoristos 3 hours ago [-]
I've been down this road many times over the years, and each new promised land of logging always fails to replace syslog for me. Nothing else is as widely-compatible and performant. Fortunately, there are some great tools out there to modernize the network-admin experience: syslog-ng is a favorite of mine.
lokar 5 hours ago [-]
Yeah, for a modern large scale distributed system both the client api and implementations are pretty bad.
_kb 19 minutes ago [-]
Syslog is still everywhere on a lot of hardware and appliance style devices.
For those bringing that into modern telemetry stack I can't recommend Grafana Alloy enough (https://grafana.com/docs/alloy/latest/). There's a loki.source.syslog component that lets you ingest RFC5424 or RFC3164 format messages over UDP, TCP, and with mTLS. These can then be parsed, filtered, transformed, and routed to anywhere else you require.
If you're stuck with a device that's UDP only it can deploy to a nearby host (e.g. container runtime on a switch, or local ingest node), perform log collection, then queue and forward via OTel to central or cloud infra with encryption, delivery and integrity guarantees from there. It's a really neat and powerful tool.
ang_cire 4 hours ago [-]
An intro article on syslog servers in the year of our Lord 2026?
Did you know you can replace your noSQL db with VSAM too (honestly better than mongo).
sophacles 2 hours ago [-]
I'd reckon that even in this modern age there are on the order of 10^9 people who don't know what a syslog server is.
nailer 4 hours ago [-]
Mongo stopped having major data loss issues after a decade - my understanding is it's pretty stable now. You might hate the company for beta testing on your data but that's a separate issue.
ang_cire 1 hours ago [-]
I was mostly just being cheeky :P
I don't have anything against mongo in particular, the real nosql problem imo is people overusing it.
5 hours ago [-]
IronWolve 6 hours ago [-]
We moved to splunk now and mostly happy with it, mix of windows/linux/etc logs.
But with AI, I can see opensource alternatives getting better.
unethical_ban 3 hours ago [-]
I've never been in the sales room. I've heard from multiple corps that Splunk is quite expensive. I've been to one place that ran its own Elastic stack, which was a bit of cost in its own right because they had to run all the infrastructure and storage themselves, and had two FTEs whose main job was keeping the thing running. I don't know what Splunk offers from a UI/features perspective that Kibana couldn't do.
Avicebron 3 hours ago [-]
I agree (as the owner of an elastic stack at work). A second FTE sounds like a luxury though..
QuinnyPig 6 hours ago [-]
"Splunk that runs locally with a worse UX, but on balance doesn't eat all the resources you throw at it and doesn't charge you a kidney."
brianjlogan 5 hours ago [-]
There's far cheaper alternatives to Splunk that are still a step up from traditional syslog.
KubeCon over the last couple of years was showing the market was a glut with Observability vendors which is just time series and log management. (traces are logs with a span id).
I'd very much recommend a more modern log stack than a traditional syslog server.
There are many articles covering the limitations of Syslog. Better to replace that component by utilities like OpenTelemetry and JSON structured logging.
You can run a single binary version of Loki https://grafana.com/docs/loki/latest/get-started/deployment-...
Or use something like the Otel collector to send your logs to a remote host.
I have done my fair share of rsyslog and syslog-ng.
I would not say a "Syslog" server belongs in a modern stack.
In fact, the queue management and at least the possibility of some rudimentary end-to-end cryptographic integrity checks are some of the stronger points of rsyslog. Splunk Cloud and Elastic, as far as I know, lacks the latter completely which rules them out as a single log sink for environments with that type of requirements.
A modicum of research reveals that even the rsyslog documentation starts out with UDP for remote delivery: https://docs.rsyslog.com/doc/getting_started/beginner_tutori...
Well, maybe go observe how a broad array of sites implement it in practice, then you might take it more seriously. Maybe you don't implement it that way, but a lot of people will just follow the tutorials or shortcut their way to something that works (but is brittle).
At any rate, I was responding directly to the claim that "No one has suggested running syslog over unreliable transport" which is obviously untrue.
Audit logs are a distinct feature.
Excess load can't take down a well-engineered log collection infrastructure. There can be overload, but the backpressure should propagate downstream and senders and intermediaries should buffer locally if needed. Once the collectors are able to catch up again, the spooled messages will be dispatched, and the backlog should recover.
A well-engineered logging system for sites that care about integrity and durability should look a lot like a distributed message queue.
> Audit logs are a distinct feature.
In my experience, this is not always as distinct as one might hope. On multiple occasions in my career, a customer demanded we perform research using our logs to answer, and the information they sought were not in the class of logs that were considered "audit logs" in advance. Everyone chooses differently what qualifies as "audit logs"; it doesn't have an objective definition.
TCP alone won't get you there. It's certainly not durable in and of itself. All TCP can do is ensure that streamed data is received in the correct order, and confirm that a segment's data was successfully placed into the right buffer on the receiver side. You also need immutable storage, stronger integrity checks than what TCP itself provides, and many other requirements I haven't researched in a while.
Edit: being pedantic -- it's syslog-ng actually.
Once everything is sent to the syslog server it can be bounced to whatever 'non-traditional' stack you want.
My firewalls, PDUs, rear-door heat exchangers, etc, do not talk (nor can they run) "Loki". Just about everything in existence can (a) talk syslog and/or (b) send SNMP traps.
What you can do is have syslog ingestion for your Loki/Elastic/Splunk/whatever you use. But it will not magically structure your logs for you beyond the standard syslog date, host, severity, system, line of text format.
Both syslog products you mention are solid and mature, and absolutely has an important place in any modern environment. They manage queues, throttle, guarantee delivery and support any backend you can wish for.
Their respective rule engines are fast. They do log ingestion, and they slice data into fields and can enrich it (for example by resolving dns). This means rules are set in advance, and traps on data can be triggered immediately (not subject to races).
For those bringing that into modern telemetry stack I can't recommend Grafana Alloy enough (https://grafana.com/docs/alloy/latest/). There's a loki.source.syslog component that lets you ingest RFC5424 or RFC3164 format messages over UDP, TCP, and with mTLS. These can then be parsed, filtered, transformed, and routed to anywhere else you require.
If you're stuck with a device that's UDP only it can deploy to a nearby host (e.g. container runtime on a switch, or local ingest node), perform log collection, then queue and forward via OTel to central or cloud infra with encryption, delivery and integrity guarantees from there. It's a really neat and powerful tool.
Did you know you can replace your noSQL db with VSAM too (honestly better than mongo).
I don't have anything against mongo in particular, the real nosql problem imo is people overusing it.
But with AI, I can see opensource alternatives getting better.
KubeCon over the last couple of years was showing the market was a glut with Observability vendors which is just time series and log management. (traces are logs with a span id).