How to Get Syslog into Splunk and What Can Go Wrong

Maciek Stopa with Claude Fable 519 min readUpdated #splunk, #syslog, #sc4s, #otel, #edge-processor, #data-onboarding, #gdi

There are five popular ways to get syslog into Splunk, each with its own failure modes, and Splunk's Validated Architectures point to SC4S as the default for a dedicated syslog tier. This guide covers all five, with most of the space given to how each one breaks in production.

This post is the syslog deep dive of my field guide to every way of getting data into Splunk.

Why syslog is messy

Syslog is a push-based logging protocol. It is what, for example, firewalls, routers, switches, wireless controllers, and load balancers emit: usually the appliances you cannot install an agent on. In a SIEM context, that is often the highest-value security data you have.

Syslog ran for twenty years with no specification. Eric Allman wrote it in 1981 as the logging side of BSD sendmail, and it spread as a de facto standard: fire-and-forget UDP datagrams to port 514, no delivery guarantee, no backpressure. When the IETF finally wrote a specification, RFC 3164 (2001), it was published as informational: not a protocol to implement, but a description of the traffic already on the wire.

RFC 5424 (2009) is the actual standard: a versioned header, structured data, timestamps that carry a year and a timezone. RFC 5425 added TLS-wrapped syslog on port 6514, and RFC 6587 documented how to frame syslog over a TCP stream: newline-delimited in practice, or an octet-counting scheme that few devices use. Unfortunately, seventeen years later, most network gear still emits RFC 3164-ish lines, or something fully proprietary that only loosely resembles it.

One protocol, four wire formats

RFC 3164 (BSD syslog)

The 2001 write-up of existing practice. Most network gear still sends something like this.

<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8

Field by field
FieldValueMeaning
PRI<34>Priority = facility × 8 + severity: here facility 4 (auth) and severity 2 (crit).
TIMESTAMPOct 11 22:14:15No year, no timezone, and days 1 to 9 get a padding space that breaks single-space parsers.
HOSTNAMEmymachineUnverified text the device chose: short name from one source, FQDN from the next, sometimes nothing at all.
TAGsu:Program name, 32 characters max. Everything from the colon on is technically already message content.
MSG'su root' failed for lonvick on /dev/pts/8Free text. No structure is guaranteed past this point.

The traps in one line: no year, no timezone, space-padded days, and a hostname nobody verified.

RFC 5424

The 2009 standard: versioned header, real timestamp, structured data.

<165>1 2026-08-02T22:14:15.003Z fw01.example.com evntslog 1024 ID47 [exampleSDID@32473 iut="3" eventSource="Application"] An application event log entry

Field by field
FieldValueMeaning
PRI<165>Same math as RFC 3164: facility 20 (local4) × 8 + severity 5 (notice).
VERSION1Always 1 so far. A digit right after the PRI is the quickest way to spot RFC 5424.
TIMESTAMP2026-08-02T22:14:15.003ZRFC 3339 format: year, sub-second precision, explicit timezone. Everything the RFC 3164 timestamp lacks.
HOSTNAMEfw01.example.comFQDN preferred by the RFC, but still whatever the device decides to send.
APP-NAMEevntslogThe program name. A dash means nil.
PROCID1024Process ID, or a dash for nil.
MSGIDID47Message type identifier, or a dash.
STRUCTURED-DATA[exampleSDID@32473 iut="3" eventSource="Application"]Key-value pairs under a registered SD-ID, or a dash. The field almost no network vendor populates.
MSGAn application event log entryFree text, optionally UTF-8 with a byte-order mark.

The catch: seventeen years on, most gear still does not send it, and mixed 3164/5424 fleets need a receiver port per dialect.

RFC 6587 octet counting

TCP framing with a byte-count prefix instead of newline splitting.

84 <165>1 2026-08-02T22:14:15Z fw01.example.com sshd - - - Accepted publickey for admin

Field by field
FieldValueMeaning
MSG-LEN84Byte count of the frame that follows. The receiver reads exactly 84 bytes: no delimiter guessing, and embedded newlines survive.
SYSLOG-MSG<165>1 2026-08-02T22:14:15Z fw01.example.com sshd - - - Accepted publickey for adminA complete RFC 5424 message, exactly as counted.

Almost no device sends this. Most TCP syslog is newline-delimited instead, which is why multiline events get split.

What a firewall actually sends

A Cisco ASA line. It follows neither RFC, which is the point.

<166>Aug 02 2026 22:14:15: %ASA-6-302013: Built outbound TCP connection 4051 for outside:203.0.113.5/443 (203.0.113.5/443) to inside:10.1.1.9/52001 (10.1.1.9/52001)

Field by field
FieldValueMeaning
PRI<166>Facility 20 (local4) × 8 + severity 6 (info). The severity appears again in the vendor tag.
TIMESTAMPAug 02 2026 22:14:15:A year RFC 3164 never had, a zero-padded day where the RFC wants space padding, and a trailing colon.
VENDOR TAG%ASA-6-302013:Cisco format: platform, severity (again), and message ID. Not a valid RFC 3164 TAG.
MSGBuilt outbound TCP connection 4051 for outside:203.0.113.5/443 (203.0.113.5/443) to inside:10.1.1.9/52001 (10.1.1.9/52001)The useful part. Add-ons parse it with regexes keyed on the message ID.

No RFC produces this line, and there is no hostname unless "logging device-id" is configured. This is why SC4S ships vendor filters and why strict receivers land real traffic in fallback.

That history leaves four properties, and every failure mode in this article grows out of at least one:

  • no delivery guarantee
  • no single message format
  • no year and no timezone in RFC 3164 timestamps
  • no size discipline: a long event will meet a truncation limit somewhere

The common receivers

Outside the Splunk world, a handful of tools dominate syslog reception.

  • rsyslog is the default system logger on most Linux distributions and the performance reference point. Its author, Rainer Gerhards, also created RELP, the only widely deployed reliable syslog transport.
  • syslog-ng is its long-time rival, available in open source and commercial editions. It is the engine inside Splunk's own SC4S.
  • Kiwi Syslog Server is the classic GUI syslog server for Windows teams, current as Kiwi Syslog Server NG after the legacy edition left support in March 2026. NXLog is a cross-platform log collector with strong Windows support that often fills the same receiver role.
  • Fluent Bit and the OpenTelemetry Collector, the modern pipeline agents, accept syslog too.

What can go wrong no matter which method you pick

Some failure modes are shared by all five methods; no tool choice makes them disappear.

  • UDP loss is silent and unmeasurable at the receiver. You cannot count packets that never arrived. Kernel-level drops show up in netstat -su and buffer statistics; loss on the wire shows up nowhere. Balázs Scheidler, the creator of syslog-ng and founder of Axoflow, estimates that users who start measuring typically find 30-40% of syslog-over-UDP messages missing in high-traffic setups.
  • Every receiver restart drops in-flight UDP. Upgrades, config reloads, host patching: no method fixes this. Redundancy (an LB or a VRRP pair) shrinks the window for planned work, but UDP gives the failover machinery no signal about datagrams in flight, so you cannot guarantee zero loss. The architecture question is only how long the window is and how often you open it.
  • TCP and TLS do not make delivery reliable. The sender's TCP stack acknowledges bytes into a socket buffer; if the receiver dies, those buffered messages are gone and the sending application cannot tell. Rainer Gerhards wrote the canonical analysis: 10 to 1,600 messages can vanish with every broken connection, and without keepalive the sender may not notice a dead receiver for hours. RELP fixes this with real acknowledgments, but only rsyslog speaks it. What remains is disk buffering after the first hop, and that machinery fails too: rsyslog's disk queues lose messages on restart, syslog-ng's disk buffers corrupt on resize and orphan on config changes, and the OTel Collector's persistent queue spent a year unable to survive its own corruption. Treat it as production software with failure modes, not as a checkbox.
  • Port 514 hides two different problems. The server-side privilege problem (ports below 1024 need root or CAP_NET_BIND_SERVICE) has an easy answer: listen on a high port like 5514 instead. But that answer assumes your devices can follow you there, and plenty of legacy gear hard-codes destination 514 and cannot be told otherwise. For those senders you are back to listening on 514 anyway, or rewriting the port in the network (a DNAT/redirect rule) before packets reach the receiver.
  • Device timestamps can't be trusted. RFC 3164 timestamps have no year and no timezone. In practice: replayed data stamped with the current year, dashboards that go dark at the start of a month after a timestamp misparse, and Splunk grabbing digits of an IP address as the year.
  • "Which host sent this?" has two answers, and both fail. The hostname in the message header is whatever the device chose to write: short name for one source, FQDN for the next, sometimes nothing. When the header can't be trusted, receivers fall back to the source IP and resolve it to a readable name with a DNS lookup as events arrive. That stalls the whole pipeline whenever DNS is slow. Also, the packet's source IP is honest only when the device talks to the receiver directly; put a relay or a load balancer that doesn't preserve client IPs in the path, and every event's origin becomes that middlebox.
  • Truncation lives at every layer, each with its own limit. A UDP datagram caps an unfragmented event around 1,500 bytes on a standard MTU; receiver parsers have their own ceilings (the OTel syslog receiver, for one, rejects messages over its 8,192-byte default outright); Splunk's TRUNCATE defaults to 10,000. A long event hits whichever limit comes first. Some layers trim it, some drop it whole, and either way it happens silently.
  • Load balancers and UDP syslog don't go well together. Flow hashing, source-IP preservation, and health checks are all problems here. The cloud-specific catch: an NLB-style health check probes a TCP or HTTP port, never the UDP service itself, so a dead syslog daemon on a live host keeps receiving traffic. And per SC4S's load-balancer guidance, my own contribution to the project, front-side load balancing often distributes unevenly, so one receiver's buffers fill while the next node idles.
  • Multiline messages break framing assumptions everywhere. By default, syslog framing assumes one line is one event, but plenty of sources send more. Depending on transport and timing, receivers either split multiline messages into fragments or glue unrelated lines together.
  • Traffic is rarely spread evenly across senders. One chatty firewall can produce most of your total event volume, and receivers typically process everything from one sender on one worker, so that single stream saturates one CPU while the rest of the box idles (the mechanism behind this is explained in the SC4S section).

A syslog architecture that holds up in production

It does three things:

  1. Collects at the edge. Receivers sit as close to the sources as possible, with deliberately simple failover designs instead of load balancers. The SC4S architecture page calls edge collection the most reliable way to gather syslog and concedes it can at best be made "mostly available".
  2. Treats the first hop as lossy and every later hop as at-least-once. The device-to-receiver hop cannot be fixed; everything after it can buffer and acknowledge: disk buffers, HEC acknowledgments where supported, S2S useACK.
  3. Respects backpressure. A stalled TCP receiver can block logging on the sending device itself, and some devices block their data plane when logging blocks. Sometimes dropping is the safer failure mode; decide per source class, compliance logs versus operational noise.

What you can actually do about UDP loss is measure around it: baseline events per second for every source, alert on deviation (a tstats by host against expected rates), and inject canary events where the source allows it.

Method 1: direct TCP/UDP inputs on Splunk

The setup is a few lines of inputs.conf on an indexer or heavy forwarder: [udp://514], [tcp://514], or [tcp-ssl:6514]. Each port carries exactly one sourcetype.

Direct TCP/UDP input is not supported in Splunk Cloud; this method is on-prem only.

What can break

The structural problem: the syslog receiver is splunkd itself, so receiver health and indexing health are the same thing.

  • Every restart is a loss window. Config changes, upgrades, host patching: each one drops whatever UDP arrives while splunkd is down.
  • Backpressure reaches the sending devices. When indexing queues block, UDP drops silently and TCP senders stall; some devices then drop locally rather than buffer.
  • The input defaults mangle syslog. UDP inputs prepend Splunk's own timestamp and host to every event, the PRI header is stripped before facility and severity can become fields, and sourcetype=syslog activates a built-in transform that silently rewrites the host field from message content. All three surface as broken add-on field extractions.
  • One port is one sourcetype, and UDP is stream-merged. Splunk "performs event merging on the data stream", so until LINE_BREAKER is right, unrelated devices' lines get glued into single events. Mixed devices on 514 become undifferentiated "syslog" either way.
  • The Universal Forwarder variant is worse. A UF can listen too, but its default maxKBps=256 throughput cap silently delays data, there is no parsing tier, and the Validated Architectures state that Splunk "strongly discourages this practice in any production environment."

Example tickets: direct inputs

A sample of public Splunk Answers threads for this method. The input defaults account for most of them.

6 more tickets

Ticket states are as of publication, August 2026.

Verdict: lab and proof-of-concept only. Lantern's syslog page concedes that customers do this "to avoid architecting a syslog server, which introduces further problems."

Method 2: your own rsyslog/syslog-ng tier

This is the classic DIY tier, and for a decade it was the recommended one. rsyslog or syslog-ng listens on 514 and writes each device's events to its own file; logrotate keeps the disk alive. A Universal Forwarder tails the file tree, takes the host from the file path, and assigns a sourcetype per directory.

Turning that sketch into production is mostly tuning: receiver queues sized for bursts, rotation that does not race the writer or the tailing UF, the UF's throughput cap raised from its low default, and a keepalived pair for HA (DNS round-robin is not failover).

There is also a leaner variant that skips the UF; it gets its own subsection below.

What can break

  • The UF's throughput cap defaults to 256 KB/s. Data arrives hours late under load and nobody knows why, because nothing errors.
  • The file handoff between daemon and UF is fragile. A logrotate misconfiguration fills the disk, the receiver dies, and the whole fleet's syslog goes silently missing until someone notices. Scale hurts too: thousands of active files overflow the UF's file-descriptor cache, and large rotated files push its reader into single-threaded batch mode with multi-hour lag.
  • The reliability machinery is its own failure surface. rsyslog's disk queues and syslog-ng's disk buffers corrupt, orphan, and lose messages on restart in documented ways, and recovering an orphaned disk buffer is a manual procedure. The ticket list below samples both trackers; the same pattern returns in every other method.
  • It is two systems to keep in sync. Every new device type means a receiver filter change plus UF inputs/props changes, usually with restarts on both. Timestamp policy (does the written file keep device time or receiver time?) is a decision to make once and document. And hand-grown rulesets end up understood by exactly one admin.

The straight-to-HEC variant

A leaner variant skips the UF entirely: rsyslog's omhttp or syslog-ng's http() destination posts straight to HEC. Two catches. omhttp is contrib-grade: it has silently stopped sending to HEC (a 2020 report fixed only in June 2026) and its documented retry pattern deadlocked the daemon until a mid-2026 fix; both are in the ticket list below. And syslog-ng's http() batches punish routing mistakes: one event bound for a bad index poisons the whole batch, up to 1,000 events.

Example tickets: rsyslog and syslog-ng

A sample from the daemons' own trackers and Splunk Answers. The queue and buffer machinery dominates.

12 more tickets

Ticket states are as of publication, August 2026.

Verdict: the Validated Architectures now present SC4S as the turn-key recommendation and this tier as the alternative for shops with existing syslog infrastructure and specific requirements. Keep it if it already exists and is well-run; do not build it new in 2026.

Method 3: SC4S (Splunk Connect for Syslog)

SC4S is Splunk's answer to all of the above. It is a container (podman, Docker, or Kubernetes) wrapping syslog-ng OSE plus a curated filter library covering roughly 90 vendors. The filters classify devices automatically and ship events to HEC with sourcetype, index, and CIM-friendly metadata already set (though what SC4S sets sometimes diverges from what the Splunkbase add-ons expect; details in the what-can-break list). It's Splunk supported.

Setup is short, and the quickstart guide covers it end to end. On the host, apply the documented sysctl UDP buffer values and free port 514 from the local syslogd. Then run the container with host networking, an env file holding the HEC URL and token, and the systemd unit from the docs.

On the Splunk side: create the indexes SC4S expects (netops, netfw, netauth, osnix, and the rest) or remap them, and scope the HEC token to them. Never enable indexer acknowledgement on that token: the underlying syslog-ng http destination does not support it, and enabling it floods HEC with 4xx errors.

Devices are routed by the auto-detection filters, by dedicated ports for ambiguous vendors, and by csv overrides. Remember to tune three things:

  • Enable, size, and mount the disk buffer. Durability requires the persistent volume (SC4S_PERSIST_MOUNT=splunk-sc4s-var:/var/lib/syslog-ng) to actually be configured. Otherwise the buffer lives in the container's ephemeral filesystem and is wiped on every restart. And the size variable is allocated per worker queue file, not once, so real disk use is a multiple of it.
  • Point HEC at the indexers directly. The docs recommend against an intermediate heavy-forwarder tier, and native multi-URL balancing is sanctioned only for ten or fewer indexers with syslog-exclusive HEC; beyond that, you can front HEC with an external load balancer.
  • Turn on the monitoring hook most deployments ignore. SC4S ships internal spl.sc4syslog.* metrics (received, dropped, queued per destination) to a metrics index with a ready-made dashboard, which I actually wrote. This is an easy way to detect drops and queue growth. The working checklist: metrics index plus dashboard, an alert on dst.dropped growth, a saved search watching the fallback index, and per-source EPS baselines.

What can break

  • Vendor filters are a moving target. Device firmware upgrades silently break classification, look-alike events get claimed by the wrong vendor's filter, and SC4S's own upgrades reclassify: sourcetypes shift, dashboards and CIM mappings might break. Pin versions, test in staging, and watch the fallback index: unknown devices land there, and nobody watches it.
  • The default config is not sized for scale. The official benchmark page shows a single TCP connection doing 14.6k messages per second with defaults versus 86k tuned, and UDP dropping 51% at 27k EPS with default settings. The tuning guidance is mandatory, not optional.
  • The disk buffer is a lifecycle trap. Capacity is frozen at creation: on a resize attempt syslog-ng logs "Continuing with the old one" and keeps the old size. A config change can orphan the buffer, an orphaned buffer never replays without a manual recovery procedure, and a full buffer volume can prevent the container from restarting at all, because the image pull the unit runs at start needs headroom on the same disk. What fills it is destination backpressure: an indexer outage or a Splunk Cloud maintenance window stalls SC4S until HEC drains.
  • A restart depends on the container registry. The systemd unit re-pulls the container image on every start, so a registry or network outage blocks SC4S from restarting entirely. Pin a local image.
  • Add-on compatibility is not always guaranteed. SC4S parsers replace the index-time work Splunkbase add-ons would do, and sometimes its sourcetypes diverge from what some add-ons expect, so the add-ons' sourcetype renames and sed-style _raw rewrites never run. Indexed _raw is rewritten too: the default template drops the original timestamp and header, which matters for compliance and for any second SIEM reading the same feed.
  • The migration counting trap. Cutting over from a raw rsyslog/syslog-ng tier to SC4S can lower total event counts, because SC4S's strict validation discards malformed and almost-compliant messages the old tier passed through untouched (the per-technology SC4S_DISABLE_DROP_INVALID_* flags are the escape hatch). Teams read the delta as data loss and distrust the new tier. At cutover, compare per-source counts between old and new paths and triage the difference before judging.

Example tickets: SC4S

A sample from SC4S's public issue tracker and Splunk Answers. The project fixes steadily; the point is what kind of thing breaks.

19 more tickets

Ticket states are as of publication, August 2026.

Verdict: still the default choice for a dedicated syslog tier in Splunk environments, and the one the Validated Architectures recommend. Plan for filter drift, treat the tuning as mandatory, accept the observability gap, and turn the monitoring on before you need it.

Method 4: OpenTelemetry Collector

The Collector receives syslog with its own building blocks: a syslog receiver that parses RFC 3164 or 5424 over TCP or UDP, and raw udplog/tcplog receivers plus parsing operators for everything the strict parser rejects. Events then flow through processors into the splunk_hec exporter, with sourcetype and index set in exporter config and an optional disk-backed sending queue. Applications and agents pushing OTLP natively are a separate ingestion story, covered in the field guide.

What can break

  • Real devices violate RFCs, and the strict receiver rejects them. A trailing space was enough to break parsing until a May 2024 fix, and messages without a PRI header were rejected outright until opt-in support arrived in 2024. For everything the parser still rejects, you fall back to udplog plus parsing operators and end up hand-building parsers after all. One receiver also speaks exactly one dialect: auto-detection was requested and closed "not planned", so mixed 3164/5424 fleets need a port per dialect.
  • There is no vendor classification library. SC4S's main value must be re-implemented by hand, and Splunk add-on and CIM compatibility is entirely on you (for Splunk's own distribution, a first CIM sourcetype transform was proposed only in July 2026 and is unmerged at the time of writing).
  • The defaults are lossy, and backpressure is missing where it matters. The in-memory queue dies with the process, memory_limiter drops by design, and the persistent queue is opt-in with its own corruption history. Under overload the syslog receiver OOMs the collector while the OTLP receiver stays flat, because block_on_overflow does not propagate to the input.
  • No HEC indexer acknowledgment, closed "not planned". HTTP 200 counts as success even if Splunk fails to index. An end-to-end delivery guarantee is impossible on this path.
  • The syslog components are young. udplog and tcplog are still in alpha, octet-counting support shipped broken, CEF-over-syslog parsing does not exist (the workaround is OTTL's ParseCEF), and high-EPS UDP tuning is thinly documented compared to rsyslog and syslog-ng.

Example tickets: OTel Collector

A sample from the opentelemetry-collector-contrib tracker. Note the dates: the syslog path matured mostly between 2024 and 2026, and several requests were closed as not planned.

14 more tickets

Ticket states are as of publication, August 2026.

Verdict: the right choice only if your organization is already OTel-first and accepts DIY parsing and community-level support as the price of one agent everywhere. Otherwise you take on the risks of a young component and get nothing in return.

Method 5: Edge Processor and Ingest Processor

Edge Processor runs on Linux nodes you manage, controlled from a Splunk Cloud control plane or, since Splunk Enterprise 10.0, an on-prem one. Its syslog source speaks RFC 3164, 5424, and 6587 over TCP and UDP, with mTLS support. Ports below 1024 require running it as root. You select the RFC per source, and that selection fixes how the transport is handled; devices that follow no RFC go through "Other formats" plus manual SPL2 extraction.

Ingest Processor is the SaaS sibling. Its only inbound ports are HEC (8088) and S2S from forwarders (9997), so it processes syslog that something else, like SC4S, already delivered. It does not replace an edge tier. Both are covered as processing tools in the field guide.

What can break

  • "Edge Processors currently provide no data delivery guarantees." That is Splunk's own documentation, verbatim. The queue in front of a stalled destination is not persistent: the same page says a shutdown or restart while data is in flight can cause data loss. For UDP delivery problems, the troubleshooting docs' remedy amounts to keep sending until data arrives, or switch to TCP.
  • Timestamp and host handling is DIY. There is no vendor-aware timestamp parsing: extracting _time is your SPL2 pipeline's job via strptime, timezone is one setting shared by every Edge Processor in the tenant, and an RFC mismatch stamps events with Host <nil> and Source edge-source, breaking host-based routing and searches. Moving RFC 3164 sources off SC4S onto EP quietly regresses timestamp quality. And any pipeline that touches _time wins over what the forwarder assigned, so in UF-to-EP-to-indexer paths device timestamps can be silently replaced mid-flight.
  • The observability gap is wider than SC4S's. Monitoring stops at sourcetype and pipeline granularity: you infer drops by comparing a pipeline's inbound and outbound volume, and nothing attributes a drop to a specific sender.
  • No vendor filter library. Classification is whatever your SPL2 pipelines do, the team now needs SPL2, and the shared failure modes still apply: UDP restart loss, load-balancer problems, node-group sizing.

Edge Processor has no public issue tracker, so this method gets no ticket list.

Verdict: interesting when you are already in the Splunk Cloud and SPL2 ecosystem and consolidating pipelines; not the mature default for a syslog tier, and not a place for must-not-lose data.

Share on LinkedIn