App logging in Cloud Foundry | Cloud Foundry Docs (2024)

Page last updated:

Loggregator, the Cloud Foundry (Cloud Foundry) component responsible for logging, provides a stream of log output from your app and from Cloud Foundry system components that interact with your app during updates and execution.

Overview

By default, Loggregator streams logs to your terminal. To persist more than the limited amount of logging information that Loggregator can buffer, you can drain logs to a third-party log management service. For more information, see Third-party log management services.

Cloud Foundry gathers and stores logs in a best-effort manner. If a client cannot consume log entries quickly enough, the Loggregator buffer might need to overwrite some lines before the client has consumed them. A syslog drain or a command-line interface (CLI) tail can usually keep up with the flow of app logs.

Cloud Foundry gathers and stores logs in a best-effort manner. If a client cannot consume log entriesquickly enough, the Loggregator buffer might need to overwrite some lines before the client has consumed them. A syslogdrain or a CLI tail can usually keep up with the flow of app logs.

Contents of a log entry

Every log entry contains four fields:

• Timestamp

• Log type, or origin code

• Channel: either OUT, for logs emitted on stdout, or ERR, for logs emitted on stderr

• Message

Loggregator assigns the timestamp when it receives log data. The log data is opaque to Loggregator, which only puts itin the message field of the log entry. Apps or system components sending log data to Loggregator might include their owntimestamps, which then appear in the message field.

Origin codes distinguish the different log types. Origin codes from system components have three letters. The app origin code is APP, followed by slash anda digit that indicates the app instance.

Many frameworks write to an app log that is separate from stdout and stderr. This is not supported by Loggregator. Your app must write to stdout orstderr for its logs to be in the Loggregator stream. Ensure that the buildpack your app useswrites logs to stdout and stderr only. Some buildpacks do this, and some do not.

Log types and their messages

Different types of logs have different message formats, as shown in the following examples. The digit appended to the codeindicates the instance index: 0 is the first instance, 1 is the second, and so on.

API

Users make API calls to request changes in app state. The Cloud Controller, the Cloud Foundry component responsible for the API, logs theactions that the Cloud Controller takes in response.

For example:

2016-06-14T14:10:05.36-0700 [API/0] OUT Updated app with guid cdabc600-0b73-48e1-b7d2-26af2c63f933 ({"name"=>"spring-music", "instances"=>1, "memory"=>512, "environment_json"=>"PRIVATE DATA HIDDEN"})

STG

The Diego Cell or the Droplet Execution Agent emits STG logs when staging or restaging an app. These actions implement the state requested by the user. After the droplet is uploaded, STG messages end and CELL messages begin. For STG, the instance index is almost always 0.

For example:

2016-06-14T14:10:27.91-0700 [STG/0] OUT Staging...

RTR

The Gorouter emits RTR logs when it routes HTTP requests to the app. Gorouter messages include the app name followed by a Gorouter timestamp and selections from the HTTP request.

For example:

2016-06-14T10:51:32.51-0700 [RTR/1] OUT www.example.com - [14/06/2016:17:51:32.459 +0000] "GET /user/ HTTP/1.1" 200 0 103455 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.123 Safari/537.30" 192.0.2.132:46359 x_forwarded_for:"198.51.100.120" x_forwarded_proto:"http" vcap_request_id:9365d216-623a-45cb-6ef6-eba495c19fa8 response_time:0.059468637 app_id:79cc58aa-3737-43ae-ac71-39a2843b5178

Zipkin trace logging

If Zipkin trace logging is allowed in Cloud Foundry, Gorouter access log entries contain Zipkin HTTP headers.

Here is an example access log entry containing Zipkin headers:

2016-11-23T16:04:01.49-0800 [RTR/0] OUT www.example.com - [24/11/2016:00:04:01.227 +0000] "GET / HTTP/1.1" 200 0 109 "-" "curl/7.43.0" 10.0.2.150:4070 10.0.48.66:60815 x_forwarded_for:"198.51.100.120" x_forwarded_proto:"http" vcap_request_id:87f9d899-c7a4-46cd-7b76-4ec35ce9921b response_time:0.263000966 app_id:8e5d6451-b369-4423-bce8-3a7a9e479dbb app_index:0 x_b3_traceid:"2d5610bf5e0f7241" x_b3_spanid:"2d5610bf5e0f7241" x_b3_parentspanid:"-"

For more information about Zipkin tracing, see Zipkin tracking in HTTP headers in HTTP Routing.

LGR

Loggregator emits LGR logs to indicate problems with logging. Examples include can't reach syslog drain url and dropped log messages due to high rate.

APP

Every app emits APP logs to Loggregator.

For example:

2016-06-14T14:10:15.18-0700 [APP/0] OUT Exit status 0

Each app might have a configured log rate limit. If the app logs exceed the configured log rate limit, you see a log entry indicating that the limit was exceeded.

For example:

2022-09-15T01:59:33.99+0000 [APP/PROC/WEB/0] OUT app instance exceeded log rate limit (16384 bytes/sec)

SSH

The Diego Cell emits SSH logs when a user accesses an application container through SSH by using the Cloud Foundry Command Line Interface (cf CLI) cf ssh command.

For example:

2016-06-14T14:16:11.49-0700 [SSH/0] OUT Successful remote access by 192.0.2.33:7856

For more information about the cf ssh command, see the Cloud Foundry CLI reference guide.

CELL

The Diego Cell emits CELL logs when it starts or stops the app. These actions implement the state requested by the user. The Diego Cell also emits messages when an app fails.

For example:

2016-06-14T13:44:38.14-0700 [CELL/0] OUT Successfully created container

Writing to the log from your app

Your app must write logs to stderr or stdout. Both are typically buffered, and you can flush the buffer beforedelivering the message to Loggregator.

Alternatively, you can write log entries to stderr or stdout synchronously. This approach is mainly used fordebugging because it might affect app performance.

Viewing Logs

To view logs, run the cf logs command. You can tail, dump, or filter log output. For more information about the cf logs command, see the Cloud Foundry CLI reference guide.

Tailing logs

To stream Loggregator output to the terminal:

1. Run:

   cf logs APP-NAME

   Where APP-NAME is the name of your app. The command returns output similar to this example:

   Connected, tailing logs for app spring-music in org example / space development as admin@example.com...
   2016-06-14T15:16:12.70-0700 [RTR/4] OUT www.example.com - [14/06/2016:22:16:12.582 +0000] "GET / HTTP/1.1" 200 0 103455 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36" 192.0.2.206:27743 x_forwarded_for:"203.0.113.222" x_forwarded_proto:"http" vcap_request_id:bd3e6ed1-5dd0-43ab-70ed-5d232b577b09 response_time:0.12050583 app_id:79bb58ab-3737-43be-ac70-39a2843b51772016-06-14T15:16:20.06-0700 [RTR/4] OUT www.example.com - [14/06/2016:22:16:20.034 +0000] "GET /test/ HTTP/1.1" 200 0 6879 "http://www.example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36" 192.0.2.206:2228 x_forwarded_for:"203.0.113.222" x_forwarded_proto:"http" vcap_request_id:a31f0b1d-3827-4b8f-57e3-6f42d189f025 response_time:0.033311281 app_id:79bb58aa-3747-43be-ac70-39a3843b51782016-06-14T15:16:22.44-0700 [RTR/4] OUT www.example.com - [14/06/2016:22:17:22.415 +0000] "GET /test5/ HTTP/1.1" 200 0 5461 "http://www.example.com/test5" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36" 192.0.2.206:2228 x_forwarded_for:"203.0.113.322" x_forwarded_proto:"http" vcap_request_id:5d6855a2-4a79-4432-7927-de8215f5a2c7 response_time:0.029211609 app_id:79bb58aa-3737-43bb-ac70-39a2943b5178 ...

Press Ctrl+C or Command+C on your keyboard to exit the real-time stream.

Dump logs

To display all of the lines in the Loggregator buffer:

1. Run:

   cf logs APP-NAME --recent

   Where APP-NAME is the name of your app.

Filter logs

To view some subset of log output, run cf logs APP-NAME in conjunction with filtering commands of your choice, where APP-NAME is the name of your app. In the following example, run cf logs spring-music --recent | grep -v RTR to exclude all Gorouter logs:

$ cf logs spring-musicConnected, tailing logs for app spring-music in org example / space development as admin@example.com...2016-06-14T15:16:12.70-0700 [RTR/4] OUT www.example.com - [14/06/2016:22:16:12.582 +0000] "GET / HTTP/1.1" 200 0 103455 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36" 192.0.2.206:27743 x_forwarded_for:"203.0.113.222" x_forwarded_proto:"http" vcap_request_id:bd3e6ed1-5dd0-43ab-70ed-5d232b577b09 response_time:0.12050583 app_id:79bb58ab-3737-43be-ac70-39a2843b51772016-06-14T15:16:20.06-0700 [RTR/4] OUT www.example.com - [14/06/2016:22:16:20.034 +0000] "GET /test/ HTTP/1.1" 200 0 6879 "http://www.example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36" 192.0.2.206:2228 x_forwarded_for:"203.0.113.222" x_forwarded_proto:"http" vcap_request_id:a31f0b1d-3827-4b8f-57e3-6f42d189f025 response_time:0.033311281 app_id:79bb58aa-3747-43be-ac70-39a3843b51782016-06-14T15:16:22.44-0700 [RTR/4] OUT www.example.com - [14/06/2016:22:17:22.415 +0000] "GET /test5/ HTTP/1.1" 200 0 5461 "http://www.example.com/test5" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36" 192.0.2.206:2228 x_forwarded_for:"203.0.113.322" x_forwarded_proto:"http" vcap_request_id:5d6855a2-4a79-4432-7927-de8215f5a2c7 response_time:0.029211609 app_id:79bb58aa-3737-43bb-ac70-39a2943b5178 ...

Use Ctrl-C (^C) to exit the real-time stream.

Dumping logs

To display all of the lines in the Loggregator buffer, run cf logs APP-NAME --recent, where APP-NAME is the name ofyour app.

Filtering logs

To view some subset of log output, run cf logs APP-NAME in conjunction with filtering commands of your choice.Replace APP-NAME with the name of your app. In the following example, grep -v RTR excludes all Gorouter logs:

$ cf logs spring-music --recent | grep -v RTR2016-06-14T14:10:05.36-0700 [API/0] OUT Updated app with guid cdabc604-0b73-47e1-a7d5-24af2c63f723 ({"name"=>"spring-music", "instances"=>1, "memory"=>512, "environment_json"=>"PRIVATE DATA HIDDEN"})2016-06-14T14:10:14.52-0700 [APP/0] OUT - Gracefully stopping, waiting for requests to finish2016-06-14T14:10:14.52-0700 [CELL/0] OUT Exit status 02016-06-14T14:10:14.54-0700 [APP/0] OUT === puma shutdown: 2016-06-14 21:10:14 +0000 ===2016-06-14T14:10:14.54-0700 [APP/0] OUT - Goodbye!2016-06-14T14:10:14.56-0700 [CELL/0] OUT Creating container ...

Log ordering

Both operators and developers must ensure log ordering in drains.

• Diego uses a nanosecond-based timestamp that can be ingested properly by Splunk. For more information, see the Splunkdocumentation.

• The Elastic Stack can ingest the nanosecond timestamps but only supports millisecond precision, so timestamps are truncated. For more information, see the Elasticsearch documentation and Date type has not enough precision for the logging use case on GitHub.

If you are developing a client that displays a stream of Cloud Foundry logs to users, you can order the logs to improve the debugging experience for your user. Here are some general tips for ordering logs:

• For CLIs, batch the logs and display the logs you have in that timeframe, sorted by timestamp.

• For web clients, use dynamic HTML to insert older logs into the sorting as they appear. This creates complete, ordered logs.

• Java app developers might want to convert stack traces into a single log entity. To simplify log ordering for Javaapps, use the multi-line Java message workaround to convert your multi-line stack traces into a single log entity.For more information, see Multi-line Java message workaround.

  By changing the Java log output, you can force your app to reformat stack trace messages, replacing newlinecharacters with a token. Set your log parsing code to replace that token with newline characters again to display thelogs properly in Kibana.