AWS Logs Insights

Use AWS Logs Insights to make sense of complex logs

The basic parts of a Logs insights query is as follows.

1. Fields: These are keys of the AWS log message format. There are usually two values (when you look at the AWS console)

   1. timestamp: This is pretty straight forward.
   2. message: The entire logged message in JSON or plain text.
   3. logStream: The CloudWatch Logstream
   4. log: The specific log.

2. Parse: Parse the text of the message based on certain conditions. See below for details.

3. Filter: Use this to filter the output.

   For example: filter using timestamp

   filter @timestamp >= timestamp('2024-08-10T00:00:00Z')
   

   Filter using text in the message.

   filter @message like /ERROR|WARN/
   

   Filter using Log stream.

   filter @logStream like <log-stream-pattern>
   

4. Sort: Sort the output: For example; descending using timestamp

   sort @timestamp desc
   

5. Limit: Limit the output to the first x results.

   limit 30
   

6. Display: Display only certain fields

   display @timestamp,level
   

Parsing

Since the entire second field of logs insights is message , we can apply additional queries on the text (or JSON) message.

We’d need to apply a regex to parse the messages out of the logs.

{"level": "info", "grpc.method": "GET"}

| parse @message '"level":"*"' as level

References

1. General: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html
2. Querying: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-Parse.html