Page MenuHomePhabricator

[services] Lib - Log everything to files
AbandonedPublic

Authored by karol on Jul 21 2022, 3:47 AM.
Tags
None
Referenced Files
F3301084: D4597.diff
Sun, Nov 17, 11:49 PM
Unknown Object (File)
Fri, Nov 8, 8:28 PM
Unknown Object (File)
Fri, Nov 8, 8:13 AM
Unknown Object (File)
Tue, Nov 5, 2:05 AM
Unknown Object (File)
Oct 7 2024, 4:53 AM
Unknown Object (File)
Sep 27 2024, 8:54 AM
Unknown Object (File)
Sep 26 2024, 6:14 PM
Unknown Object (File)
Sep 26 2024, 6:13 PM

Details

Reviewers
tomek
max
Summary

Depends on D4596

Use directory /logs added in the previous diff to put logs there. We want to put all the logs in this directory.

Test Plan

None for now. Once the previous diff is accepted, I'll be able to update this.

Diff Detail

Repository
rCOMM Comm
Lint
No Lint Coverage
Unit
No Test Coverage

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 21 2022, 3:55 AM
Harbormaster failed remote builds in B10718: Diff 14757!
karol edited the summary of this revision. (Show Details)
karol edited the test plan for this revision. (Show Details)
karol added reviewers: tomek, max.

One TODO: What if we spawn more than one instance of the same service in one docker container? We should probably use some uuid to avoid having the logs mixed up.

Requesting review for the same reason as the previous diff.

max requested changes to this revision.Jul 21 2022, 6:41 AM

I think it's not a good idea to explicitly out all the logs to file only. Actually, it's a bad practice for microservices.

When we have an output to stdout and stderr as a first we can distinguish and separate log types.
Also, we can use log aggregation services. For example local log aggregation and analyzing the logs from the central dashboard from all the microservices fleet.
Using the standard output that glog provides, we can redirect the stderr and stdout from the service to the file like command 2> error.txt 1> output.txt and distinguish them.

I'm sure we shouldn't explicitly use the log file in the code or at last, add the starting flag to use the logging to the files with the distinguish error and info logging to different files and use the config file to provide these file names. For example, such approaches use Nginx, MySQL, and other services.

This revision now requires changes to proceed.Jul 21 2022, 6:41 AM
In D4597#131546, @max wrote:

I think it's not a good idea to explicitly out all the logs to file only. Actually, it's a bad practice for microservices.

Interesting. Where does this statement come from? Do you have any sources?

I'm sure we shouldn't explicitly use the log file in the code or at last, add the starting flag to use the logging to the files with the distinguish error and info logging to different files

I'm not sure if it is a good idea. We want to use logging for debugging, right? I assume it's good to use the same file for info and error because it lets us see what happened after what and what exactly caused a problem - what was an exact flow. If we stream err and info into different files, we'll have to jump between multiple files and explicitly check the timestamps of the logs to deduce what actions cause a certain problem.

and use the config file to provide these file names.

Glog handles file naming on its own so I don't see why we would customize this.

For example, such approaches use Nginx, MySQL, and other services.

Do you have sources for that?

When we have an output to stdout and stderr as a first we can distinguish and separate log types.
Also, we can use log aggregation services. For example local log aggregation and analyzing the logs from the central dashboard from all the microservices fleet.
Using the standard output that glog provides, we can redirect the stderr and stdout from the service to the file like command 2> error.txt 1> output.txt and distinguish them.

I see. I think it's worth checking whether it is possible to use both, std out and output files at the same time, using glog. Would that be ok?

max requested changes to this revision.Jul 22 2022, 4:09 AM
In D4597#131760, @karol wrote:
In D4597#131546, @max wrote:

I think it's not a good idea to explicitly out all the logs to file only. Actually, it's a bad practice for microservices.

Interesting. Where does this statement come from? Do you have any sources?

For example local/remote log aggregation and analyzing the logs. Logs aggregator needs to open the file or listen for its updates instead of getting the events directly from Syslog in case of using the direct stdout and stderr. If you have a fleet of microservices in production how do you analyze the logs from files? If you are explicitly using files in the code can you switch to the standard output? No. If you use standard output can you switch to using files? Yes. It's about flexibility without touching the code.

Syslog already has the functionality to distinguish info/error messages, parsing by date, and many more. It removes or packs the old logs (make logs rotation) as well. When you are logging something, you also should keep in mind the log sizes and log rotation.

I'm sure we shouldn't explicitly use the log file in the code or at last, add the starting flag to use the logging to the files with the distinguish error and info logging to different files

I'm not sure if it is a good idea. We want to use logging for debugging, right? I assume it's good to use the same file for info and error because it lets us see what happened after what and what exactly caused a problem - what was an exact flow. If we stream err and info into different files, we'll have to jump between multiple files and explicitly check the timestamps of the logs to deduce what actions cause a certain problem.

You can point both stdout and stderr to the single file when starting the app using myapp &> logfile.txt . Also, the common workflow is not to run something and open the log file, there is another one which is: you run something in a different terminal and see the output right out the way without the need to save to file and open it after. That's why logging to the standard output is a flexible way: you can do both.

and use the config file to provide these file names.

Glog handles file naming on its own so I don't see why we would customize this.

That makes sense! But we should customize the logs directory.

For example, such approaches use Nginx, MySQL, and other services.

Do you have sources for that?

For example, Nginx logging - Uses log files or syslog. We can also use syslog to process the logs (use standard stdout, stderr in that case).

When we have an output to stdout and stderr as a first we can distinguish and separate log types.
Also, we can use log aggregation services. For example local log aggregation and analyzing the logs from the central dashboard from all the microservices fleet.
Using the standard output that glog provides, we can redirect the stderr and stdout from the service to the file like command 2> error.txt 1> output.txt and distinguish them.

I see. I think it's worth checking whether it is possible to use both, std out and output files at the same time, using glog. Would that be ok?

I think we should use a flag for standard output logging or for file logging, in this case, we will have the flexibility to use both without any changes to the code.

This revision now requires changes to proceed.Jul 22 2022, 4:09 AM

Agree with @max that we should use stdout / stderr instead of trying to handle log persistence ourselves

Ok, I think this diff can be abandoned