Page MenuHomePhabricator

[services] Lib - Move Item.h
ClosedPublic

Authored by karol on May 4 2022, 6:21 AM.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Nov 10, 2:56 PM
Unknown Object (File)
Thu, Nov 7, 9:35 PM
Unknown Object (File)
Thu, Nov 7, 6:12 PM
Unknown Object (File)
Tue, Nov 5, 3:55 PM
Unknown Object (File)
Sat, Nov 2, 12:27 PM
Unknown Object (File)
Fri, Oct 25, 7:20 PM
Unknown Object (File)
Fri, Oct 25, 7:13 PM
Unknown Object (File)
Thu, Oct 17, 11:09 PM

Details

Summary

Depends on D3908

Moving Item.h from both blob/backup to the mutual "lib" space.

Test Plan

No real changes are applied, just moving stuff around, so services should still build and that's the test plan

cd services

yarn run-blob-service
yarn run-backup-service

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

karol edited the test plan for this revision. (Show Details)
karol added a reviewer: tomek.
ashoat requested changes to this revision.May 4 2022, 11:00 AM
ashoat added a subscriber: max.

Hmm... I'm a bit confused about this lib_src folder. I can see why it's named lib_src instead of src... because at some point it is copied into the same folder as the specific service's src. I also noticed there is a lib inside the Docker container that is different from our services/lib.

That said, I'm concerned about this approach for two reasons:

  1. Based on @geekbrother's feedback in ENG-1060, it seems like we will probably end up needing to figure out a way to build this that doesn't require Docker. Since Docker is the one that moves the folders into one place, we will need to figure out a different approach anyways, that probably relies on the directory structure in the actual repo.
  2. I think it's more simple to use the same directory structure we have in the repo. That way we don't have to reason about two distinct directory structures.

What do you think of changing the code so that instead of lib_src we have lib/src, instead of lib/double-conversion we have 3rd-party/double-conversion (or something similar)? The idea would be to have the same directory structure inside Docker that we have in the repo (so CMake would work just as well inside or outside Docker), and for anything additional that is needed inside Docker we would put it in a folder that doesn't overlap with any folder in the repo.

This revision now requires changes to proceed.May 4 2022, 11:00 AM

I also noticed there is a lib inside the Docker container that is different from our services/lib.

I don't understand what is the problem here? Do you mean /lib? This folder is probably in every Unix. All contents of services/lib are being copied to transferred. So the folder structure is flattened. Something like this happens:

cp -r services/lib/* /transferred
cp -r services/backup/* /transferred # or blob, depends on the service

I couldn't name this src as it would get overwritten by specific service's src.

I think it's more simple to use the same directory structure we have in the repo. That way we don't have to reason about two distinct directory structures.

Ok but at the same time in D3904 you said

I don't think it's crazy for the folder to have a different name inside vs. outside the container, and the extra line in the Dockerfile isn't a big deal in my opinion.

So I think we should be consistent here. Either we take a laid back approach and allow ourselves to modify the structure or we keep it tight. I see some opposite ideas here, do I miss something?

What do you think of changing the code so that instead of lib_src we have lib/src

Do you mean /lib/src/ or /transferred/lib/src? I bet the latter. What's wrong with lib_src anyway? I don't get it to be honest. It indicates that the code comes from the mutual space. And I don't see it as problematic at all.

instead of lib/double-conversion we have 3rd-party/double-conversion (or something similar)?

Do we have something like lib/double-conversion? I don't see anything like it, can you specify what you mean, please?

The idea would be to have the same directory structure inside Docker that we have in the repo (so CMake would work just as well inside or outside Docker), and for anything additional that is needed inside Docker we would put it in a folder that doesn't overlap with any folder in the repo.

The idea's good but as I said above, this is something opposite to what you stated in another diff so we should decide what approach do we take.

In D3909#109802, @karol-bisztyga wrote:

I also noticed there is a lib inside the Docker container that is different from our services/lib.

I don't understand what is the problem here? Do you mean /lib? This folder is probably in every Unix. All contents of services/lib are being copied to transferred. So the folder structure is flattened. Something like this happens:

cp -r services/lib/* /transferred
cp -r services/backup/* /transferred # or blob, depends on the service

I couldn't name this src as it would get overwritten by specific service's src.

I think I recall this lib folder having third-party dependencies. I guess it just feels confusing to have such a generic name, especially given it means something different inside of Docker versus in our repo.

I think it's more simple to use the same directory structure we have in the repo. That way we don't have to reason about two distinct directory structures.

Ok but at the same time in D3904 you said

I don't think it's crazy for the folder to have a different name inside vs. outside the container, and the extra line in the Dockerfile isn't a big deal in my opinion.

So I think we should be consistent here. Either we take a laid back approach and allow ourselves to modify the structure or we keep it tight. I see some opposite ideas here, do I miss something?

Yeah you're absolutely right, this was inconsistent of me. After thinking about it more – I think it would be ideal if we could preserve the same structure between the repo and what's inside Docker. This will have the additional benefit of making things more consistent when we want to build outside of Docker, which I think is a goal based on recent research from @geekbrother.

What do you think of trying to preserve the same structure, and to use either .dockerignore or selective COPY instructions in the Dockerfile to copy over just what we need?

What do you think of changing the code so that instead of lib_src we have lib/src

Do you mean /lib/src/ or /transferred/lib/src? I bet the latter. What's wrong with lib_src anyway? I don't get it to be honest. It indicates that the code comes from the mutual space. And I don't see it as problematic at all.

I don't think lib_src is a good name. From the name I don't know what's going on it with, what it means, why it exists. To be honest, lib/src isn't much better though...

Right now on master we have a folder services/lib that inside it only has a single folder lib_src. This is the sort of deeply nested, poorly named structure that I've been pushing us for months to avoid. I think the directory structure has recently been improved but I wish that I didn't have to push for it – that you could own the process of coming up with a consistent, minimally nested, clearly named directory structure that could be mirrored between Docker and outside of it.

instead of lib/double-conversion we have 3rd-party/double-conversion (or something similar)?

Do we have something like lib/double-conversion? I don't see anything like it, can you specify what you mean, please?

I don't have time to dig up why I thought this was the case right now. git grep "lib/double-conversion" gives many results, so I suspect that should help clarify what I'm referring to.

Given the lib_src folder has already been created on master I'll accept this to unblock, but can you prioritize addressing the concerns laid out above?

The goal is:

consistent, minimally nested, clearly named directory structure that could be mirrored between Docker and outside of it

It might be a good idea to schedule a chat to talk through some ideas before putting up diffs for it.

This revision is now accepted and ready to land.May 5 2022, 9:11 PM

I think it's good to use .dockerignore, we can discuss this in https://linear.app/comm/issue/ENG-1089/dont-copy-dockerfile-into-service-images I think this is the right task for this.

So, long story short, the goal is to have the same structure on a docker container as on the host.

I don't think lib_src is a good name. From the name I don't know what's going on it with, what it means, why it exists. To be honest, lib/src isn't much better though...

Right now on master we have a folder services/lib that inside it only has a single folder lib_src. This is the sort of deeply nested, poorly named structure that I've been pushing us for months to avoid. I think the directory structure has recently been improved but I wish that I didn't have to push for it – that you could own the process of coming up with a consistent, minimally nested, clearly named directory structure that could be mirrored between Docker and outside of it.

This gives me nothing. You just dissed the existing structure. As long as you don't come up with an alternative I cannot read your mind and see what is your idea of an ideal directory structure. For me this structure's good. For you, not. Now, I can keep coming up with different names and you can keep hating every single one of them just saying that it's bad, nothing else. Where are we going to go with this?

As a side note - should we ever give negative feedback without an alternative? What values does it bring?

I don't understand why would you accept this? What is the purpose of accepting here if you think the directory structure is broken so much? What am I supposed to do with this? Land this, create a follow-up task, and hear again that I shouldn't do this again? I'm going to request changes I think.

Given the lib_src folder has already been created on master I'll accept this to unblock, but can you prioritize addressing the concerns laid out above?

What do you mean by "prioritize addressing the concerns laid out above"? Creating a task? A follow-up diff right away? Or maybe changing the folder name in this diff (if so, why this is accepted)?

See, I want to cooperate in the most efficient way possible, but for now, I have a feeling that anything I do will be wrong and you're going to have a problem with whatever path I take. The problem is, you don't specify what exactly you expect, and then I do something and it is usually not what you'd like, and then I get an essay from you on what I could do better, etc.

Back to the topic, I assume you wanted some follow-up without editing this directory name here. I don't understand why is that. You didn't have a problem with renaming scripts in D3906, so why don't we change the folder name here (as this is the first diff in this stack right now that modifies anything in this directory so it seems to be a good place)?

It's really frustrating that I have to keep writing these long essays... @karol-bisztyga the amount of time I spend writing essays on your diffs, generally repeating the same thing day-in-day-out, is deeply frustrating for me.

As long as you don't come up with an alternative I cannot read your mind and see what is your idea of an ideal directory structure. For me this structure's good. For you, not.

I am not always going to be able to give you better suggestions. This is something you'll need to get used to... your reviewer won't always know what the right solution is.

Right now, we have a huge issue where reviewing your diffs is very expensive. We need to move more of that work away from the reviewers, and to the diff author.

I should be able to tell you things like "this looks fishy" or "I don't like this, what are some alternatives?" without having to spend hours myself coming up with solutions.

Now, I can keep coming up with different names and you can keep hating every single one of them just saying that it's bad, nothing else. Where are we going to go with this?

I strongly disagree. You absolutely can come up with better names. All you need to do is to listen to the feedback, grok it, and try to respond to it.

I have spent a lot of time trying to explain this feedback to you. Are you still confused about the feedback? I am telling you (repeatedly, over the course of many diffs) the same thing:

  1. The structure should not be deeply nested
  2. The names of things should clearly explain what they are
  3. We should have the same structure within the Docker container as outside

I am not going to be able to give you the answer here. It's your job to take this feedback, consider it, grok it, and then to come back with a proposal that fits the feedback.

As a side note - should we ever give negative feedback without an alternative? What values does it bring?

We absolutely, definitely, 100% should do this. We need to do more of this. I have been pushing @palys-swm to do more of it. This idea you have that feedback should only be given when alternatives are presented is very unhealthy.

The value should be clear. If something is fishy / bad about the proposed code, the value is to improve it. The diff author should be able to take the feedback and independently come up with alternatives.

A senior engineer doesn't need solutions spelled out to them. They can be given a high-level problem, and they can come up with proposed solutions for the problem. That is what we are looking for here from you.

I don't understand why would you accept this? What is the purpose of accepting here if you think the directory structure is broken so much? What am I supposed to do with this? Land this, create a follow-up task, and hear again that I shouldn't do this again? I'm going to request changes I think.

Up to you. I made it clear why I accepted this when I accepted it, so I'm not sure what you're confused about. The directory structure has already been polluted, so I don't think this diff is making things worse.

See, I want to cooperate in the most efficient way possible, but for now, I have a feeling that anything I do will be wrong and you're going to have a problem with whatever path I take. The problem is, you don't specify what exactly you expect, and then I do something and it is usually not what you'd like, and then I get an essay from you on what I could do better, etc.

In these situations, you should ALWAYS schedule a meeting. I've given you this feedback before – if you're about to update a diff and you think it will lead to more wasted time (like I am currently wasting time), then PLEASE save us BOTH the time and schedule a meeting.

Back to the topic, I assume you wanted some follow-up without editing this directory name here. I don't understand why is that. You didn't have a problem with renaming scripts in D3906, so why don't we change the folder name here (as this is the first diff in this stack right now that modifies anything in this directory so it seems to be a good place)?

What I want is for you to come up with a directory structure that addresses the feedback I have repeatedly communicated to you. I am not sure what you are suggesting now (what folder name?).

I don't think a meeting is necessary - all doubts have been answered.

karol added inline comments.
services/backup/CMakeLists.txt
31 ↗(On Diff #12401)

This should be removed

services/blob/CMakeLists.txt
32 ↗(On Diff #12401)

This should be removed

remove non-existent paths from cmakelists

This revision is now accepted and ready to land.May 9 2022, 9:22 AM
tomek requested changes to this revision.May 10 2022, 1:06 AM

Why do we move BaseReactor.h here? Shouldn't we move only Item, just like the title suggests?

This revision now requires changes to proceed.May 10 2022, 1:06 AM

You are right, this is an oversight

I got information that Tomek is sick and I got a green light from Ashoat to land this. Therefore I'm removing Tomek from the review and landing this. @palys-swm if you ever want to return to this, feel free to do it.

This revision is now accepted and ready to land.May 10 2022, 6:32 AM
This revision was automatically updated to reflect the committed changes.