how to integrate ninja and RBE to support remote build AOSP so fast?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

how to integrate ninja and RBE to support remote build AOSP so fast?

alexander zhu
hi,

Android Building group,

I noticed that One PPT called "Remote Execution and Caching for Android Platform" . 
Inside it has one page , it shows that ninja can call re.warper to execute clang/java on remote server.

In that PPT it shows the result is quite good from my point of view. So, I want to know how it works? how can i build one system like this? I have read pages https://cloud.google.com/remote-build-execution/docs/overview . I learned that RBE is for Bazel .

So I learned Bazel, I found that ,Bazel has its own BUILD file to be the input to compile code. So I confused ,could any one can tell me ? How does ninja call re-wrapper and send "compile" command to remote execution? How does this progress know the out file is exist in Remote Cache? 


I need to build one system like this to make the android and chips driver compile as fast as I can. thanks

<a imageanchor="1" href="about:invalid#zClosurez" style="clear: left; margin-bottom: 1em; float: left; margin-right: 1em;">google.png



--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to
[hidden email]
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/991eb2fb-8096-4c45-965f-04f5da2bc1ed%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: how to integrate ninja and RBE to support remote build AOSP so fast?

Dan Willemsen-2
Someone should be reaching out to you directly about RBE, but I'd like to clarify that the remote execution API has more clients than just Bazel (even though it is in the bazelbuild github organization). The rewrapper support in AOSP is unrelated to Bazel, and currently integrates with the build similarly to ccache -- as a compiler wrapper.

- Dan

On Monday, March 23, 2020 at 7:39:55 AM UTC-7 [hidden email] wrote:
hi,

Android Building group,

I noticed that One PPT called "Remote Execution and Caching for Android Platform" . 
Inside it has one page , it shows that ninja can call re.warper to execute clang/java on remote server.

In that PPT it shows the result is quite good from my point of view. So, I want to know how it works? how can i build one system like this? I have read pages https://cloud.google.com/remote-build-execution/docs/overview . I learned that RBE is for Bazel .

So I learned Bazel, I found that ,Bazel has its own BUILD file to be the input to compile code. So I confused ,could any one can tell me ? How does ninja call re-wrapper and send "compile" command to remote execution? How does this progress know the out file is exist in Remote Cache? 


I need to build one system like this to make the android and chips driver compile as fast as I can. thanks

google.png



--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to
[hidden email]
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/43f97c87-1404-4bd8-9d72-0fe4f7ac712d%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: how to integrate ninja and RBE to support remote build AOSP so fast?

alexander zhu
Hi,

Dan,

You are so kind . I got your point. I will try to learn it.

在 2020年3月24日星期二 UTC+8下午12:19:51,Dan Willemsen写道:
Someone should be reaching out to you directly about RBE, but I'd like to clarify that the <a href="https://github.com/bazelbuild/remote-apis" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fbazelbuild%2Fremote-apis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGoqNRy_AV0HuXl3yRopt4WApoZZw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fbazelbuild%2Fremote-apis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGoqNRy_AV0HuXl3yRopt4WApoZZw&#39;;return true;">remote execution API has more clients than just Bazel (even though it is in the bazelbuild github organization). The rewrapper support in AOSP is unrelated to Bazel, and currently integrates with the build similarly to ccache -- as a compiler wrapper.

- Dan

On Monday, March 23, 2020 at 7:39:55 AM UTC-7 [hidden email] wrote:
hi,

Android Building group,

I noticed that One PPT called "Remote Execution and Caching for Android Platform" . 
Inside it has one page , it shows that ninja can call re.warper to execute clang/java on remote server.

In that PPT it shows the result is quite good from my point of view. So, I want to know how it works? how can i build one system like this? I have read pages <a href="https://cloud.google.com/remote-build-execution/docs/overview" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://cloud.google.com/remote-build-execution/docs/overview&#39;;return true;" onclick="this.href=&#39;https://cloud.google.com/remote-build-execution/docs/overview&#39;;return true;">https://cloud.google.com/remote-build-execution/docs/overview . I learned that RBE is for Bazel .

So I learned Bazel, I found that ,Bazel has its own BUILD file to be the input to compile code. So I confused ,could any one can tell me ? How does ninja call re-wrapper and send "compile" command to remote execution? How does this progress know the out file is exist in Remote Cache? 


I need to build one system like this to make the android and chips driver compile as fast as I can. thanks

google.png



--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to
[hidden email]
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/bc2c3fb0-b718-45e3-badf-19db7558aca3%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: how to integrate ninja and RBE to support remote build AOSP so fast?

Yun Ma-2
In reply to this post by Dan Willemsen-2
Hi Dan,

Do we have more detail document about how to setup/use the RBE? We have interesting about this build feature.

Thanks
Yun

On Monday, March 23, 2020 at 9:19:51 PM UTC-7, Dan Willemsen wrote:
Someone should be reaching out to you directly about RBE, but I'd like to clarify that the <a href="https://github.com/bazelbuild/remote-apis" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fbazelbuild%2Fremote-apis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGoqNRy_AV0HuXl3yRopt4WApoZZw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fbazelbuild%2Fremote-apis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGoqNRy_AV0HuXl3yRopt4WApoZZw&#39;;return true;">remote execution API has more clients than just Bazel (even though it is in the bazelbuild github organization). The rewrapper support in AOSP is unrelated to Bazel, and currently integrates with the build similarly to ccache -- as a compiler wrapper.

- Dan

On Monday, March 23, 2020 at 7:39:55 AM UTC-7 [hidden email] wrote:
hi,

Android Building group,

I noticed that One PPT called "Remote Execution and Caching for Android Platform" . 
Inside it has one page , it shows that ninja can call re.warper to execute clang/java on remote server.

In that PPT it shows the result is quite good from my point of view. So, I want to know how it works? how can i build one system like this? I have read pages <a href="https://cloud.google.com/remote-build-execution/docs/overview" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://cloud.google.com/remote-build-execution/docs/overview&#39;;return true;" onclick="this.href=&#39;https://cloud.google.com/remote-build-execution/docs/overview&#39;;return true;">https://cloud.google.com/remote-build-execution/docs/overview . I learned that RBE is for Bazel .

So I learned Bazel, I found that ,Bazel has its own BUILD file to be the input to compile code. So I confused ,could any one can tell me ? How does ninja call re-wrapper and send "compile" command to remote execution? How does this progress know the out file is exist in Remote Cache? 


I need to build one system like this to make the android and chips driver compile as fast as I can. thanks

google.png



--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to
[hidden email]
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/44ad6fbe-9174-4752-af23-425621ad1bfb%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: how to integrate ninja and RBE to support remote build AOSP so fast?

Joe Hicks
Hi Yun,

I will reach out directly to you to discuss.

Thanks,
Joe

On Wednesday, May 13, 2020 at 3:24:55 PM UTC-4 Yun Ma wrote:
Hi Dan,

Do we have more detail document about how to setup/use the RBE? We have interesting about this build feature.

Thanks
Yun

On Monday, March 23, 2020 at 9:19:51 PM UTC-7, Dan Willemsen wrote:
Someone should be reaching out to you directly about RBE, but I'd like to clarify that the remote execution API has more clients than just Bazel (even though it is in the bazelbuild github organization). The rewrapper support in AOSP is unrelated to Bazel, and currently integrates with the build similarly to ccache -- as a compiler wrapper.

- Dan

On Monday, March 23, 2020 at 7:39:55 AM UTC-7 [hidden email] wrote:
hi,

Android Building group,

I noticed that One PPT called "Remote Execution and Caching for Android Platform" . 
Inside it has one page , it shows that ninja can call re.warper to execute clang/java on remote server.

In that PPT it shows the result is quite good from my point of view. So, I want to know how it works? how can i build one system like this? I have read pages https://cloud.google.com/remote-build-execution/docs/overview . I learned that RBE is for Bazel .

So I learned Bazel, I found that ,Bazel has its own BUILD file to be the input to compile code. So I confused ,could any one can tell me ? How does ninja call re-wrapper and send "compile" command to remote execution? How does this progress know the out file is exist in Remote Cache? 


I need to build one system like this to make the android and chips driver compile as fast as I can. thanks

google.png



--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to
[hidden email]
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/4fd3217c-49f0-4c48-908e-7d29142b68fb%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: how to integrate ninja and RBE to support remote build AOSP so fast?

Chang Kyo
Hi Joe:
I also want to know the detail about setup/use the RBE.

Does it mean if I have a lot of machine(virtual or physical),
I can separate the AOSP compile process on them?

Because as android version rising, the compile time rising also.
aosp_11_compile_time.png
i'm afraid it will impact our product develop time next year.

Thanks a lot.
Chang
Joe Hicks 在 2020年5月14日 星期四上午4:11:45 [UTC+8] 的信中寫道:
Hi Yun,

I will reach out directly to you to discuss.

Thanks,
Joe

On Wednesday, May 13, 2020 at 3:24:55 PM UTC-4 Yun Ma wrote:
Hi Dan,

Do we have more detail document about how to setup/use the RBE? We have interesting about this build feature.

Thanks
Yun

On Monday, March 23, 2020 at 9:19:51 PM UTC-7, Dan Willemsen wrote:
Someone should be reaching out to you directly about RBE, but I'd like to clarify that the remote execution API has more clients than just Bazel (even though it is in the bazelbuild github organization). The rewrapper support in AOSP is unrelated to Bazel, and currently integrates with the build similarly to ccache -- as a compiler wrapper.

- Dan

On Monday, March 23, 2020 at 7:39:55 AM UTC-7 [hidden email] wrote:
hi,

Android Building group,

I noticed that One PPT called "Remote Execution and Caching for Android Platform" . 
Inside it has one page , it shows that ninja can call re.warper to execute clang/java on remote server.

In that PPT it shows the result is quite good from my point of view. So, I want to know how it works? how can i build one system like this? I have read pages https://cloud.google.com/remote-build-execution/docs/overview . I learned that RBE is for Bazel .

So I learned Bazel, I found that ,Bazel has its own BUILD file to be the input to compile code. So I confused ,could any one can tell me ? How does ninja call re-wrapper and send "compile" command to remote execution? How does this progress know the out file is exist in Remote Cache? 


I need to build one system like this to make the android and chips driver compile as fast as I can. thanks

google.png



--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to
[hidden email]
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/80c2b419-81a3-44ea-92ee-5156a8a6140an%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: how to integrate ninja and RBE to support remote build AOSP so fast?

Joe Hicks
Thanks Chang Kyo. I will reach out to you directly to discuss.

Thanks again,

Joe

On Mon, Sep 14, 2020 at 11:41 AM Chang Kyo <[hidden email]> wrote:
Hi Joe:
I also want to know the detail about setup/use the RBE.

Does it mean if I have a lot of machine(virtual or physical),
I can separate the AOSP compile process on them?

Because as android version rising, the compile time rising also.
aosp_11_compile_time.png
i'm afraid it will impact our product develop time next year.

Thanks a lot.
Chang
Joe Hicks 在 2020年5月14日 星期四上午4:11:45 [UTC+8] 的信中寫道:
Hi Yun,

I will reach out directly to you to discuss.

Thanks,
Joe

On Wednesday, May 13, 2020 at 3:24:55 PM UTC-4 Yun Ma wrote:
Hi Dan,

Do we have more detail document about how to setup/use the RBE? We have interesting about this build feature.

Thanks
Yun

On Monday, March 23, 2020 at 9:19:51 PM UTC-7, Dan Willemsen wrote:
Someone should be reaching out to you directly about RBE, but I'd like to clarify that the remote execution API has more clients than just Bazel (even though it is in the bazelbuild github organization). The rewrapper support in AOSP is unrelated to Bazel, and currently integrates with the build similarly to ccache -- as a compiler wrapper.

- Dan

On Monday, March 23, 2020 at 7:39:55 AM UTC-7 [hidden email] wrote:
hi,

Android Building group,

I noticed that One PPT called "Remote Execution and Caching for Android Platform" . 
Inside it has one page , it shows that ninja can call re.warper to execute clang/java on remote server.

In that PPT it shows the result is quite good from my point of view. So, I want to know how it works? how can i build one system like this? I have read pages https://cloud.google.com/remote-build-execution/docs/overview . I learned that RBE is for Bazel .

So I learned Bazel, I found that ,Bazel has its own BUILD file to be the input to compile code. So I confused ,could any one can tell me ? How does ninja call re-wrapper and send "compile" command to remote execution? How does this progress know the out file is exist in Remote Cache? 


I need to build one system like this to make the android and chips driver compile as fast as I can. thanks

google.png



--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to
[hidden email]
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Android Building" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/android-building/ZnhRNvpgOg8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/80c2b419-81a3-44ea-92ee-5156a8a6140an%40googlegroups.com.


--
Joe Hicks
Product Manager, Google Core Developer

--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to
[hidden email]
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/CAMDFopgj6j4SH5Exm%2BzvJahJvoqHYgv78OGeLGs2buNK%2B-M00w%40mail.gmail.com.