Linker issue when indirect branch obfuscation is active #34

NewDwarf · 2023-01-10T12:58:45Z

I faced with the interesting issue specific to the indirect branch obfuscation in c++ code.
The linker reports

ld: error: relocation refers to a discarded section: .text._ZNSt6__ndk19allocatorIcE10deallocateEPcm
>>> defined in second.o
>>> section group signature: _ZNSt6__ndk19allocatorIcE10deallocateEPcm
>>> prevailing definition is in first.o
>>> referenced by second.cpp
>>>               second.o:(.data+0xD28)

ld: error: relocation refers to a discarded section: .text.__clang_call_terminate
>>> defined in second.o
>>> section group signature: __clang_call_terminate
>>> prevailing definition is in first.o
>>> referenced by second.cpp
>>>               second.o:(.data+0xD38)
>>> referenced by second.cpp
>>>               second.o:(.data+0xD40)
>>> referenced by second.cpp
>>>               second.o:(.data+0xD48)
>>> referenced 13 more times

The data references here are the offset to the IndirectBranchingTargetAddress table.
The issue can be reproduced using only the -mllvm -enable-indibran parameter.

Here is the sample files first.cpp and second.cpp used to create the shared library.

first.cpp

#include <iostream>
#include <string>

int first()
{
  char buffer[20];
  std::string str ("Test string...");
  std::size_t length = str.copy(buffer, 6, 5);
  buffer[length] = '\0';
  std::cout << "buffer contains: " << buffer << '\n';
  return 0;
}

second.cpp

#include <iostream>
#include <string>

int second ()
{
  std::string base="this is a test string.";
  std::string str2="n example";
  std::string str3="sample phrase";
  std::string str4="useful.";

  // replace signatures used in the same order as described above:

  // Using positions:                 0123456789*123456789*12345
  std::string str=base;           // "this is a test string."
  str.replace(9,5,str2);          // "this is an example string." (1)
  str.replace(19,6,str3,7,6);     // "this is an example phrase." (2)
  str.replace(8,10,"just a");     // "this is just a phrase."     (3)
  str.replace(8,6,"a shorty",7);  // "this is a short phrase."    (4)
  str.replace(22,1,3,'!');        // "this is a short phrase!!!"  (5)

  // Using iterators:                                               0123456789*123456789*
  str.replace(str.begin(),str.end()-3,str3);                    // "sample phrase!!!"      (1)
  str.replace(str.begin(),str.begin()+6,"replace");             // "replace phrase!!!"     (3)
  str.replace(str.begin()+8,str.begin()+14,"is coolness",7);    // "replace is cool!!!"    (4)
  str.replace(str.begin()+12,str.end()-4,4,'o');                // "replace is cooool!!!"  (5)
  str.replace(str.begin()+11,str.end(),str4.begin(),str4.end());// "replace is useful."    (6)
  std::cout << str << '\n';
  return 0;
}

Compile them

clang++ -target aarch64-none-linux-android21 -mllvm -enable-bcfobf -mllvm -enable-splitobf -mllvm -split_num=9 -O2 first.cpp -c -mllvm -enable-indibran
and
clang++ -target aarch64-none-linux-android21 -mllvm -enable-bcfobf -mllvm -enable-splitobf -mllvm -split_num=9 -O2 second.cpp -c -mllvm -enable-indibran

-mllvm -enable-bcfobf -mllvm -enable-splitobf -mllvm -split_num=9 are used to emit more code with the branches so that to give the chance to the indirect branch obfuscator to construct the IndirectBranchingTargetAddress table.

Finally, link the object files
clang++ -target aarch64-none-linux-android21 -fPIC -shared -Wl,-soname,libsample.so -o libsample.so first.o second.o

to reproduce the issue.

The text was updated successfully, but these errors were encountered:

61bcdefg · 2023-01-10T14:04:26Z

Can you reproduce this problem when building for iOS/MacOS? I don't have Android Studio installed.
Linking for macOS looks clear but I am not sure I did everything clearly.

Naville · 2023-01-10T14:27:51Z

Probably COMDAT issue, just remove all comdats

NewDwarf · 2023-01-10T14:34:02Z

Can you reproduce this problem when building for iOS/MacOS? I don't have Android Studio installed. Linking for macOS looks clear but I am not sure I did everything clearly.

You don't need Android Studio. Just install NDK for you host machine from https://developer.android.com/ndk/downloads
I guess any clang compiler will understand the -target aarch64-none-linux-android21
At least it works for me.

clang -target aarch64-none-linux-android21 -mllvm -enable-bcfobf -mllvm -enable-splitobf -mllvm -split_num=9 -O2 second.cpp -c -mllvm -enable-indibran --sysroot=/path/to/NDK/25/NDK/toolchains/llvm/prebuilt/darwin-x86_64/sysroot

clang -target aarch64-none-linux-android21 -mllvm -enable-bcfobf -mllvm -enable-splitobf -mllvm -split_num=9 -O2 first.cpp -c -mllvm -enable-indibran --sysroot=/path/to/NDK/25/NDK/toolchains/llvm/prebuilt/darwin-x86_64/sysroot

After that invoke the linker via NDK's clang

NDK/25/NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ -target aarch64-none-linux-android21 -fPIC -shared -Wl,-soname,libsample.so -o libsample.so first.o second.o

This can help to reproduce the issue quickly.

Naville · 2023-01-10T14:40:35Z

Can you reproduce this problem when building for iOS/MacOS? I don't have Android Studio installed.
Linking for macOS looks clear but I am not sure I did everything clearly.

MachO has no such concept as COMDAT, so this bug will never trigger on Darwin

61bcdefg · 2023-01-10T15:29:05Z

MachO has no such concept as COMDAT, so this bug will never trigger on Darwin

Thank you for your reply, I don't know anything about ELF files, I need some time to understand it and try to fix it.

Naville · 2023-01-10T15:33:44Z

MachO has no such concept as COMDAT, so this bug will never trigger on Darwin

Thank you for your reply, I don't know anything about ELF files, I need some time to understand it and try to fix it.

As mentioned, just remove all the COMDATs should fix the issue

NewDwarf · 2023-01-10T19:28:34Z

@NeHyci I did some tests to give you details to understand how to fix it.

Let's try to reproduce it as simple as possible.
Create three source files:

header.hpp

template <typename T> T myMax(T x, T y)
{
    return (x > y) ? x : y;
}

foo.cpp
int foo(int a, int b) { return myMax(a, b); }

bar.cpp
int bar(int a, int b) { return myMax(a, b); }

The linker will give such output:

ld: error: relocation refers to a discarded section: .text._Z5myMaxIiET_S0_S0_
>>> defined in bar.o
>>> section group signature: _Z5myMaxIiET_S0_S0_
>>> prevailing definition is in foo.o
>>> referenced by bar.cpp
>>>               bar.o:(.data+0x0)
>>> referenced by bar.cpp
>>>               bar.o:(.data+0x8)
>>> referenced by bar.cpp
>>>               bar.o:(.data+0x10)
>>> referenced 2 more times
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Let's generate human readable bitcode for both cpp files for normal situation (without indirect branch obfuscation).
foo.ll.ok:

; ModuleID = 'foo.cpp'
source_filename = "foo.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

; Function Attrs: mustprogress nofree norecurse nosync nounwind readnone uwtable willreturn
define dso_local i32 @_Z3fooii(i32 %0, i32 %1) local_unnamed_addr #0 {
  %3 = icmp sgt i32 %0, %1
  %4 = select i1 %3, i32 %0, i32 %1
  ret i32 %4
}

attributes #0 = { mustprogress nofree norecurse nosync nounwind readnone uwtable willreturn "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+neon,+outline-atomics" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 1, !"branch-target-enforcement", i32 0}
!2 = !{i32 1, !"sign-return-address", i32 0}
!3 = !{i32 1, !"sign-return-address-all", i32 0}
!4 = !{i32 1, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"clang version 14.0.0 (https://github.com/apple/llvm-project.git c33f34f65cd5eda385964c4176906d55d7236206)"}

bar.ll.ok:

; ModuleID = 'bar.cpp'
source_filename = "bar.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

; Function Attrs: mustprogress nofree norecurse nosync nounwind readnone uwtable willreturn
define dso_local i32 @_Z3barii(i32 %0, i32 %1) local_unnamed_addr #0 {
  %3 = icmp sgt i32 %0, %1
  %4 = select i1 %3, i32 %0, i32 %1
  ret i32 %4
}

attributes #0 = { mustprogress nofree norecurse nosync nounwind readnone uwtable willreturn "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+neon,+outline-atomics" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 1, !"branch-target-enforcement", i32 0}
!2 = !{i32 1, !"sign-return-address", i32 0}
!3 = !{i32 1, !"sign-return-address-all", i32 0}
!4 = !{i32 1, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"clang version 14.0.0 (https://github.com/apple/llvm-project.git c33f34f65cd5eda385964c4176906d55d7236206)"}

As we can see, there are no COMDAT

But the obfuscated by the indirect branch bitcode has COMDAT!!!
bar.ll.bad

; ModuleID = 'bar.cpp'
source_filename = "bar.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

@IndirectBranchingGlobalTable = private global [3 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %8), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %7), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %8)]
@HikariConditionalLocalIndirectBranchingTable = private global [2 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %7), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %8)]
@llvm.compiler.used = appending global [2 x i8*] [i8* bitcast ([2 x i8*]* @HikariConditionalLocalIndirectBranchingTable to i8*), i8* bitcast ([3 x i8*]* @IndirectBranchingGlobalTable to i8*)], section "llvm.metadata"

; Function Attrs: mustprogress nounwind uwtable
define dso_local i32 @_Z3barii(i32 %0, i32 %1) local_unnamed_addr #0 {
  %3 = tail call i32 @_Z5myMaxIiET_S0_S0_(i32 %0, i32 %1)
  ret i32 %3
}

; Function Attrs: mustprogress nounwind uwtable
define linkonce_odr dso_local i32 @_Z5myMaxIiET_S0_S0_(i32 %0, i32 %1) #0 comdat {
  %3 = icmp sgt i32 %0, %1
  %4 = zext i1 %3 to i64
  %5 = getelementptr [2 x i8*], [2 x i8*]* @HikariConditionalLocalIndirectBranchingTable, i64 0, i64 %4
  %6 = load i8**, i8** %5, align 8
  indirectbr i8** %6, [label %7, label %8]

7:                                                ; preds = %2
  br label %8

8:                                                ; preds = %2, %7
  %9 = phi i32 [ %1, %7 ], [ %0, %2 ]
  ret i32 %9
}

attributes #0 = { mustprogress nounwind uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+neon,+outline-atomics" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 1, !"branch-target-enforcement", i32 0}
!2 = !{i32 1, !"sign-return-address", i32 0}
!3 = !{i32 1, !"sign-return-address-all", i32 0}
!4 = !{i32 1, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"clang version 14.0.0 (https://github.com/apple/llvm-project.git c33f34f65cd5eda385964c4176906d55d7236206)"}

foo.ll.bad

; ModuleID = 'foo.cpp'
source_filename = "foo.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

@IndirectBranchingGlobalTable = private global [3 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %8), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %7), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %8)]
@HikariConditionalLocalIndirectBranchingTable = private global [2 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %7), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %8)]
@llvm.compiler.used = appending global [2 x i8*] [i8* bitcast ([2 x i8*]* @HikariConditionalLocalIndirectBranchingTable to i8*), i8* bitcast ([3 x i8*]* @IndirectBranchingGlobalTable to i8*)], section "llvm.metadata"

; Function Attrs: mustprogress nounwind uwtable
define dso_local i32 @_Z3fooii(i32 %0, i32 %1) local_unnamed_addr #0 {
  %3 = tail call i32 @_Z5myMaxIiET_S0_S0_(i32 %0, i32 %1)
  ret i32 %3
}

; Function Attrs: mustprogress nounwind uwtable
define linkonce_odr dso_local i32 @_Z5myMaxIiET_S0_S0_(i32 %0, i32 %1) #0 comdat {
  %3 = icmp sgt i32 %0, %1
  %4 = zext i1 %3 to i64
  %5 = getelementptr [2 x i8*], [2 x i8*]* @HikariConditionalLocalIndirectBranchingTable, i64 0, i64 %4
  %6 = load i8**, i8** %5, align 8
  indirectbr i8** %6, [label %7, label %8]

7:                                                ; preds = %2
  br label %8

8:                                                ; preds = %2, %7
  %9 = phi i32 [ %1, %7 ], [ %0, %2 ]
  ret i32 %9
}

attributes #0 = { mustprogress nounwind uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+neon,+outline-atomics" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 1, !"branch-target-enforcement", i32 0}
!2 = !{i32 1, !"sign-return-address", i32 0}
!3 = !{i32 1, !"sign-return-address-all", i32 0}
!4 = !{i32 1, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"clang version 14.0.0 (https://github.com/apple/llvm-project.git c33f34f65cd5eda385964c4176906d55d7236206)"}

@Naville How to deal with COMDAT here?
When we don't use the indirect branch obfuscation, foo.ll.ok and bar.ll.ok inline the template function from the header.hpp into the body of the caller functions. Everything clear in this scenario.

When we use indirect branch obfuscation, both foo.ll.bad and bar.ll.bad have definition of the template function from the header.hpp inside and it is marked as comdat. If we will remove the COMDAT mark from the template function definition, we will get duplicated definitions and linking also fails.

If we completely remove define linkonce_odr dso_local i32 @_Z5myMaxIiET_S0_S0_(i32 %0, i32 %1) #0 comdat
The linker can also fail in the case if we remove all function definitions.

NewDwarf · 2023-01-10T19:42:08Z

@Naville In my understanding, the root of the problem are the header files from STL which has a lot of definitions of small (2-3 lines) template functions which are normally always inlined.
When we run the indirect branch obfuscator, we make small template function defined in the header file too large to incorporate it in the caller and we have copies of these functions in each compilation unit.
These functions are marked as COMDAT and we get ld: error: relocation refers to a discarded section
Otherwise, we would get ld: error: duplicate symbol error

Probably, I didn't get your idea how to manage this issue...

NewDwarf · 2023-01-10T19:46:43Z

@Naville I guess, you meant that we have to do each function as local to the specific compilation unit. So it will be kind of the static function (from the C perspective)
So, if we change
define linkonce_odr dso_local i32 @Z5myMaxIiET_S0_S0(i32 %0, i32 %1) #0 comdat
on
define dso_local i32 @Z5myMaxIiET_S0_S0(i32 %0, i32 %1) #0
it will get invisible for all other object modules and linking will not fail.

Could you, please, confirm that?

And another one question.
$Z5myMaxIiET_S0_S0 = comdat any
To get rid of any COMDAT section in the object file, we have to remove it too?

NewDwarf · 2023-01-10T20:16:04Z

@NeHyci I edited manually both *bad.ll files with removing
$Z5myMaxIiET_S0_S0 = comdat any
and changing function declaration from
define linkonce_odr dso_local i32 @Z5myMaxIiET_S0_S0(i32 %0, i32 %1) #0 comdat
on
define dso_local i32 @Z5myMaxIiET_S0_S0(i32 %0, i32 %1) #0

After that, I was able to link the bitcode modules without issues!
So, you need to find a way to remove it programmatically.

Naville · 2023-01-11T02:16:04Z

define linkonce_odr dso_local i32 @Z5myMaxIiET_S0_S0(i32 %0, i32 %1) #0 comdat

I doubt changing linkage is required

NewDwarf · 2023-01-11T07:04:43Z

The main question for me is why inlining transformation won't work when the indirect branching is active.
Maybe it is because the inliner is not able to find the room to insert the function being inlined from the STL code into the function modified by the indirect branch obfuscator?
@Naville Could you confirm that the indirect branch obfuscator runs before inline transformation? If so, can we change this behavior so that to run the indirect branch obfuscator AFTER inlining, this would resolve the problem.

Naville · 2023-01-11T07:06:43Z

Could you confirm that the indirect branch obfuscator runs before inline transformation?

-debug-pass=Executions

If so, can we change this behavior so that to run the indirect branch obfuscator AFTER inlining

You can

Naville · 2023-01-11T07:08:36Z

this would resolve the problem.

Because your current pass pipeline is stupid, you need to think about where your pass fits the best in the pipeline

NewDwarf · 2023-01-11T07:23:12Z

@NeHyci I think the best solution for this issue would be pipeline management so that to run indirect branch obfuscation AFTER the inliner.

61bcdefg · 2023-01-11T07:59:24Z

@NewDwarf I'm still trying to build NDK....It was very slow

NewDwarf · 2023-01-11T08:14:20Z

@NewDwarf I'm still trying to build NDK....It was very slow

If you need any help to assist to build exactly the same version of clang as NDK has, just ping me.

61bcdefg · 2023-01-11T09:11:08Z

@NeHyci I think the best solution for this issue would be pipeline management so that to run indirect branch obfuscation AFTER the inliner.

But how do we ensure that the Pass Plugin executes after the inliner and it works for all pipelines?

NewDwarf · 2023-01-11T09:24:05Z

@NeHyci I think the best solution for this issue would be pipeline management so that to run indirect branch obfuscation AFTER the inliner.

But how do we ensure that the Pass Plugin executes after the inliner and it works for all pipelines?

It's simple. As @Naville mentioned, we have to use the hidden parameter -debug-pass=Executions
NDK/25/NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ -target aarch64-none-linux-android21 -O2 -c 2.cpp -mllvm -enable-indibran -mllvm -debug-pass=Executions
Above command gives you complete statistic.

61bcdefg · 2023-01-11T09:29:04Z

It's simple. As @Naville mentioned, we have to use the hidden parameter -debug-pass=Executions

NDK/25/NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ -target aarch64-none-linux-android21 -O2 -c 2.cpp -mllvm -enable-indibran -mllvm -debug-pass=Executions

Above command gives you complete statistic.

I mean if we use Obfuscation Pass as the LLVM Pass Plugin, how can we make it execute after AlwaysInliner in all pipelines? I've taken a cursory look at the source code in lib/Passes and this seems impossible.

Naville · 2023-01-11T09:30:18Z

Injection order is execution order for Transform passes

NewDwarf · 2023-01-11T09:37:28Z

Statistic output produced by the clang/opt

[2023-01-11 12:32:33.323157000] 0x7fda1d70e760     Executing Pass 'Partially inline calls to library functions' on Function '_Z6secondRNSt6__ndk112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEES6_'...
[2023-01-11 12:32:33.323169000] 0x7fda1d70e760      Freeing Pass 'Partially inline calls to library functions' on Function '_Z6secondRNSt6__ndk112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEES6_'...

The order is in llvm//lib/Transforms/Scalar/PartiallyInlineLibCalls.cpp

char PartiallyInlineLibCallsLegacyPass::ID = 0;
INITIALIZE_PASS_BEGIN(PartiallyInlineLibCallsLegacyPass,
                      "partially-inline-libcalls",
                      "Partially inline calls to library functions", false,
                      false)
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
INITIALIZE_PASS_END(PartiallyInlineLibCallsLegacyPass,
                    "partially-inline-libcalls",
                    "Partially inline calls to library functions", false, false)

Naville · 2023-01-11T09:39:39Z

This is not the pass you are looking for? This is for library calls

NewDwarf · 2023-01-11T09:41:14Z

This is not the pass you are looking for? This is for library calls

Forgot to mention, I have been already working with such code

#include <string>

int second (std::string &str1, std::string &str2)
{
  if (str1.compare(str2)) printf("Not equal\n");

  return 0;
}

Naville · 2023-01-11T09:42:08Z

Honestly don't understand your reasoning here anymore, we'll review the actual full patch when submit it

NewDwarf · 2023-01-11T09:45:39Z

Honestly don't understand your reasoning here anymore, we'll review the actual full patch when submit it

I am new in compilers.
...but how LLVM works very impressed me to learn it in more details.
This is why I am here.
...this is a main reason why my progress is too slow at this moment as I have to understand many concepts before in such complex project.

Naville · 2023-01-11T09:49:14Z

No worries, we will review your patch together. It's just information lost during our discussion here

NewDwarf · 2023-01-11T10:25:36Z

@Naville Could you, please, give a clue why I don't see "Enable Obfuscation" statistic in the -debug-pass=Executions output?
I guess, we have to get "Enable Obfuscation" in the output as the obfuscation pass is registered as
INITIALIZE_PASS_BEGIN(Obfuscation, "obfus", "Enable Obfuscation", true, true)

The naive approach is just add as the dependency "PartiallyInlineLibCallsLegacyPass" in the obfuscator pass registration

char Obfuscation::ID = 0;
INITIALIZE_PASS_BEGIN(Obfuscation, "obfus", "Enable Obfuscation", true, true)
INITIALIZE_PASS_DEPENDENCY(PartiallyInlineLibCallsLegacyPass);
INITIALIZE_PASS_DEPENDENCY(AntiClassDump);
INITIALIZE_PASS_DEPENDENCY(BogusControlFlow);
...

but the main question here - is PartiallyInlineLibCallsLegacyPass always passed to the pipeline?
I checked it quickly on the samples which produces no inlined code. PartiallyInlineLibCallsLegacyPass is executed in these cases as well.

Naville · 2023-01-11T10:49:23Z

"Enable Obfuscation" statistic in the -debug-pass=Executions output?

The output is the passname, which you change by overriding "StringRef getPassName() const" in LPM

e naive approach is just add as the dependency "PartiallyInlineLibCallsLegacyPass" in the obfuscator pass registration

Wrong approach,wrong pass

NewDwarf · 2023-01-12T05:23:49Z

I noticed that optimization level 0 (-O0) also inserts COMDAT's regardless of using the indirect branch obfuscation.
And such code (not obfuscated by indirect branching but compiled as -O0) is linked fine.

Using the same code

//header.hpp
template <typename T> T myMax(T x, T y) { return (x > y) ? x : y; }

// foo.cpp
#include "header.hpp"
int foo(int a, int b) { return myMax(a, b); }

// bar.cpp
#include "header.hpp"
int bar(int a, int b) { return myMax(a, b); }

The bitcode with -0O and DISABLED indirect branch obfuscation looks
foo.ll

; ModuleID = 'foo.cpp'
source_filename = "foo.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

; Function Attrs: mustprogress noinline optnone uwtable
define dso_local noundef i32 @_Z3fooii(i32 noundef %0, i32 noundef %1) #0 {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = call noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %5, i32 noundef %6)
  ret i32 %7
}

; Function Attrs: mustprogress noinline nounwind optnone uwtable
define linkonce_odr dso_local noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %0, i32 noundef %1) #1 comdat {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = icmp sgt i32 %5, %6
  br i1 %7, label %8, label %10

8:                                                ; preds = %2
  %9 = load i32, i32* %3, align 4
  br label %12

10:                                               ; preds = %2
  %11 = load i32, i32* %4, align 4
  br label %12

12:                                               ; preds = %10, %8
  %13 = phi i32 [ %9, %8 ], [ %11, %10 ]
  ret i32 %13
}

attributes #0 = { mustprogress noinline optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }
attributes #1 = { mustprogress noinline nounwind optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 8, !"branch-target-enforcement", i32 0}
!2 = !{i32 8, !"sign-return-address", i32 0}
!3 = !{i32 8, !"sign-return-address-all", i32 0}
!4 = !{i32 8, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"Android (dev, based on r450784d) clang version 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0cca074e9238af8b4106c30add4418f6)"}

and bar.ll

; ModuleID = 'bar.cpp'
source_filename = "bar.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

; Function Attrs: mustprogress noinline optnone uwtable
define dso_local noundef i32 @_Z3barii(i32 noundef %0, i32 noundef %1) #0 {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = call noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %5, i32 noundef %6)
  ret i32 %7
}

; Function Attrs: mustprogress noinline nounwind optnone uwtable
define linkonce_odr dso_local noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %0, i32 noundef %1) #1 comdat {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = icmp sgt i32 %5, %6
  br i1 %7, label %8, label %10

8:                                                ; preds = %2
  %9 = load i32, i32* %3, align 4
  br label %12

10:                                               ; preds = %2
  %11 = load i32, i32* %4, align 4
  br label %12

12:                                               ; preds = %10, %8
  %13 = phi i32 [ %9, %8 ], [ %11, %10 ]
  ret i32 %13
}

attributes #0 = { mustprogress noinline optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }
attributes #1 = { mustprogress noinline nounwind optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 8, !"branch-target-enforcement", i32 0}
!2 = !{i32 8, !"sign-return-address", i32 0}
!3 = !{i32 8, !"sign-return-address-all", i32 0}
!4 = !{i32 8, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"Android (dev, based on r450784d) clang version 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0cca074e9238af8b4106c30add4418f6)"}

In contrast, the code with ENABLED indirect branch obfuscation
foo.ll.bad

; ModuleID = 'foo.cpp'
source_filename = "foo.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

@IndirectBranchingGlobalTable = private global [3 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %12), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %17), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %22)]
@HikariConditionalLocalIndirectBranchingTable = private global [2 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %17), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %12)]
@llvm.compiler.used = appending global [2 x i8*] [i8* bitcast ([3 x i8*]* @IndirectBranchingGlobalTable to i8*), i8* bitcast ([2 x i8*]* @HikariConditionalLocalIndirectBranchingTable to i8*)], section "llvm.metadata"

; Function Attrs: mustprogress noinline optnone uwtable
define dso_local noundef i32 @_Z3fooii(i32 noundef %0, i32 noundef %1) #0 {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = call noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %5, i32 noundef %6)
  ret i32 %7
}

; Function Attrs: mustprogress noinline nounwind optnone uwtable
define linkonce_odr dso_local noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %0, i32 noundef %1) #1 comdat {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = icmp sgt i32 %5, %6
  %8 = zext i1 %7 to i32
  %9 = getelementptr [2 x i8*], [2 x i8*]* @HikariConditionalLocalIndirectBranchingTable, i32 0, i32 %8
  %10 = load i8*, i8** %9, align 8
  %11 = load i8**, i8** %9, align 8
  indirectbr i8** %11, [label %17, label %12]

12:                                               ; preds = %2
  %13 = load i32, i32* %3, align 4
  %14 = getelementptr [3 x i8*], [3 x i8*]* @IndirectBranchingGlobalTable, i32 0, i32 2
  %15 = load i8*, i8** %14, align 8
  %16 = load i8**, i8** %14, align 8
  indirectbr i8** %16, [label %22]

17:                                               ; preds = %2
  %18 = load i32, i32* %4, align 4
  %19 = getelementptr [3 x i8*], [3 x i8*]* @IndirectBranchingGlobalTable, i32 0, i32 2
  %20 = load i8*, i8** %19, align 8
  %21 = load i8**, i8** %19, align 8
  indirectbr i8** %21, [label %22]

22:                                               ; preds = %17, %12
  %23 = phi i32 [ %13, %12 ], [ %18, %17 ]
  ret i32 %23
}

attributes #0 = { mustprogress noinline optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }
attributes #1 = { mustprogress noinline nounwind optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 8, !"branch-target-enforcement", i32 0}
!2 = !{i32 8, !"sign-return-address", i32 0}
!3 = !{i32 8, !"sign-return-address-all", i32 0}
!4 = !{i32 8, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"Android (dev, based on r450784d) clang version 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0cca074e9238af8b4106c30add4418f6)"}

and bar.ll.bad

; ModuleID = 'bar.cpp'
source_filename = "bar.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

@IndirectBranchingGlobalTable = private global [3 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %12), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %17), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %22)]
@HikariConditionalLocalIndirectBranchingTable = private global [2 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %17), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %12)]
@llvm.compiler.used = appending global [2 x i8*] [i8* bitcast ([3 x i8*]* @IndirectBranchingGlobalTable to i8*), i8* bitcast ([2 x i8*]* @HikariConditionalLocalIndirectBranchingTable to i8*)], section "llvm.metadata"

; Function Attrs: mustprogress noinline optnone uwtable
define dso_local noundef i32 @_Z3barii(i32 noundef %0, i32 noundef %1) #0 {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = call noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %5, i32 noundef %6)
  ret i32 %7
}

; Function Attrs: mustprogress noinline nounwind optnone uwtable
define linkonce_odr dso_local noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %0, i32 noundef %1) #1 comdat {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = icmp sgt i32 %5, %6
  %8 = zext i1 %7 to i32
  %9 = getelementptr [2 x i8*], [2 x i8*]* @HikariConditionalLocalIndirectBranchingTable, i32 0, i32 %8
  %10 = load i8*, i8** %9, align 8
  %11 = load i8**, i8** %9, align 8
  indirectbr i8** %11, [label %17, label %12]

12:                                               ; preds = %2
  %13 = load i32, i32* %3, align 4
  %14 = getelementptr [3 x i8*], [3 x i8*]* @IndirectBranchingGlobalTable, i32 0, i32 2
  %15 = load i8*, i8** %14, align 8
  %16 = load i8**, i8** %14, align 8
  indirectbr i8** %16, [label %22]

17:                                               ; preds = %2
  %18 = load i32, i32* %4, align 4
  %19 = getelementptr [3 x i8*], [3 x i8*]* @IndirectBranchingGlobalTable, i32 0, i32 2
  %20 = load i8*, i8** %19, align 8
  %21 = load i8**, i8** %19, align 8
  indirectbr i8** %21, [label %22]

22:                                               ; preds = %17, %12
  %23 = phi i32 [ %13, %12 ], [ %18, %17 ]
  ret i32 %23
}

attributes #0 = { mustprogress noinline optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }
attributes #1 = { mustprogress noinline nounwind optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 8, !"branch-target-enforcement", i32 0}
!2 = !{i32 8, !"sign-return-address", i32 0}
!3 = !{i32 8, !"sign-return-address-all", i32 0}
!4 = !{i32 8, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"Android (dev, based on r450784d) clang version 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0cca074e9238af8b4106c30add4418f6)"}

NewDwarf · 2023-01-12T05:39:24Z

If we remove any references to the @IndirectBranchingGlobalTable and @HikariConditionalLocalIndirectBranchingTable from the bitcode, linking works fine.
So something wrong with these tables.
For example

; ModuleID = 'bar.cpp'
source_filename = "bar.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

;@IndirectBranchingGlobalTable = private global [3 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %12), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %17), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %22)]
;@HikariConditionalLocalIndirectBranchingTable = private global [2 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %17), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %12)]
;@llvm.compiler.used = appending global [2 x i8*] [i8* bitcast ([3 x i8*]* @IndirectBranchingGlobalTable to i8*), i8* bitcast ([2 x i8*]* @HikariConditionalLocalIndirectBranchingTable to i8*)], section "llvm.metadata"

; Function Attrs: mustprogress noinline optnone uwtable
define dso_local noundef i32 @_Z3barii(i32 noundef %0, i32 noundef %1) #0 {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = call noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %5, i32 noundef %6)
  ret i32 %7
}

; Function Attrs: mustprogress noinline nounwind optnone uwtable
define linkonce_odr dso_local noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %0, i32 noundef %1) #1 comdat {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = icmp sgt i32 %5, %6
  %8 = zext i1 %7 to i32

  ret i32 0
}

attributes #0 = { mustprogress noinline optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }
attributes #1 = { mustprogress noinline nounwind optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 8, !"branch-target-enforcement", i32 0}
!2 = !{i32 8, !"sign-return-address", i32 0}
!3 = !{i32 8, !"sign-return-address-all", i32 0}
!4 = !{i32 8, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"Android (dev, based on r450784d) clang version 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0cca074e9238af8b4106c30add4418f6)"}

Naville · 2023-01-12T05:40:21Z

Again, comdat issue

Naville · 2023-01-12T05:41:18Z

GlobalValue::setComdat(nullptr)

NewDwarf · 2023-01-12T06:01:28Z

@Naville Quick question to you.
I am trying to edit bitcode manually to understand better the original problem but I get such error on translating ll to bc

bar.ll.bad:36:14: error: explicit pointee type doesn't match operand's pointee type
  %11 = load i8**, i8** %9, align 8
             ^

If I correctly understood, this is because of opaque pointers.
How to fix this issue manually to make llvm-as happy and don't break the code semantic?

Naville · 2023-01-12T06:03:18Z

How to fix this issue manually to make llvm-as happy and don't break the code semantic?

Disable opaque pointer mode

NewDwarf · 2023-01-12T06:08:33Z

GlobalValue::setComdat(nullptr)

The problem here is who marks the functions as comdat.
I am almost sure this happens after running the indirect branch obfuscation and if traverse all functions in the module by
F->setComdat(nullptr);
immediately after running indirect branch obfuscation, I don't get the result as comdats will be inserted later.

NewDwarf · 2023-01-12T06:30:15Z

@Naville Thanks a lot for assistance. Removing comdats directly before running the indirect branch obfuscation helped!
Now linking works perfect.

NewDwarf · 2023-01-12T06:32:00Z

@NeHyci The patch is very simple

diff --git a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
index 97ac85b49..cd6bffca2 100755
--- a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
+++ b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
@@ -66,6 +66,7 @@ struct IndirectBranch : public FunctionPass {
     if (!this->initialized)
       initialize(*Func.getParent());
     errs() << "Running IndirectBranch On " << Func.getName() << "\n";
+    Func.setComdat(nullptr);
     vector<BranchInst *> BIs;
     for (Instruction &Inst : instructions(Func))
       if (BranchInst *BI = dyn_cast<BranchInst>(&Inst))

NewDwarf · 2023-01-12T06:41:27Z

@Naville But, anyway, it would be nice to understand original issue as the linker is able to resolve correctly comdats of any type. When the indirect branch tables are inserted, linking of comdats modules gets broken.
I tried to pass the -mllvm -opaque-pointers=0 parameter
NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ -target aarch64-none-linux-android21 -mllvm -opaque-pointers=0 -O0 -c foo.cpp -emit-llvm -S -mllvm -enable-indibran -o foo.ll.bad
but ll -> bc translation still doesn't work throwing the same error message

NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/llvm-as foo.ll.bad 
NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/llvm-as: foo.ll.bad:36:14: error: explicit pointee type doesn't match operand's pointee type (i8** vs i8*)
  %11 = load i8**, i8** %9, align 8
             ^

when I pass -mllvm -opaque-pointers=1, error message is changed

NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/llvm-as foo.ll.bad 
foo.ll.bad:8:53: warning: ptr type is only supported in -opaque-pointers mode
@IndirectBranchingGlobalTable = private global [3 x ptr] [ptr blockaddress(@_Z5myMaxIiET_S0_S0_, %12), ptr blockaddress(@_Z5myMaxIiET_S0_S0_, %17), ptr blockaddress(@_Z5myMaxIiET_S0_S0_, %22)]
                                                    ^
NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/llvm-as: foo.ll.bad:8:53: error: expected type
@IndirectBranchingGlobalTable = private global [3 x ptr] [ptr blockaddress(@_Z5myMaxIiET_S0_S0_, %12), ptr blockaddress(@_Z5myMaxIiET_S0_S0_, %17), ptr blockaddress(@_Z5myMaxIiET_S0_S0_, %22)]
                                                    ^

Naville · 2023-01-12T07:28:09Z

Wrong, if you use clang, you instruct clang to emit opaque pointer IR (or not), that -mllvm only configures the LLVM level option, CFE still emits IR in default type system

NewDwarf · 2023-01-12T07:32:54Z

Wrong, if you use clang, you instruct clang to emit opaque pointer IR (or not), that -mllvm only configures the LLVM level option, CFE still emits IR in default type system

NDK's clang doesn't accept the parameter -Xclang -no-opaque-pointers.

NewDwarf · 2023-01-12T10:10:26Z

Func.setComdat(nullptr); works only for

template <typename T> T myMax(T x, T y) { return (x > y) ? x : y; }

But it doesn't work for NDK STL. It is definitely necessary to look at the "PartiallyInlineLibCallsLegacyPass" direction and run the indirect branch obfuscator only when "PartiallyInlineLibCallsLegacyPass" finish its work.
@Naville Can we do this using existing obfuscator's scheduler or we need to create standalone indirect branch pass and push it in the pass pipeline after the "PartiallyInlineLibCallsLegacyPass" pass?

Naville · 2023-01-12T10:12:32Z

What on earth makes you think PartiallyInlineLibCallsLegacyPass is related, I'm curious

Naville · 2023-01-12T10:13:12Z

Func.setComdat(nullptr); works only for

Try the opposite, branchtable->setComdat(Func.getComdat())

NewDwarf · 2023-01-12T15:09:17Z

@Naville I had some more changes which filtered some symbols in the indirect branch pass. I removed them and checked again
Func.setComdat(nullptr);
I confirm that this solution works.

NewDwarf · 2023-01-12T15:11:30Z

@Naville Are there any clang/opt options to check LLVM transformation after the specific pass?
Say, to dump the bitcode of the specific module before transformation and after transformation.

NewDwarf · 2023-01-12T21:45:10Z

@Naville Are there any clang/opt options to check LLVM transformation after the specific pass? Say, to dump the bitcode of the specific module before transformation and after transformation.
Looks like -mllvm -print-after-all is exactly what I need!

NewDwarf · 2023-02-27T19:21:30Z

@NeHyci Did you have a chance to reproduce this issue?

61bcdefg · 2023-02-28T11:42:31Z

@NeHyci Did you have a chance to reproduce this issue?

I think this obfuscator is designed primarily for iOS/MacOS, so issues for other platforms are a low priority. And NDK takes a lot of time to build, I don't have a lot of free time.

NewDwarf · 2023-02-28T12:24:23Z

It takes ~1 hour to build from the scratch on my MacBook pro 15'' 2019. Each time, when you update the obfuscation specific codebase, recompiling takes about 1-2 minutes.
Of cause, it is up to you, but if you need some assistance for building NDK, just let me know.
As I already mentioned, below patch works for me.

diff --git a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
index 97ac85b49..cd6bffca2 100755
--- a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
+++ b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
@@ -66,6 +66,7 @@ struct IndirectBranch : public FunctionPass {
     if (!this->initialized)
       initialize(*Func.getParent());
     errs() << "Running IndirectBranch On " << Func.getName() << "\n";
+    Func.setComdat(nullptr);
     vector<BranchInst *> BIs;
     for (Instruction &Inst : instructions(Func))
       if (BranchInst *BI = dyn_cast<BranchInst>(&Inst))

BRYNHILDRINTHEDARKNESS · 2023-03-01T14:45:23Z

It takes ~1 hour to build from the scratch on my MacBook pro 15'' 2019. Each time, when you update the obfuscation specific codebase, recompiling takes about 1-2 minutes. Of cause, it is up to you, but if you need some assistance for building NDK, just let me know. As I already mentioned, below patch works for me.
diff --git a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
index 97ac85b49..cd6bffca2 100755
--- a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
+++ b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
@@ -66,6 +66,7 @@ struct IndirectBranch : public FunctionPass {
     if (!this->initialized)
       initialize(*Func.getParent());
     errs() << "Running IndirectBranch On " << Func.getName() << "\n";
+    Func.setComdat(nullptr);
     vector<BranchInst *> BIs;
     for (Instruction &Inst : instructions(Func))
       if (BranchInst *BI = dyn_cast<BranchInst>(&Inst))

您好，能否指导一下如何在ndk25中使用这个混淆？

BRYNHILDRINTHEDARKNESS · 2023-03-01T14:46:49Z

It takes ~1 hour to build from the scratch on my MacBook pro 15'' 2019. Each time, when you update the obfuscation specific codebase, recompiling takes about 1-2 minutes. Of cause, it is up to you, but if you need some assistance for building NDK, just let me know. As I already mentioned, below patch works for me.
diff --git a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
index 97ac85b49..cd6bffca2 100755
--- a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
+++ b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
@@ -66,6 +66,7 @@ struct IndirectBranch : public FunctionPass {
     if (!this->initialized)
       initialize(*Func.getParent());
     errs() << "Running IndirectBranch On " << Func.getName() << "\n";
+    Func.setComdat(nullptr);
     vector<BranchInst *> BIs;
     for (Instruction &Inst : instructions(Func))
       if (BranchInst *BI = dyn_cast<BranchInst>(&Inst))

我编译完后将clang、clang++、clang-format替换进ndk25中，另外补齐了缺失的.h，但用Android studio编译的时候还是会出错，请问我的方法正确吗？

Naville · 2023-03-01T15:30:48Z

It takes ~1 hour to build from the scratch on my MacBook pro 15'' 2019. Each time, when you update the obfuscation specific codebase, recompiling takes about 1-2 minutes. Of cause, it is up to you, but if you need some assistance for building NDK, just let me know. As I already mentioned, below patch works for me.
diff --git a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
index 97ac85b49..cd6bffca2 100755
--- a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
+++ b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
@@ -66,6 +66,7 @@ struct IndirectBranch : public FunctionPass {
     if (!this->initialized)
       initialize(*Func.getParent());
     errs() << "Running IndirectBranch On " << Func.getName() << "\n";
+    Func.setComdat(nullptr);
     vector<BranchInst *> BIs;
     for (Instruction &Inst : instructions(Func))
       if (BranchInst *BI = dyn_cast<BranchInst>(&Inst))
我编译完后将clang、clang++、clang-format替换进ndk25中，另外补齐了缺失的.h，但用Android studio编译的时候还是会出错，请问我的方法正确吗？

NDK官方的整套链路是一坨狗屎, 你需要照着原版的clang设置自己编译的那份的LLVM的三个版本号, 然后只替换几个关键文件.

EDIT: Replied in Telegram

y0xc · 2024-07-09T06:57:16Z

It's strange that before the commit of fix pass pipelines, I integrated it into NDK and used it normally. However, after that, I also encountered such issue

NewDwarf added the bug Something isn't working label Jan 10, 2023

Linker issue when indirect branch obfuscation is active #34

Linker issue when indirect branch obfuscation is active #34

Comments

NewDwarf commented Jan 10, 2023 • edited Loading

61bcdefg commented Jan 10, 2023 • edited by NewDwarf Loading

Naville commented Jan 10, 2023 • edited Loading

NewDwarf commented Jan 10, 2023

Naville commented Jan 10, 2023

61bcdefg commented Jan 10, 2023

Naville commented Jan 10, 2023

NewDwarf commented Jan 10, 2023

NewDwarf commented Jan 10, 2023

NewDwarf commented Jan 10, 2023 • edited Loading

NewDwarf commented Jan 10, 2023

Naville commented Jan 11, 2023

NewDwarf commented Jan 11, 2023

Naville commented Jan 11, 2023 • edited Loading

Naville commented Jan 11, 2023

NewDwarf commented Jan 11, 2023

61bcdefg commented Jan 11, 2023 • edited Loading

NewDwarf commented Jan 11, 2023

61bcdefg commented Jan 11, 2023 • edited Loading

NewDwarf commented Jan 11, 2023 • edited Loading

61bcdefg commented Jan 11, 2023 • edited Loading

Naville commented Jan 11, 2023

NewDwarf commented Jan 11, 2023

Naville commented Jan 11, 2023

NewDwarf commented Jan 11, 2023 • edited Loading

Naville commented Jan 11, 2023

NewDwarf commented Jan 11, 2023

Naville commented Jan 11, 2023

NewDwarf commented Jan 11, 2023

Naville commented Jan 11, 2023 • edited Loading

NewDwarf commented Jan 12, 2023

NewDwarf commented Jan 12, 2023 • edited Loading

Naville commented Jan 12, 2023

Naville commented Jan 12, 2023

NewDwarf commented Jan 12, 2023

Naville commented Jan 12, 2023 • edited Loading

NewDwarf commented Jan 12, 2023

NewDwarf commented Jan 12, 2023

NewDwarf commented Jan 12, 2023

NewDwarf commented Jan 12, 2023

Naville commented Jan 12, 2023

NewDwarf commented Jan 12, 2023

NewDwarf commented Jan 12, 2023 • edited Loading

Naville commented Jan 12, 2023

Naville commented Jan 12, 2023

NewDwarf commented Jan 12, 2023

NewDwarf commented Jan 12, 2023

NewDwarf commented Jan 12, 2023

NewDwarf commented Feb 27, 2023

61bcdefg commented Feb 28, 2023 • edited Loading

NewDwarf commented Feb 28, 2023

BRYNHILDRINTHEDARKNESS commented Mar 1, 2023

BRYNHILDRINTHEDARKNESS commented Mar 1, 2023

Naville commented Mar 1, 2023 • edited Loading

y0xc commented Jul 9, 2024

NewDwarf commented Jan 10, 2023 •

edited

Loading

61bcdefg commented Jan 10, 2023 •

edited by NewDwarf

Loading

Naville commented Jan 10, 2023 •

edited

Loading

NewDwarf commented Jan 10, 2023 •

edited

Loading

Naville commented Jan 11, 2023 •

edited

Loading

61bcdefg commented Jan 11, 2023 •

edited

Loading

61bcdefg commented Jan 11, 2023 •

edited

Loading

NewDwarf commented Jan 11, 2023 •

edited

Loading

61bcdefg commented Jan 11, 2023 •

edited

Loading

NewDwarf commented Jan 11, 2023 •

edited

Loading

Naville commented Jan 11, 2023 •

edited

Loading

NewDwarf commented Jan 12, 2023 •

edited

Loading

Naville commented Jan 12, 2023 •

edited

Loading

NewDwarf commented Jan 12, 2023 •

edited

Loading

61bcdefg commented Feb 28, 2023 •

edited

Loading

Naville commented Mar 1, 2023 •

edited

Loading