Summary:
This patch introduces stack frame compression to GWP-ASan. Each stack frame is
variable-length integer encoded as the difference between frame[i] and
frame[i - 1]. Furthermore, we use zig-zag encoding on the difference to ensure
that negative differences are also encoded into a relatively small number of
bytes.
Examples of what the compression looks like can be seen in
`gwp_asan/tests/compression.cpp`.
This compression can reduce the memory consumption cost of stack traces by
~50%.
Reviewers: vlad.tsyrklevich
Reviewed By: vlad.tsyrklevich
Subscribers: mgorny, #sanitizers, llvm-commits, eugenis, morehouse
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D66189
llvm-svn: 369048
On Darwin we have two external symbolizers: atos and llvm-symbolizer.
atos was changed to use posix_spawn (instead of fork+execv) in a
previous commit [1]. Let's use posix_spawn for llvm-symbolizer as well.
Our hope is that eventually we can transition to posix_spawn on other
platforms too.
[1] 399408a92f
llvm-svn: 369021
Now that LLVM moved to C++14, `COMPILER_RT_HAS_STD_CXX11_FLAG` should
become `COMPILER_RT_HAS_STD_CXX14_FLAG`.
I ran into this issue when replacing llvm::make_unique with
std::make_unique in an X-ray unit test. We are correctly passing
`-std=c++14`, but this got overwritten further down the invocation by
the compiler-rt flags. Given that this unit test is using LLVM headers,
this is bound to break sooner than later, regardless of my change.
Differential revision: https://reviews.llvm.org/D66271
llvm-svn: 368960
The X-ray unit tests in compiler-rt are overriding the C++ version by
explicitly passing -std=c++11 in the compiler invocation. This poses a
problem as these tests are including LLVM headers that can now use C++14
features. I'm temporarily reverting this as I investigate the correct
solution.
llvm-svn: 368952
On Darwin, we currently use forkpty to communicate with the "atos"
symbolizer. There are several problems that fork[pty] has, e.g. that
after fork, interceptors are still active and this sometimes causes
crashes or hangs. This is especially problematic for TSan, which uses
interceptors for OS-provided locks and mutexes, and even Libc functions
use those.
This patch replaces forkpty with posix_spawn on Darwin. Since
posix_spawn doesn't fork (at least on Darwin), the interceptors are not
a problem. Another benefit is that we'll handle post-fork failures (e.g.
sandbox disallows "exec") gracefully now.
Related revisions and previous attempts that were blocked by or had to
be revered due to test failures:
https://reviews.llvm.org/D48451https://reviews.llvm.org/D40032
Reviewed By: kubamracek
Differential Revision: https://reviews.llvm.org/D65253
llvm-svn: 368947
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.
Differential revision: https://reviews.llvm.org/D66259
llvm-svn: 368946
Summary:
Android requires additional stats in mallinfo. While we can provide
right away the number of bytes mapped (Primary+Secondary), there was
no way to get the number of free bytes (only makes sense for the
Primary since the Secondary unmaps everything on deallocation).
An approximation could be `StatMapped - StatAllocated`, but since we
are mapping in `1<<17` increments for the 64-bit Primary, it's fairly
inaccurate.
So we introduce `StatFree` (note it's `Free`, not `Freed`!), which
keeps track of the amount of Primary blocks currently unallocated.
Reviewers: cferris, eugenis, vitalybuka, hctim, morehouse
Reviewed By: morehouse
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D66112
llvm-svn: 368866
Summary:
Updates the function signature and comments for backtracing (and printing
backtraces). This update brings GWP-ASan in line with future requirements for
stack frame compression, wherein the length of the trace is provided
explicitly, rather than relying on nullptr-termination.
Reviewers: vlad.tsyrklevich
Reviewed By: vlad.tsyrklevich
Subscribers: #sanitizers, llvm-commits, morehouse
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D66099
llvm-svn: 368619
Summary:
This bug occurred when a plug-in requested that a binary not be
symbolized while the script is trying to symbolize a stack frame. In
this case `self.frame_no` would not be incremented. This would cause
subsequent stack frames that are symbolized to be incorrectly numbered.
To fix this `get_symbolized_lines()` has been modified to take an
argument that indicates whether the stack frame counter should
incremented. In `process_line_posix()` `get_symbolized_lines(None, ...)`
is now used in in the case where we don't want to symbolize a line so
that we can keep the frame counter increment in a single function.
A test case is included. The test uses a dummy plugin that always asks
`asan_symbolize.py` script to not symbolize the first binary that the
script asks about. Prior to the patch this would cause the output to
script to look something like
```
#0 0x0
#0 0x0 in do_access
#1 0x0 in main
```
This is the second attempt at landing this patch. The first (r368373)
failed due to failing some android bots and so was reverted in r368472.
The new test is now disabled for Android. It turns out that the patch
also fails for iOS too so it is also disabled for that family of
platforms too.
rdar://problem/49476995
Reviewers: kubamracek, yln, samsonov, dvyukov, vitalybuka
Subscribers: #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D65495
llvm-svn: 368603
Summary:
Few corrections with no functional change:
- replacing `%zd` with `%zu` all around: the values are unsigned
- prefer `MAP_ANONYMOUS` to `MAP_ANON` (it's deprecated)
- remove the unused `enum LinkerInitialized`
- mark a parameter as `UNUSED` in Fuchsia's `getRandom`
- correct the casing of a variable and use `nullptr` instead of 0 for
pointers in `list.h`
- reorder some `typedef` to be consistent between `signed` and
`unsigned`
Reviewers: eugenis, vitalybuka, morehouse, hctim
Reviewed By: vitalybuka, morehouse
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D65660
llvm-svn: 368585
Summary:
Because the dynamic linker for 32-bit executables on 64-bit FreeBSD uses
the environment variable `LD_32_LIBRARY_PATH` instead of
`LD_LIBRARY_PATH` to find needed dynamic libraries, running the 32-bit
parts of the dynamic ASan tests will fail with errors similar to:
```
ld-elf32.so.1: Shared object "libclang_rt.asan-i386.so" not found, required by "Asan-i386-inline-Dynamic-Test"
```
This adds support for setting up `LD_32_LIBRARY_PATH` for the unit and
regression tests. It will likely also require a minor change to the
`TestingConfig` class in `llvm/utils/lit/lit`.
Reviewers: emaste, kcc, rnk, arichardson
Reviewed By: arichardson
Subscribers: kubamracek, krytarowski, fedor.sergeev, delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D65772
llvm-svn: 368516
Ensure that malloc_default_zone and malloc_zone_from_ptr return the
sanitizer-installed malloc zone even when MallocStackLogging (MSL) is
requested. This prevents crashes in certain situations. Note that the
sanitizers and MSL cannot be used together. If both are enabled, MSL
functionality is essentially deactivated since it only hooks the default
allocator which is replaced by a custom sanitizer allocator.
rdar://53686175
Reviewed By: kubamracek
Differential Revision: https://reviews.llvm.org/D65990
llvm-svn: 368492
Summary:
The purpose is to be able to extract the number of new edges added to
the original (i.e. output) corpus directory after doing the merge. Use case
example: in ClusterFuzz, we do merge after every fuzzing session, to avoid
uploading too many corpus files, and we also record coverage stats at that
point. Having a separate line indicating stats after reading the initial output
corpus directory would make the stats extraction easier for both humans and
parsing scripts.
Context: https://github.com/google/clusterfuzz/issues/802.
Reviewers: morehouse, hctim
Reviewed By: hctim
Subscribers: delcypher, #sanitizers, llvm-commits, kcc
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D66020
llvm-svn: 368461
Summary:
This bug occurred when a plug-in requested that a binary not be
symbolized while the script is trying to symbolize a stack frame. In
this case `self.frame_no` would not be incremented. This would cause
subsequent stack frames that are symbolized to be incorrectly numbered.
To fix this `get_symbolized_lines()` has been modified to take an
argument that indicates whether the stack frame counter should
incremented. In `process_line_posix()` `get_symbolized_lines(None, ...)`
is now used in in the case where we don't want to symbolize a line so
that we can keep the frame counter increment in a single function.
A test case is included. The test uses a dummy plugin that always asks
`asan_symbolize.py` script to not symbolize the first binary that the
script asks about. Prior to the patch this would cause the output to
script to look something like
```
#0 0x0
#0 0x0 in do_access
#1 0x0 in main
```
rdar://problem/49476995
Reviewers: kubamracek, yln, samsonov, dvyukov, vitalybuka
Subscribers: #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D65495
llvm-svn: 368373
See D65364 for the code model requirements for tagged globals. Because
of the relocations used these requirements cannot be checked at link
time so they must be checked at runtime.
Differential Revision: https://reviews.llvm.org/D65968
llvm-svn: 368351
Summary:
Also slightly cleaned up the comments and changed the header's extension
back to `.h` as per comments on https://reviews.llvm.org/D65812.
New methods added:
* `ConsumeProbability` returns [0.0, 1.0] by consuming an unsigned integer value
from the input data and dividing that value by the integer's max value.
* `ConsumeFloatingPointInRange` returns a floating point value in the given
range. Relies on `ConsumeProbability` method. This method does not have the
limitation of `std::uniform_real_distribution` that requires the given range
to be <= the floating point type's max. If the range is too large, this
implementation will additionally call `ConsumeBool` to decide whether the
result will be in the first or the second half of the range.
* `ConsumeFloatingPoint` returns a floating point value in the range
`[std::numeric_limits<T>::lowest(), std::numeric_limits<T>::min()]`.
Tested on Linux, Mac, Windows.
Reviewers: morehouse
Reviewed By: morehouse
Subscribers: kubamracek, mgorny, dberris, delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D65905
llvm-svn: 368331
The fallback to the alternative implementation of TSD with TLS
is only needed for the static version of ASan for NetBSD.
The same code cannot be reused for the dynamic version of ASan as
TLS breaks and TSD code works.
llvm-svn: 368219
HWASan+globals build fix in rL368111 unfortunately didn't fix the
problem when clang_cflags specified -fuse-ld=ld.gold. Change the order
to force lld in an attempt to fix the Android sanitizer bot.
llvm-svn: 368218
We're using relocations that are unsupported by the version of gold on the
bot, so force the use of lld. One of the tests is already using lld,
so this should be safe.
llvm-svn: 368111
Summary:
It appears that since https://reviews.llvm.org/D54889, BackgroundThread()
crashes immediately because cur_thread()-> will return a null pointer
which is then dereferenced. I'm not sure why I only see this issue on
FreeBSD and not Linux since it should also be unintialized on other platforms.
Reviewers: yuri, dvyukov, dim, emaste
Subscribers: kubamracek, krytarowski, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D65705
llvm-svn: 368103
Globals are instrumented by adding a pointer tag to their symbol values
and emitting metadata into a special section that allows the runtime to tag
their memory when the library is loaded.
Due to order of initialization issues explained in more detail in the comments,
shadow initialization cannot happen during regular global initialization.
Instead, the location of the global section is marked using an ELF note,
and we require libc support for calling a function provided by the HWASAN
runtime when libraries are loaded and unloaded.
Based on ideas discussed with @evgeny777 in D56672.
Differential Revision: https://reviews.llvm.org/D65770
llvm-svn: 368102
in madvise mode, the shadow pages will be migrated only via madvise explicit calls.
Reviewers: vitalybuka
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D65775
llvm-svn: 368090
Summary:
.hpp makes more sense for this header as it's C++ only, plus it
contains the actual implementation.
Reviewers: Dor1s
Reviewed By: Dor1s
Subscribers: kubamracek, dberris, mgorny, delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D65812
llvm-svn: 368054
Third landing attempt: Added "if (HAVE_LIBCXX)" to keep Green Dragon green.
Haven't found a better way to pass the libcxx include path for building
compiler-rt with libcxx; this seems to be missing only for xray.
Differential Revision: https://reviews.llvm.org/D65307
llvm-svn: 367962
r356153 changed default build option of static libcxx to no PIC. We now
need to explicitly specify CMAKE_POSITION_INDEPENDENT_CODE to get PIC
libcxx.
Differential Revision: https://reviews.llvm.org/D65773
llvm-svn: 367943
Once we start instrumenting globals, all addresses including those of string literals
that we pass to the operating system will start being tagged. Since we can't rely
on the operating system to be able to cope with these addresses, we need to untag
them before passing them to the operating system. This change introduces a macro
that does so and uses it everywhere it is needed.
Differential Revision: https://reviews.llvm.org/D65768
llvm-svn: 367938
Summary:
FuzzedDataProvider is a helper class for writing fuzz targets that fuzz
multple inputs simultaneously. The header is supposed to be used for fuzzing
engine agnostic fuzz targets (i.e. the same target can be used with libFuzzer,
AFL, honggfuzz, and other engines). The common thing though is that fuzz targets
are typically compiled with clang, as it provides all sanitizers as well as
different coverage instrumentation modes. Therefore, making this FDP class a
part of the compiler-rt installation package would make it easier to develop
and distribute fuzz targets across different projects, build systems, etc.
Some context also available in https://github.com/google/oss-fuzz/pull/2547.
This CL does not delete the header from `lib/fuzzer/utils` directory in order to
provide the downstream users some time for a smooth migration to the new
header location.
Reviewers: kcc, morehouse
Reviewed By: morehouse
Subscribers: lebedev.ri, kubamracek, dberris, mgorny, delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D65661
llvm-svn: 367917
SanitizerCommon.PthreadDestructorIterations currently FAILs on Solaris:
[ RUN ] SanitizerCommon.PthreadDestructorIterations
/vol/llvm/src/compiler-rt/local/lib/sanitizer_common/tests/sanitizer_posix_test.cc:58: Failure
Value of: destructor_executed
Actual: true
Expected: false
[ FAILED ] SanitizerCommon.PthreadDestructorIterations (1 ms)
It turns out that destructor is called 4 times after the first call to SpawnThread, but
5 times after the second. While PTHREAD_DESTRUCTOR_ITERATIONS is 4 in
<limits.h>, the Solaris pthread_key_create(3C) man page documents
If, after all the destructors have been called for all keys with non-
null values, there are still some keys with non-null values, the
process will be repeated. POSIX requires that this process be executed
at least PTHREAD_DESTRUCTOR_ITERATIONS times. Solaris calls the
destructors repeatedly until all values with associated destructors are
NULL. Destructors that set new values can cause an infinite loop.
The patch adjusts the test case to allow for this.
Tested on x86_64-pc-solaris2.11.
Differential Revision: https://reviews.llvm.org/D65055
llvm-svn: 367705
Like r367463, but for tsan/{benchmarks,dd,go}.
The files benchmarks aren't referenced in the build anywhere and where added
in 2012 with the comment "no Makefiles yet".
llvm-svn: 367567
A bot complains:
/b/sanitizer-x86_64-linux-autoconf/build/llvm/projects/compiler-rt/lib/sanitizer_common/tests/malloc_stress_transfer_test.cpp:2: Streams are highly discouraged. [readability/streams] [3]
/b/sanitizer-x86_64-linux-autoconf/build/llvm/projects/compiler-rt/lib/sanitizer_common/tests/sanitizer_libc_test.cpp:11: Streams are highly discouraged. [readability/streams] [3]
lib/CMakeFiles/SanitizerLintCheck.dir/build.make:57: recipe for target 'lib/CMakeFiles/SanitizerLintCheck' failed
I do not know why this apparently wasn't a problem when the files
had extension .cc.
llvm-svn: 367493
See https://reviews.llvm.org/D58620 for discussion, and for the commands
I ran. In addition I also ran
for f in $(svn diff | diffstat | grep .cc | cut -f 2 -d ' '); do rg $(basename $f) . ; done
and manually updated references to renamed files found by that.
llvm-svn: 367467
See https://reviews.llvm.org/D58620 for discussion, and for the commands
I ran. In addition I also ran
for f in $(svn diff | diffstat | grep .cc | cut -f 2 -d ' '); do rg $f . ; done
and manually updated (many) references to renamed files found by that.
llvm-svn: 367463
See https://reviews.llvm.org/D58620 for discussion, and for the commands
I ran. In addition I also ran
for f in $(svn diff | diffstat | grep .cc | cut -f 2 -d ' '); do rg $f . ; done
and manually updated references to renamed files found by that.
llvm-svn: 367456
See https://reviews.llvm.org/D58620 for discussion, and for the commands
I ran. In addition I also ran
for f in $(svn diff | diffstat | grep .cc | cut -f 2 -d ' '); do rg $f . ; done
and manually updated references to renamed files found by that.
llvm-svn: 367452
compiler-rt's builtin library has generic implementations of many
functions, and then per-arch optimized implementations of some.
In the CMake build, both filter_builtin_sources() and an explicit loop
at the end of the build file (see D37166) filter out the generic
versions if a per-arch file is present.
The GN build wasn't doing this filtering. Just do the filtering manually
and explicitly, instead of being clever.
While here, also remove files from the mingw/arm build that are
redundantly listed after D39938 / r318139 (both from the CMake and the
GN build).
While here, also fix a target_os -> target_cpu typo.
Differential Revision: https://reviews.llvm.org/D65512
llvm-svn: 367448
Summary:
MSAN was broken on FreeBSD by https://reviews.llvm.org/D55703: after this
change accesses to the key variable call __tls_get_addr, which is
intercepted. The interceptor then calls GetCurrentThread which calls
MsanTSDGet which again calls __tls_get_addr, etc...
Using the default implementation in the SANITIZER_FREEBSD case fixes MSAN
for me.
I then applied the same change to ASAN (introduced in https://reviews.llvm.org/D55596)
but that did not work yet. In the ASAN case, we get infinite recursion
again during initialization, this time because calling pthread_key_create() early on
results in infinite recursion. pthread_key_create() calls sysctlbyname()
which is intercepted but COMMON_INTERCEPTOR_NOTHING_IS_INITIALIZED returns
true, so the interceptor calls internal_sysctlbyname() which then ends up
calling the interceptor again. I fixed this issue by using dlsym() to get
the libc version of sysctlbyname() instead.
This fixes https://llvm.org/PR40761
Reviewers: vitalybuka, krytarowski, devnexen, dim, bsdjhb, #sanitizers, MaskRay
Reviewed By: MaskRay
Subscribers: MaskRay, emaste, kubamracek, jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D65221
llvm-svn: 367442
Mitch Phillips