Commit Graph

252 Commits

Author SHA1 Message Date
Lang Hames
543e0dcc0a [RuntimeDyld] Fix conservative over-allocation of memory for common symbols.
The previous allocation code was over-estimating the amount of memory required.

No test case: we don't currently have a good way to detect conervative
over-allocation.

llvm-svn: 267041
2016-04-21 20:08:06 +00:00
Kevin Enderby
81e8b7d949 Thread Expected<...> up from libObject’s getName() for symbols to allow llvm-objdump to produce a good error message.
Produce another specific error message for a malformed Mach-O file when a symbol’s
string index is past the end of the string table.  The existing test case in test/Object/macho-invalid.test
for macho-invalid-symbol-name-past-eof now reports the error with the message indicating
that a symbol at a specific index has a bad sting index and that bad string index value.
 
Again converting interfaces to Expected<> from ErrorOr<> does involve
touching a number of places. Where the existing code reported the error with a
string message or an error code it was converted to do the same.  There is some
code for this that could be factored into a routine but I would like to leave that for
the code owners post-commit to do as they want for handling an llvm::Error.  An
example of how this could be done is shown in the diff in
lib/ExecutionEngine/RuntimeDyld/RuntimeDyldImpl.h which had a Check() routine
already for std::error_code so I added one like it for llvm::Error .

Also there some were bugs in the existing code that did not deal with the
old ErrorOr<> return values.  So now with Expected<> since they must be
checked and the error handled, I added a TODO and a comment:
“// TODO: Actually report errors helpfully” and a call something like
consumeError(NameOrErr.takeError()) so the buggy code will not crash
since needed to deal with the Error.

Note there fixes needed to lld that goes along with this that I will commit right after this.
So expect lld not to built after this commit and before the next one.

llvm-svn: 266919
2016-04-20 21:24:34 +00:00
Kevin Enderby
5afbc1cda7 Fix a crash in running llvm-objdump -t with an invalid Mach-O file already
in the test suite. While this is not really an interesting tool and option to run
on a Mach-O file to show the symbol table in a generic libObject format
it shouldn’t crash.

The reason for the crash was in MachOObjectFile::getSymbolType() when it was
calling MachOObjectFile::getSymbolSection() without checking its return value
for the error case.

What makes this fix require a fair bit of diffs is that the method getSymbolType() is
in the class ObjectFile defined without an ErrorOr<> so I needed to add that all
the sub classes.  And all of the uses needed to be updated and the return value
needed to be checked for the error case.

The MachOObjectFile version of getSymbolType() “can” get an error in trying to
come up with the libObject’s internal SymbolRef::Type when the Mach-O symbol
symbol type is an N_SECT type because the code is trying to select from the
SymbolRef::ST_Data or SymbolRef::ST_Function values for the SymbolRef::Type.
And it needs the Mach-O section to use isData() and isBSS to determine if
it will return SymbolRef::ST_Data.

One other possible fix I considered is to simply return SymbolRef::ST_Other
when MachOObjectFile::getSymbolSection() returned an error.  But since in
the past when I did such changes that “ate an error in the libObject code” I
was asked instead to push the error out of the libObject code I chose not
to implement the fix this way.

As currently written both the COFF and ELF versions of getSymbolType()
can’t get an error.  But if isReservedSectionNumber() wanted to check for
the two known negative values rather than allowing all negative values or
the code wanted to add the same check as in getSymbolAddress() to use
getSection() and check for the error then these versions of getSymbolType()
could return errors.

At the end of the day the error printed now is the generic “Invalid data was
encountered while parsing the file” for object_error::parse_failed.  In the
future when we thread Lang’s new TypedError for recoverable error handling
though libObject this will improve.  And where the added // Diagnostic(…
comment is, it would be changed to produce and error message
like “bad section index (42) for symbol at index 8” for this case.

llvm-svn: 264187
2016-03-23 20:27:00 +00:00
Lang Hames
b0934294b2 [RuntimeDyld] Add a notifyObjectLoaded method to RuntimeDyld::MemoryManager.
This is a more generic version of the MCJITMemoryManager::notifyObjectLoaded
method: It provides only a RuntimeDyld reference (rather than an
ExecutionEngine), and so can be used with ORC JIT stacks.

llvm-svn: 257296
2016-01-10 23:59:41 +00:00
Lang Hames
b2b7a3c179 [RuntimeDyld] Add alignment arguments to the reserveAllocationSpace method of
RuntimeDyld::MemoryManager.

The RuntimeDyld::MemoryManager::reserveAllocationSpace method is called when
object files are loaded, and gives clients a chance to pre-allocate memory for
all segments. Previously only the size of each segment (code, ro-data, rw-data)
was supplied but not the alignment. This hasn't caused any problems so far, as
most clients allocate via the MemoryBlock interface which returns page-aligned
blocks. Adding alignment arguments enables finer grained allocation while still
satisfying alignment restrictions.

llvm-svn: 257294
2016-01-10 18:51:50 +00:00
Lang Hames
859d73ce95 [Orc][RuntimeDyld] Prevent duplicate calls to finalizeMemory on shared memory
managers.

Prior to this patch, recursive finalization (where finalization of one
RuntimeDyld instance triggers finalization of another instance on which the
first depends) could trigger memory access failures: When the inner (dependent)
RuntimeDyld instance and its memory manager are finalized, memory allocated
(but not yet relocated) by the outer instance is locked, and relocation in the
outer instance fails with a memory access error.

This patch adds a latch to the RuntimeDyld::MemoryManager base class that is
checked by a new method: RuntimeDyld::finalizeWithMemoryManagerLocking, ensuring
that shared memory managers are only finalized by the outermost RuntimeDyld
instance.

This allows ORC clients to supply the same memory manager to multiple calls to
addModuleSet. In particular it enables the use of user-supplied memory managers
with the CompileOnDemandLayer which must reuse the supplied memory manager for
each function that is lazily compiled.

llvm-svn: 257263
2016-01-09 19:50:40 +00:00
Keno Fischer
eb59d468d9 [RuntimeDyld] DenseMap -> std::unordered_map
DenseMap is most applicable when both keys and values are small.
In this case, the value violates that assumption, causing quite
significant memory overhead. A std::unordered_map is more appropriate
in this case (or at least fixed the memory problems I was seeing).

Differential Revision: http://reviews.llvm.org/D14910

llvm-svn: 254651
2015-12-03 21:27:59 +00:00
Sanjoy Das
d5658b0896 [RuntimeDyld] Don't allocate unnecessary stub buffer space
Summary:
For relocation types that are known to not require stub functions, there
is no need to allocate extra space for the stub functions.

Reviewers: lhames, reames, maksfb

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D14676

llvm-svn: 253920
2015-11-23 21:47:51 +00:00
Sanjoy Das
8082592ac9 [RuntimeDyld] Add bounds checking to SectionEntry::advanceStubOffset
Summary:
Change SectionEntry to keep track of the size of its underlying
allocation, and use that to bounds check advanceStubOffset.

Reviewers: lhames, andrew.w.kaylor, reames

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D14675

llvm-svn: 253919
2015-11-23 21:47:46 +00:00
Sanjoy Das
277776a520 [RuntimeDyld] Add accessors to SectionEntry; NFC
Summary:
Remove naked access to the data members in `SectionEntry` and route
accesses through accessor functions.  This makes it obvious how the
instances of the class are used, and will also facilitate adding bounds
checking to `advanceStubOffset` in a later change.

Reviewers: lhames, loladiro, andrew.w.kaylor

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D14674

llvm-svn: 253918
2015-11-23 21:47:41 +00:00
Keno Fischer
ddad187ce7 [RuntimeDyld] Ignore ST_FILE symbols when constructing GlobalSymbolTable
Summary: ELF's STT_File symbols may overlap with regular globals in
other files, so we should ignore them here in order to avoid having
bogus entries in the symbol table that confuse us when resolving relocations.

Reviewers: lhames

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D13888

llvm-svn: 250942
2015-10-21 20:22:04 +00:00
Lang Hames
a32d71be4c [RuntimeDyld] Add support for absolute symbols.
llvm-svn: 250639
2015-10-18 01:41:37 +00:00
Lang Hames
86a4593dd2 [RuntimeDyld] Don't try to get the contents of sections that don't have any
(e.g. bss sections).

MachO and ELF have been silently letting this pass, but COFFObjectFile contains
an assertion to catch this kind of (ab)use of the getSectionContents, and this
was causing the JIT to crash on COFF objects with BSS sections. This patch
should fix that.

llvm-svn: 250371
2015-10-15 06:41:45 +00:00
Keno Fischer
2cd66e9270 [RuntimeDyld] Fix performance problem in resolveRelocations with many sections
Summary:
Rather than just iterating over all sections and checking whether we have relocations for them, iterate over the relocation map instead. This showed up heavily in an artificial julia benchmark that does lots of compilation. On that particular benchmark, this patch gives
~15% performance improvements. As far as I can tell the primary reason why the original
loop was so expensive is that Relocations[i] actually constructs a relocationList (allocating memory & doing lots of other unnecessary computing) if none is found.

Reviewers: lhames

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D13545

llvm-svn: 249942
2015-10-10 05:37:02 +00:00
Lang Hames
79fce4711b [RuntimeDyld] Fix a bug in debugging output: all sections should be dumped
before any relocations have been applied, and again after all relocations have
been applied.

Previously each section was dumped before and after relocations targetting it
were applied, but this only shows the impact of relocations that point to other
symbols in the same section.

llvm-svn: 247335
2015-09-10 20:44:36 +00:00
Lang Hames
7e66c6c5e3 [RuntimeDyld] Make sure code-sections aren't under-aligned.
Code-section alignment should be at least as high as the minimum
stub alignment. If the section alignment is lower it can cause
padding to be emitted resulting in alignment errors if the section
is mapped to a higher alignment on the target.

E.g. If a text section with a 4-byte alignment gets 4-bytes of
padding to guarantee 8-byte alignment for stubs but is re-mapped to
an 8-byte alignment on the target, the 4-bytes of padding will push
the stubs to 4-byte alignment causing a crash.

No test case: There is currently no way to control host section
alignment in llvm-rtdyld. This could be made testable by adding
a custom memory manager. I'll look at that in a follow-up patch.

llvm-svn: 245031
2015-08-14 06:26:42 +00:00
Petar Jovanovic
d22164dc3b [mips][mcjit] Calculate correct addend for HI16 and PCHI16 reloc
Previously, for O32 ABI we did not calculate correct addend for R_MIPS_HI16
and R_MIPS_PCHI16 relocations. This patch fixes that.

Patch by Vladimir Radosavljevic.

Differential Revision: http://reviews.llvm.org/D11186

llvm-svn: 244897
2015-08-13 15:12:49 +00:00
Rafael Espindola
8bab889b0f Convert getSymbolSection to return an ErrorOr.
This function can actually fail since the symbol contains an index to the
section and that can be invalid.

llvm-svn: 244375
2015-08-07 23:27:14 +00:00
Lang Hames
2e88f4fc5f [RuntimeDyld] Make LoadedObjectInfo::getLoadedSectionAddress take a SectionRef
rather than a string section name.

llvm-svn: 243456
2015-07-28 17:52:11 +00:00
Rafael Espindola
ef888a4db6 Simplify by passing in the section of the symbol. NFC.
llvm-svn: 241603
2015-07-07 16:45:55 +00:00
Lang Hames
78937c2ae5 [RuntimeDyld] Skip relocations for external symbols with 64-bit address ~0ULL.
Requested by Eugene Rozenfeld of the LLILC team, this feature allows JIT
clients to skip relocations for selected external symbols by returning ~0ULL
from their symbol resolver. If this value is returned for a given symbol,
RuntimeDyld will skip all relocations for that symbol. The client will be
responsible for applying the skipped relocations manually before the code
is executed.

llvm-svn: 241383
2015-07-04 01:35:26 +00:00
Rafael Espindola
ed067c45d4 Return ErrorOr from getSymbolAddress.
It can fail trying to get the section on ELF and COFF. This makes sure the
error is handled.

llvm-svn: 241366
2015-07-03 18:19:00 +00:00
Rafael Espindola
5d0c2ffadf Return ErrorOr from SymbolRef::getName.
This function can really fail since the string table offset can be out of
bounds.

Using ErrorOr makes sure the error is checked.

Hopefully a lot of the boilerplate code in tools/* can go away once we have
a diagnostic manager in Object.

llvm-svn: 241297
2015-07-02 20:55:21 +00:00
Rafael Espindola
edd5f84419 Expose getFlags via ELFSectionRef.
llvm-svn: 240779
2015-06-26 12:44:10 +00:00
Rafael Espindola
41401e9c80 Add a ELFSectionRef class and use it to expose getSectionType.
llvm-svn: 240778
2015-06-26 12:33:37 +00:00
Rafael Espindola
2fa80cc5fd Simplify getSymbolType.
This is still a really odd function. Most calls are in object format specific
contexts and should probably be replaced with a more direct query, but at least
now this is not too obnoxious to use.

llvm-svn: 240777
2015-06-26 12:18:49 +00:00
Rafael Espindola
d7a32ea4b8 Change how symbol sizes are handled in lib/Object.
COFF and MachO only define symbol sizes for common symbols. Reflect that
in the class hierarchy by having a method for common symbols only in the base
and a general one in ELF.

This avoids the need of using a magic value for the size, which had a few
problems
* Most callers didn't check for it.
* The ones that did could not tell the magic value from a file actually having
  that value.

llvm-svn: 240529
2015-06-24 10:20:30 +00:00
Rui Ueyama
7d09919534 Remove object_error::success and use std::error_code() instead
make_error_code(object_error) is slow because object::object_category()
uses a ManagedStatic variable. But the real problem is that the function is
called too frequently. This patch uses std::error_code() instead of
object_error::success. In most cases, we return "success", so this patch
reduces number of function calls to that function.

http://reviews.llvm.org/D10333

llvm-svn: 239409
2015-06-09 15:20:42 +00:00
Rafael Espindola
5eb02e45e3 Simplify another function that doesn't fail.
llvm-svn: 238703
2015-06-01 00:27:26 +00:00
Rafael Espindola
a4d22472f3 Simplify interface of function that doesn't fail.
llvm-svn: 238700
2015-05-31 23:52:50 +00:00
Petar Jovanovic
9720283e99 [Mips64] Add support for MCJIT for MIPS64r2 and MIPS64r6
Add support for resolving MIPS64r2 and MIPS64r6 relocations in MCJIT.

Patch by Vladimir Radosavljevic.

Differential Revision: http://reviews.llvm.org/D9667

llvm-svn: 238424
2015-05-28 13:48:41 +00:00
Keno Fischer
e6892c8149 Reapply [RuntimeDyldELF] Fold Placeholder into Addend
This reapplies r235060 and 235070, which were reverted because of test failures
in LLDB. The failure was caused because at  moment RuntimeDyld is processing
relocations for all sections, irrespective of whether we actually load them
into memory or not, but RuntimeDyld was not actually remembering where in memory
the unrelocated section is. This commit includes a fix for that issue by
remembering that pointer, though the longer term fix should be to stop processing
unneeded sections.

Original Summary:

This allows us to get rid of the original unrelocated object file after
we're done processing relocations (but before applying them).
MachO and COFF already do not require this (currently we have temporary hacks
to prevent ownership from being released, but those are brittle and should be
removed soon).

The placeholder mechanism allowed the relocation resolver to look at original
object file to obtain more information that are required to apply the
relocations. This is usually necessary in two cases:

- For relocations targetting sub-word memory locations, there may be pieces
  of the instruction at the target address which we should not override.
- Some relocations on some platforms allow an extra addend to be encoded in
  their immediate fields.

The problem is that in the second case the information cannot be recovered
after the relocations have been applied once because they will have been
overridden. In the first case we also need to be careful to not use any bits
that aren't fixed and may have been overriden by applying a first relocation.

In the past both have been fixed by just looking at original object file. This
patch attempts to recover the information from the first by looking at the
relocated object file, while the extra addend in the second case is read
upon relocation processing and addend to the regular addend.

I have tested this on X86. Other platforms represent my best understanding
of how those relocations should work, but I may have missed something because
I do not have access to those platforms.
We will keep the ugly workarounds in place for a couple of days, so this commit
can be reverted if it breaks the bots.

Differential Revision: http://reviews.llvm.org/D9028

llvm-svn: 236341
2015-05-01 20:21:45 +00:00
Keno Fischer
02628def32 [RuntimeDyldELF] Improve GOT support
Summary:
This is the first in a series of patches to eventually add support for TLS relocations to RuntimeDyld. This patch resolves an issue in the current GOT handling, where GOT entries would be reused between object files, which leads to the same situation that necessitates the GOT in the first place, i.e. that the 32-bit offset can not cover all of the address space. Thus this patch makes the GOT object-file-local.
Unfortunately, this still isn't quite enough, because the MemoryManager does not yet guarantee that sections are allocated sufficiently close to each other, even if they belong to the same object file. To address this concern, this patch also adds a small API abstraction on top of the GOT allocation mechanism that will allow (temporarily, until the MemoryManager is improved) using the stub mechanism instead of allocating a different section. The actual switch from separate section to stub mechanism will be part of a follow-on commit, so that it can be easily reverted independently at the appropriate time.

Test Plan: Includes a test case where the GOT of two object files is artificially forced to be apart by several GB.

Reviewers: lhames

Reviewed By: lhames

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D8813

llvm-svn: 234839
2015-04-14 02:10:35 +00:00
Lang Hames
2be1bbea7e [RuntimeDyld] Always allocate at least 1 byte for object sections in the JIT to
ensure that section addresses are distinct.

mapSectionAddress will fail if two sections are allocated the same address,
which can happen if any section has zero size (since malloc(0) is implementation
defined). Unfortunately I've been unable to repro this with a simple test case.

Fixes <rdar://problem/20314015>.

llvm-svn: 234299
2015-04-07 06:27:56 +00:00
Alexei Starovoitov
36df1ca5f1 [MCJIT] In debug memory dump output, don't truncate 64 bit addresses
Summary: In dumpMemorySections a cast was too short, and in resolveRelocations a format string was too short.

Test Plan:
Enable debug build and run a program which invokes MCJIT::finalizeObject(). Saw valid input as below (highlighted addresses were previously truncated):

```
Parse relocations:
Resolving relocations Section #0	**0x7f4c1337b000**
----- Contents of section socket1 before relocations -----
**0x00007f4c1337b000**: 18 01 00 00 01 01 01 0a 00 00 00 00 04 03 02 01
0x00007f4c1337b010: 7b 1a f8 ff 00 00 00 00 18 11 00 00 05 00 00 00

```

Reviewers: lhames

Reviewed By: lhames

Subscribers: llvm-commits, ast

Differential Revision: http://reviews.llvm.org/D8681

llvm-svn: 233512
2015-03-30 05:15:57 +00:00
Lang Hames
633fe146e9 [MCJIT][Orc] Refactor RTDyldMemoryManager, weave RuntimeDyld::SymbolInfo through
MCJIT.

This patch decouples the two responsibilities of the RTDyldMemoryManager class,
memory management and symbol resolution, into two new classes:
RuntimeDyld::MemoryManager and RuntimeDyld::SymbolResolver.

The symbol resolution interface is modified slightly, from:

  uint64_t getSymbolAddress(const std::string &Name);

to:

  RuntimeDyld::SymbolInfo findSymbol(const std::string &Name);

The latter passes symbol flags along with symbol addresses, allowing RuntimeDyld
and others to reason about non-strong/non-exported symbols.


The memory management interface removes the following method:

  void notifyObjectLoaded(ExecutionEngine *EE,
                          const object::ObjectFile &) {}

as it is not related to memory management. (Note: Backwards compatibility *is*
maintained for this method in MCJIT and OrcMCJITReplacement, see below).


The RTDyldMemoryManager class remains in-tree for backwards compatibility.
It inherits directly from RuntimeDyld::SymbolResolver, and indirectly from
RuntimeDyld::MemoryManager via the new MCJITMemoryManager class, which
just subclasses RuntimeDyld::MemoryManager and reintroduces the
notifyObjectLoaded method for backwards compatibility).

The EngineBuilder class retains the existing method:

  EngineBuilder&
  setMCJITMemoryManager(std::unique_ptr<RTDyldMemoryManager> mcjmm);

and includes two new methods:

  EngineBuilder&
  setMemoryManager(std::unique_ptr<MCJITMemoryManager> MM);

  EngineBuilder&
  setSymbolResolver(std::unique_ptr<RuntimeDyld::SymbolResolver> SR);

Clients should use EITHER:

A single call to setMCJITMemoryManager with an RTDyldMemoryManager.

OR (exclusive)

One call each to each of setMemoryManager and setSymbolResolver.

This patch should be fully compatible with existing uses of RTDyldMemoryManager.
If it is not it should be considered a bug, and the patch either fixed or
reverted.

If clients find the new API to be an improvement the goal will be to deprecate
and eventually remove the RTDyldMemoryManager class in favor of the new classes.

llvm-svn: 233509
2015-03-30 03:37:06 +00:00
Lang Hames
b11860362d [Orc][MCJIT][RuntimeDyld] Re-apply r231726 and r231724 with fix suggested by
Dave Blaikie. Thanks Dave!

llvm-svn: 231896
2015-03-11 00:43:26 +00:00
Eric Christopher
0c3c1893c4 Temporarily revert r231726 and r231724 as they're breaking the build.:
Author: Lang Hames <lhames@gmail.com>
Date:   Mon Mar 9 23:51:09 2015 +0000

    [Orc][MCJIT][RuntimeDyld] Add header that was accidentally left out of r231724.

Author: Lang Hames <lhames@gmail.com>
Date:   Mon Mar 9 23:44:13 2015 +0000

    [Orc][MCJIT][RuntimeDyld] Add symbol flags to symbols in RuntimeDyld. Thread the
    new types through MCJIT and Orc.

    In particular, add a 'weak' flag. When plumbed through RTDyldMemoryManager, this
    will allow us to distinguish between weak and strong definitions and find the
    right ones during symbol resolution.

llvm-svn: 231731
2015-03-10 00:33:27 +00:00
Lang Hames
3197fb4a89 [Orc][MCJIT][RuntimeDyld] Add symbol flags to symbols in RuntimeDyld. Thread the
new types through MCJIT and Orc.

In particular, add a 'weak' flag. When plumbed through RTDyldMemoryManager, this
will allow us to distinguish between weak and strong definitions and find the
right ones during symbol resolution.

llvm-svn: 231724
2015-03-09 23:44:13 +00:00
David Majnemer
1a666e0f69 ExecutionEngine: Preliminary support for dynamically loadable coff objects
Provide basic support for dynamically loadable coff objects. Only handles a subset of x64 currently.

Patch by Andy Ayers!

Differential Revision: http://reviews.llvm.org/D7793

llvm-svn: 231574
2015-03-07 20:21:27 +00:00
Rafael Espindola
127b6c3ba7 Don't deference the section_end() iterator.
Hard to test given the undefined behavior nature.

llvm-svn: 229530
2015-02-17 20:07:28 +00:00
Lang Hames
33c9433ed4 Revert r227247 and r227228: "Add weak symbol support to RuntimeDyld".
This has wider implications than I expected when I reviewed the patch: It can
cause JIT crashes where clients have used the default value for AbortOnFailure
during symbol lookup. I'm currently investigating alternative approaches and I
hope to have this back in tree soon.

llvm-svn: 227287
2015-01-28 01:30:37 +00:00
Keno Fischer
88cc26811b [ExecutionEngine] Add weak symbol support to RuntimeDyld
Support weak symbols by first looking up if there is an externally visible symbol we can find,
and only if that fails using the one in the object file we're loading.

Reviewed By: lhames
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D6950

llvm-svn: 227228
2015-01-27 20:02:31 +00:00
Lang Hames
93de2a12a3 [Orc] New JIT APIs.
This patch adds a new set of JIT APIs to LLVM. The aim of these new APIs is to
cleanly support a wider range of JIT use cases in LLVM, and encourage the
development and contribution of re-usable infrastructure for LLVM JIT use-cases.

These APIs are intended to live alongside the MCJIT APIs, and should not affect
existing clients.

Included in this patch:

1) New headers in include/llvm/ExecutionEngine/Orc that provide a set of
   components for building JIT infrastructure.
   Implementation code for these headers lives in lib/ExecutionEngine/Orc.

2) A prototype re-implementation of MCJIT (OrcMCJITReplacement) built out of the
   new components.

3) Minor changes to RTDyldMemoryManager needed to support the new components.
   These changes should not impact existing clients.

4) A new flag for lli, -use-orcmcjit, which will cause lli to use the
   OrcMCJITReplacement class as its underlying execution engine, rather than
   MCJIT itself.

Tests to follow shortly.

Special thanks to Michael Ilseman, Pete Cooper, David Blaikie, Eric Christopher,
Justin Bogner, and Jim Grosbach for extensive feedback and discussion.

llvm-svn: 226940
2015-01-23 21:25:00 +00:00
Lang Hames
2996895f28 [RuntimeDyld] Tidy up emitCommonSymbols a little. NFC.
llvm-svn: 226358
2015-01-17 00:55:05 +00:00
Lang Hames
1f7eab338f [RuntimeDyld] Remove the brace initialization that was introduced in r226341.
Evidently MSVC doesn't like it.

llvm-svn: 226349
2015-01-17 00:32:56 +00:00
Lang Hames
6bfd398022 [RuntimeDyld] Track symbol visibility in RuntimeDyld.
RuntimeDyld symbol info previously consisted of just a Section/Offset pair. This
patch replaces that pair type with a SymbolInfo class that also tracks symbol
visibility. A new method, RuntimeDyld::getExportedSymbolLoadAddress, is
introduced which only returns a non-zero result for exported symbols. For
non-exported or non-existant symbols this method will return zero. The
RuntimeDyld::getSymbolAddress method retains its current behavior, returning
non-zero results for all symbols regardless of visibility.

No in-tree clients of RuntimeDyld are changed. The newly introduced
functionality will be used by the Orc APIs.

No test case: Since this patch doesn't modify the behavior for any in-tree
clients we don't have a good tool to test this with yet. Once Orc is in we can
use it to write regression tests that test these changes.

llvm-svn: 226341
2015-01-16 23:13:56 +00:00
Rafael Espindola
0e77a94fd6 Move three methods only used by MCJIT to MCJIT.
These methods are only used by MCJIT and are very specific to it. In fact, they
are also fairly specific to the fact that we have a dynamic linker of
relocatable objects.

llvm-svn: 223964
2014-12-10 20:46:55 +00:00
Lang Hames
a5cd950c73 [MCJIT] Remove the local symbol table from RuntimeDlyd - it's not needed.
All symbols have to be stored in the global symbol to enable
cross-rtdyld-instance linking, so the local symbol table content is
redundant.

llvm-svn: 222867
2014-11-27 05:40:13 +00:00
Lang Hames
b5c7b1ff83 [MCJIT] Reapply r222828 and r222810-r222812 with fix for MSVC move-op issues.
llvm-svn: 222840
2014-11-26 16:54:40 +00:00
Aaron Ballman
9fb411431d Reverting r222828 and r222810-r222812 as they broke the build on Windows.
http://bb.pgr.jp/builders/ninja-clang-i686-msc17-R/builds/11753

llvm-svn: 222833
2014-11-26 15:27:39 +00:00
Lang Hames
829a19ae74 [MCJIT] Clean up RuntimeDyld's quirky object-ownership/modification scheme.
Previously, when loading an object file, RuntimeDyld (1) took ownership of the
ObjectFile instance (and associated MemoryBuffer), (2) potentially modified the
object in-place, and (3) returned an ObjectImage that managed ownership of the
now-modified object and provided some convenience methods. This scheme accreted
over several years as features were tacked on to RuntimeDyld, and was both
unintuitive and unsafe (See e.g. http://llvm.org/PR20722).

This patch fixes the issue by removing all ownership and in-place modification
of object files from RuntimeDyld. Existing behavior, including debugger
registration, is preserved.

Noteworthy changes include:

(1) ObjectFile instances are now passed to RuntimeDyld by const-ref.
(2) The ObjectImage and ObjectBuffer classes have been removed entirely, they
    existed to model ownership within RuntimeDyld, and so are no longer needed.
(3) RuntimeDyld::loadObject now returns an instance of a new class,
    RuntimeDyld::LoadedObjectInfo, which can be used to construct a modified
    object suitable for registration with the debugger, following the existing
    debugger registration scheme.
(4) The JITRegistrar class has been removed, and the GDBRegistrar class has been
    re-written as a JITEventListener.

This should fix http://llvm.org/PR20722 .

llvm-svn: 222810
2014-11-26 06:53:26 +00:00
Michael J. Spencer
21245af8e7 Fix covered switch warning
llvm-svn: 222209
2014-11-18 01:26:46 +00:00
Daniel Sanders
66e799ff1b [JIT] Fix more missing endian conversions (opcodes for AArch64, ARM, and Mips stub functions, and ARM target in general)
Summary:
Fixed all of the missing endian conversions that Lang Hames and I identified in
RuntimeDyldMachOARM.h.

Fixed the opcode emission in RuntimeDyldImpl::createStubFunction() for AArch64,
ARM, Mips when the host endian doesn't match the target endian.
PowerPC will need changing if it's opcodes are affected by endianness but I've
left this for now since I'm unsure if this is the case and it's the only path
that specifies the target endian.

This patch fixes MachO_ARM_PIC_relocations.s on a big-endian Mips host. This
is the last of the known issues on this host.

Reviewers: lhames

Reviewed By: lhames

Subscribers: aemerson, llvm-commits

Differential Revision: http://reviews.llvm.org/D6130

llvm-svn: 221446
2014-11-06 09:53:05 +00:00
Lang Hames
2d0d096bd1 [MCJIT] Temporarily revert r220245 - it broke several bots.
(See e.g. http://bb.pgr.jp/builders/cmake-llvm-x86_64-linux/builds/17653)

llvm-svn: 220249
2014-10-21 00:24:02 +00:00
Lang Hames
84801c217c [MCJIT] Make MCJIT honor symbol visibility settings when populating the global
symbol table.

Patch by Anthony Pesch. Thanks Anthony!

llvm-svn: 220245
2014-10-20 23:39:54 +00:00
Rafael Espindola
802912743e Remove bogus std::error_code returns form SectionRef.
There are two methods in SectionRef that can fail:

* getName: The index into the string table can be invalid.
* getContents: The section might point to invalid contents.

Every other method will always succeed and returning and std::error_code just
complicates the code. For example, a section can have an invalid alignment,
but if we are able to get to the section structure at all and create a
SectionRef, we will always be able to read that invalid alignment.

llvm-svn: 219314
2014-10-08 15:28:58 +00:00
Rafael Espindola
fdf5f7a9dd Fix indentation.
llvm-svn: 219312
2014-10-08 15:12:20 +00:00
Lang Hames
24f0c24de9 [MCJIT] Don't crash in debugging output for sections that aren't emitted.
llvm-svn: 218836
2014-10-01 21:57:47 +00:00
Lang Hames
da01602647 [MCJIT] Fix some more RuntimeDyld debugging output format specifiers.
llvm-svn: 218328
2014-09-23 19:20:57 +00:00
Lang Hames
8d4d0260a1 [MCJIT] Fix a debugging-output formatting bug in RuntimeDyld.
The mismatched mask (7 vs (ColsPerRow-1)) could lead to partial lines being
printed out of place.

llvm-svn: 218061
2014-09-18 16:43:24 +00:00
Lang Hames
69abd72e49 [MCJIT] Fix a bug RuntimeDyldImpl's read/writeBytesUnaligned methods.
The previous implementation was writing to the high-bytes of integers on BE
targets (when run on LE hosts).

http://llvm.org/PR20640

llvm-svn: 217325
2014-09-07 02:05:26 +00:00
Lang Hames
3e930a3e83 [MCJIT] Const-ify the symbol lookup operations on RuntimeDyld.
llvm-svn: 217263
2014-09-05 18:00:16 +00:00
David Blaikie
847d2a061b unique_ptrify RuntimeDyld::Dyld
llvm-svn: 217180
2014-09-04 18:37:29 +00:00
Lang Hames
778ef5b240 [MCJIT] Add command-line argument to llvm-rtdyld to specify target addresses for
sections.

This allows fine-grained control of the memory layout of hypothetical target
processes for testing purposes.

llvm-svn: 217122
2014-09-04 04:19:54 +00:00
David Blaikie
fba8b20d98 unique_ptrify RuntimeDyldImpl::loadObject
I'm not sure this is a particularly helpful API (to pass ownership and
then return it unconditionally) rather than just pass the underlying
object by non-const reference, but this was the original API so I'll
just make it more safe/stable and anyone else is free to adjust that at
their whim, of course.

llvm-svn: 217081
2014-09-03 21:34:34 +00:00
David Blaikie
ed9709d928 unique_ptrify a bunch of stuff through RuntimeDyld::loadObject
llvm-svn: 217065
2014-09-03 19:48:09 +00:00
Lang Hames
587ee6ab4f [MCJIT] Add a 'section_addr' builtin function to RuntimeDyldChecker.
The syntax of the new builtin is 'section_addr(<filename>, <section-name>)'
(similar to the stub_addr builtin, but without a symbol name). It returns the
base address of the given section in the given object file. This builtin makes
it possible to refer to the contents of sections that cannot contain symbols,
e.g. sections added by the linker itself, like __eh_frame.

llvm-svn: 217010
2014-09-03 05:01:46 +00:00
Lang Hames
e1287c01be [MCJIT] Move endian-aware read/writes from RuntimeDyldMachO into
RuntimeDyldImpl.

These are platform independent, and moving them to the base class allows
RuntimeDyldChecker to use them too.

llvm-svn: 216801
2014-08-29 23:17:47 +00:00
Lang Hames
c5cafbb074 [MCJIT] Fix format specifiers for debug output in RuntimeDyld.
More work on http://llvm.org/PR20640

llvm-svn: 216648
2014-08-28 04:25:17 +00:00
Benjamin Kramer
01e1789d38 Silence unused function warning in Release builds.
llvm-svn: 216458
2014-08-26 14:22:05 +00:00
Lang Hames
f4b3b67f57 [MCJIT] Dump section memory both before and after relocations are applied.
Also switch section memory dump format from 8 to 16 columns.

llvm-svn: 216413
2014-08-25 22:19:14 +00:00
Lang Hames
86b08f02c0 [MCJIT] Make RuntimeDyld dump section contents in -debug mode.
llvm-svn: 216400
2014-08-25 18:37:38 +00:00
Tim Northover
e19bed7d33 AArch64: remove arm64 triple enumerator.
Having both Triple::arm64 and Triple::aarch64 is extremely confusing, and
invites bugs where only one is checked. In reality, the only legitimate
difference between the two (arm64 usually means iOS) is also present in the OS
part of the triple and that's what should be checked.

We still parse the "arm64" triple, just canonicalise it to Triple::aarch64, so
there aren't any LLVM-side test changes.

llvm-svn: 213743
2014-07-23 12:32:47 +00:00
Lang Hames
17e6b9e5ab [MCJIT] Make stub_addr functionality in RuntimeDyldChecker work in release mode.
There's no reason to restrict this particular piece of RuntimeDyldChecker
functionality to +Asserts builds.

This should fix failures in MachO_x86-64_PIC_relocations.s on release bots.

llvm-svn: 213708
2014-07-22 23:50:51 +00:00
Lang Hames
f7acddde5b [MCJIT] Refactor and add stub inspection to the RuntimeDyldChecker framework.
This patch introduces a 'stub_addr' builtin that can be used to find the address
of the stub for a given (<file>, <section>, <symbol>) tuple. This address can be
used both to verify the contents of stubs (by loading from the returned address)
and to verify references to stubs (by comparing against the returned address).

Example (1) - Verifying stub contents:

Load 8 bytes (assuming a 64-bit target) from the stub for 'x' in the __text
section of f.o, and compare that value against the addres of 'x'.

# rtdyld-check: *{8}(stub_addr(f.o, __text, x) = x

Example (2) - Verifying references to stubs:

Decode the immediate of the instruction at label 'l', and verify that it's
equal to the offset from the next instruction's PC to the stub for 'y' in the
__text section of f.o (i.e. it's the correct PC-rel difference).

# rtdyld-check: decode_operand(l, 4) = stub_addr(f.o, __text, y) - next_pc(l)
l:
        movq    y@GOTPCREL(%rip), %rax

Since stub inspection requires cooperation with RuntimeDyldImpl this patch
pimpl-ifies RuntimeDyldChecker. Its implementation is moved in to a new class,
RuntimeDyldCheckerImpl, that has access to the definition of RuntimeDyldImpl.

llvm-svn: 213698
2014-07-22 22:47:39 +00:00
Ulrich Weigand
752b5c9e6e [PowerPC] ELFv2 dynamic loader support
This patch enables the new ELFv2 ABI in the runtime dynamic loader.
The loader has to implement the following features:
- In the ELFv2 ABI, do not look up a function descriptor in .opd, but
  instead use the local entry point when resolving a direct call.
- Update the TOC restore code to use the new TOC slot linkage area
  offset.
- Create PLT stubs appropriate for the ELFv2 ABI.

Note that this patch also adds common-code changes. These are necessary
because the loader must check the newly added ELF flags: the e_flags
header bits encoding the ABI version, and the st_other symbol table
entry bits encoding the local entry point offset.  There is currently
no way to access these, so I've added ObjectFile::getPlatformFlags and
SymbolRef::getOther accessors.

Reviewed by Hal Finkel.

llvm-svn: 213491
2014-07-20 23:53:14 +00:00
Lang Hames
a521688cf4 [MCJIT] Significantly refactor the RuntimeDyldMachO class.
The previous implementation of RuntimeDyldMachO mixed logic for all targets
within a single class, creating problems for readability, maintainability, and
performance. To address these issues, this patch strips the RuntimeDyldMachO
class down to just target-independent functionality, and moves all
target-specific functionality into target-specific subclasses RuntimeDyldMachO.

The new class hierarchy is as follows:

class RuntimeDyldMachO
Implemented in RuntimeDyldMachO.{h,cpp}
Contains logic that is completely independent of the target. This consists
mostly of MachO helper utilities which the derived classes use to get their
work done.


template <typename Impl>
class RuntimeDyldMachOCRTPBase<Impl> : public RuntimeDyldMachO
Implemented in RuntimeDyldMachO.h
Contains generic MachO algorithms/data structures that defer to the Impl class
for target-specific behaviors.

RuntimeDyldMachOARM : public RuntimeDyldMachOCRTPBase<RuntimeDyldMachOARM>
RuntimeDyldMachOARM64 : public RuntimeDyldMachOCRTPBase<RuntimeDyldMachOARM64>
RuntimeDyldMachOI386 : public RuntimeDyldMachOCRTPBase<RuntimeDyldMachOI386>
RuntimeDyldMachOX86_64 : public RuntimeDyldMachOCRTPBase<RuntimeDyldMachOX86_64>
Implemented in their respective *.h files in lib/ExecutionEngine/RuntimeDyld/MachOTargets
Each of these contains the relocation logic specific to their target architecture.

llvm-svn: 213293
2014-07-17 18:54:50 +00:00
Rafael Espindola
db4ed0bdab Remove 'using std::errro_code' from lib.
llvm-svn: 210871
2014-06-13 02:24:39 +00:00
Rafael Espindola
bff5d0d16a Remove all uses of 'using std::error_code' from headers.
llvm-svn: 210866
2014-06-13 01:25:41 +00:00
Lang Hames
36072da3d9 [RuntimeDyld] Add support for MachO __jump_table and __pointers sections, and
SECTDIFF relocations on 32-bit x86.

This fixes several of the MCJIT regression test failures that show up on 32-bit
builds.

<rdar://problem/16886294>

llvm-svn: 208635
2014-05-12 21:39:59 +00:00
Lang Hames
d7bfe4bb86 Back out r208257 while I investigate tester failures.
llvm-svn: 208267
2014-05-07 23:35:53 +00:00
Lang Hames
fd284c6ce1 [RuntimeDyld] Make RuntimeDyldImpl::resolveExternalSymbols preserve the
relocation entries it applies.

Prior to this patch, RuntimeDyldImpl::resolveExternalSymbols discarded
relocations for external symbols once they had been applied. This causes issues
if the client calls MCJIT::finalizeLoadedModules more than once, and updates the
location of any symbols in between (e.g. by calling MCJIT::mapSectionAddress).

No test case yet: None of our in-tree memory managers support moving sections
around. I'll have to hack up a dummy memory manager before I can write a unit
test.

Fixes <rdar://problem/16764378>

llvm-svn: 208257
2014-05-07 22:34:08 +00:00
James Molloy
bd2ffa0f6a [ARM64] Try and make the ELF MCJIT *slightly* less broken for ARM64.
A bunch of switch cases were missing, not just for ARM64 but also for
AArch64_BE. I've fixed all those, but there's zero testing as
ExecutionEngine tests are disabled when crosscompiling and I don't
have a native platform available to test on.

llvm-svn: 207626
2014-04-30 10:15:41 +00:00
David Blaikie
7a1e775a7e PR19553: Memory leak in RuntimeDyldELF::createObjectImageFromFile
This starts in MCJIT::getSymbolAddress where the
unique_ptr<object::Binary> is release()d and (after a cast) passed to a
single caller, MCJIT::addObjectFile.

addObjectFile calls RuntimeDyld::loadObject.
RuntimeDld::loadObject calls RuntimeDyldELF::createObjectFromFile

And the pointer is never owned at this point. I say this point, because
the alternative codepath, RuntimeDyldMachO::createObjectFile certainly
does take ownership, so this seemed like a good hint that this was a/the
right place to take ownership.

llvm-svn: 207580
2014-04-29 21:52:46 +00:00
Craig Topper
353eda484c [C++] Use 'nullptr'.
llvm-svn: 207083
2014-04-24 06:44:33 +00:00
Chandler Carruth
f58e376d23 [Modules] Fix potential ODR violations by sinking the DEBUG_TYPE
definition below all the header #include lines. This updates most of the
miscellaneous other lib/... directories. A few left though.

llvm-svn: 206845
2014-04-22 03:04:17 +00:00
Jim Grosbach
36f025e697 Revert "[rtdyld,c++11] Range'ify symbol table walking."
Tentative revert for
http://lab.llvm.org:8011/builders/llvm-mips-linux/builds/8305.

This reverts commit c2a58efff07294fca724f89500538f2ddbcd12ff.

llvm-svn: 206773
2014-04-21 19:23:59 +00:00
Jim Grosbach
de577e3d68 [rtdyld,c++11] Range'ify symbol table walking.
llvm-svn: 206769
2014-04-21 18:10:31 +00:00
Rafael Espindola
6956b1a517 Convert getFileOffset to getOffset and move it to its only user.
We normally don't drop functions from the C API's, but in this case I think we
can:

* The old implementation of getFileOffset was fairly broken
* The introduction of LLVMGetSymbolFileOffset was itself a C api breaking
  change as it removed LLVMGetSymbolOffset.
* It is an incredibly specialized use case. The only reason MCJIT needs it is
  because of its odd position of being a dynamic linker of .o files.

llvm-svn: 206750
2014-04-21 13:45:32 +00:00
Rafael Espindola
77314aa014 Remove section_rel_empty. Just compare begin() and end() instead.
llvm-svn: 205577
2014-04-03 22:42:22 +00:00
Christian Pirker
2a11160956 Add ARM big endian Target (armeb, thumbeb)
Reviewed at http://llvm-reviews.chandlerc.com/D3095

llvm-svn: 205007
2014-03-28 14:35:30 +00:00
Juergen Ributzka
6ff29a7b2f [MCJIT] Check if there have been errors during RuntimeDyld execution.
llvm-svn: 204837
2014-03-26 18:19:27 +00:00
Christian Pirker
99974c7242 AArch64_BE Elf support for MC-JIT runtime dynamic linker
llvm-svn: 204816
2014-03-26 14:57:32 +00:00
Juergen Ributzka
7608dc0441 [RuntimeDyld] clang-format files.
llvm-svn: 204507
2014-03-21 20:28:42 +00:00
Juergen Ributzka
046709f06b [RuntimeDyld] Allow processRelocationRef to process more than one relocation entry at a time.
Some targets require more than one relocation entry to perform a relocation.
This change allows processRelocationRef to process more than one relocation
entry at a time by passing the relocation iterator itself instead of just
the relocation entry.

Related to <rdar://problem/16199095>

llvm-svn: 204439
2014-03-21 07:26:41 +00:00
Lang Hames
868d4b3122 Add an option to MCJIT to have it forward all sections to the
RTDyldMemoryManager, regardless of whether it thinks they're "required for
execution".

Currently, RuntimeDyld only passes sections that are "required for execution"
to the RTDyldMemoryManager, and takes "required for execution" to mean exactly
"contains symbols or relocations". There are two problems with this:
(1) It can drop sections with anonymous data that is referenced by code.
(2) It leaves the JIT client no way to inspect interesting sections that aren't
    actually required to run the program (e.g dwarf sections).

A test case is still in the works.

Future work: We may want to replace this with a generic section filtering
mechanism, but that will require more consideration. For now, this flag at least
allows clients to volunteer to do the filtering themselves.

Fixes <rdar://problem/15177691>.

llvm-svn: 204398
2014-03-20 21:06:46 +00:00
Alexey Samsonov
aa4d29571c [C++11] Introduce SectionRef::relocations() to use range-based loops
Reviewers: rafael

Reviewed By: rafael

CC: llvm-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D3077

llvm-svn: 203927
2014-03-14 14:22:49 +00:00
Lang Hames
951b235be2 Make createObjectImage and createObjectImageFromFile static methods on the
relevant subclasses of RuntimeDyldImpl. This allows construction of
RuntimeDyldImpl instances to be deferred until after the target architecture is
known.

llvm-svn: 203352
2014-03-08 18:45:12 +00:00
Ahmed Charles
56440fd820 Replace OwningPtr<T> with std::unique_ptr<T>.
This compiles with no changes to clang/lld/lldb with MSVC and includes
overloads to various functions which are used by those projects and llvm
which have OwningPtr's as parameters. This should allow out of tree
projects some time to move. There are also no changes to libs/Target,
which should help out of tree targets have time to move, if necessary.

llvm-svn: 203083
2014-03-06 05:51:42 +00:00